Rockset

MARCH 1, 2023

When you deconstruct the core database architecture, deep in the heart of it you will find a single component that is performing two distinct competing functions: real-time data ingestion and query serving. When data ingestion has a flash flood moment, your queries will slow down or time out making your application flaky.

Data Ingestion 144

Data Ingestion Database Architecture Cloud Storage 144

Rockset

OCTOBER 11, 2022

In this blog, we’ll compare and contrast how Elasticsearch and Rockset handle data ingestion as well as provide practical techniques for using these systems for real-time analytics. Or, they can periodically scan their relational database to get access to the most up to date records and reindex the data in Elasticsearch.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Snowflake

MARCH 21, 2023

In February, Snowflake launched new features around streaming data ingestion and data governance and improved SQL experience and performance, with enhancements to Search Optimization Service and more. Read the announcement blog for more details and get started guides.

Retail 72

Retail Healthcare Data Ingestion Consulting 72

Cloudera

FEBRUARY 9, 2021

Cloudera Operational Database is now available in three different form-factors in Cloudera Data Platform (CDP). . If you are new to Cloudera Operational Database, see this blog post. In this blog post, we’ll look at both Apache HBase and Apache Phoenix concepts relevant to developing applications for Cloudera Operational Database.

Database 101

Database Java Data Ingestion SQL 101

Rockset

JANUARY 30, 2024

With this architecture, users can separate ingestion compute from query compute, all while accessing the same real-time data. Microbatching : An option to microbatch ingestion based on the latency requirements of the use case. This is not a hands-free operation and also involves the transfer of data across nodes.

Data Ingestion 64

Data Ingestion Utilities Architecture SQL 64

Databand.ai

AUGUST 30, 2023

The core philosophy of DataOps is to treat data as a valuable asset that must be managed and processed efficiently. It emphasizes the importance of collaboration between different teams, such as data engineers, data scientists, and business analysts, to ensure that everyone has access to the right data at the right time.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

Snowflake

SEPTEMBER 13, 2023

Snowpark External Access – public preview External Access is in public preview on AWS regions. Users can now easily connect to external network locations, including external LLMs, from their Snowpark code while maintaining high security and governance over their data. Snowpark Python Updates Snowpark support for Python 3.9

Python 77

Python SQL Data Pipeline Data Ingestion 77

Cloudera

AUGUST 26, 2021

In addition to big data workloads, Ozone is also fully integrated with authorization and data governance providers namely Apache Ranger & Apache Atlas in the CDP stack. While we walk through the steps one by one from data ingestion to analysis, we will also demonstrate how Ozone can serve as an ‘S3’ compatible object store.

Data Science 109

Data Science Cloud Hadoop Metadata 109

Cloudyard

APRIL 4, 2024

However, the volume of daily transaction data poses challenges in effectively segmenting customers and optimizing engagement. This blog post explores how Snowpark, a powerful tool for data processing within Snowflake, can be used to perform RFM segmentation and unlock actionable customer insights.

Retail 40

Retail Data Ingestion Metadata Datasets 40

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. Accelerated Data Analytics DataOps tools help automate and streamline various data processes, leading to faster and more efficient data analytics.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Monte Carlo

JUNE 22, 2023

They combined these principles with FAIR , a popular framework in the world of pharma that stands for findability, accessibility, interoperability, and reusability. When they started this journey, the data team did not want to leave modeling and architecture behind. So they agreed to leverage Data Vault 2.0

Building 52

Building Data Ingestion Data Business Analyst 52

Cloudera

FEBRUARY 16, 2021

In the previous blog post , we looked at some of the application development concepts for the Cloudera Operational Database (COD). In this blog post, we’ll see how you can use other CDP services with COD. Integrated across the Enterprise Data Lifecycle . Cloudera Shared Data Experience (SDX) . Conclusion.

Database 81

Database Machine Learning Data Lake Kafka 81

Cloudera

APRIL 9, 2021

This is part 4 in this blog series. This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The second blog dealt with creating and managing Data Enrichment pipelines.

Machine Learning 98

Machine Learning Manufacturing Data Collection Data Science 98

Knowledge Hut

APRIL 23, 2024

Two popular approaches that have emerged in recent years are data warehouse and big data. While both deal with large datasets, but when it comes to data warehouse vs big data, they have different focuses and offer distinct advantages. Analytics: Both data warehousing and big data platforms enable analytical capabilities.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

Edureka

FEBRUARY 9, 2023

As a certified Azure Data Engineer, you have the skills and expertise to design, implement and manage complex data storage and processing solutions on the Azure cloud platform. As the demand for data engineers grows, having a well-written resume that stands out from the crowd is critical.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Cloudera

JULY 9, 2021

If you’ve followed Cloudera for a while, you know we’ve long been singing the praises—or harping on the importance, depending on perspective—of a solid, standalone enterprise data strategy. The ways data strategies are implemented, the resulting outcomes and the lessons learned along the way provide important guardrails.

Government 110

Government Building Cloud Data Ingestion 110

DareData

JULY 5, 2023

In this blog post, we aim to share practical insights and techniques based on our real-world experience in developing data lake infrastructures for our clients - let's start! The Data Warehouse(s) facilitates data ingestion and enables easy access for end-users.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Snowflake

AUGUST 16, 2023

Read our Summit recap blog for highlights across industries or watch Summit sessions now on-demand. Applications Snowflake Native App Framework now available in AWS – public preview Snowflake Native Apps are an entirely new way to put data to work. Learn more about ML-Powered Functions in our blog or in Snowflake documentation.

Scala 52

Scala Transportation Kafka Data Lake 52

Ascend.io

JUNE 6, 2023

In this blog post, we delve deep into the factors that contribute to how Ascend runs in the cloud, focusing on the primary areas that contribute to costs in this context: storage, compute, networking, and retries. In the large-scale data pipeline networks operated by Ascend, storage can account for 30% of total cloud costs.

Cloud 52

Cloud Data Pipeline Data Ingestion Cloud Storage 52

Cloudera

DECEMBER 14, 2021

During the economic upheaval brought about by the COVID pandemic, the government of Australia was working with the Commonwealth Bank of Australia (CBA) to access real-time data about everyday financial transactions to understand the economic and social impacts of the crisis. Commonwealth Bank of Australia.

Transportation 107

Transportation Telecommunication Banking Data Lake 107

Snowflake

MARCH 30, 2023

Data Vault as a practice does not stipulate how you transform your data, only that you follow the same standards to populate business vault link and satellite tables as you would to populate raw vault link and satellite tables. Feature engineering: Data is transformed to support ML model training. ML workflow, ubr.to/3EJHjvm

Engineering 82

Engineering Raw Data Data Science Scala 82

Mutt Data

JULY 16, 2023

The company application allows users to apply for credit at the moment of purchase in stores with interest-free installment payments that enable borrowers to access affordable loans for their shopping. The project included: Setting up Databricks as the place to host the company data lake with delta tables to use as a single source of truth.

Data Lake 40

Data Lake Data Ingestion Banking Data Pipeline 40

ProjectPro

FEBRUARY 6, 2023

Data engineers often use Google Cloud Pub/Sub to design asynchronous workflows, publish event notifications, and stream data from several processes or devices. This blog provides an overview of Google Cloud Pub/Sub that will help you understand the framework and its suitable use cases for your data engineering projects.

Google Cloud 52

Google Cloud Cloud Cloud Storage Data Ingestion 52

Mutt Data

JULY 16, 2023

The company application allows users to apply for credit at the moment of purchase in stores with interest-free installment payments that enable borrowers to access affordable loans for their shopping. The project included: Setting up Databricks as the place to host the company data lake with delta tables to use as a single source of truth.

Data Lake 40

Data Lake AWS Data Ingestion Banking 40

Monte Carlo

MARCH 24, 2023

Integrity is crucial for meeting regulatory requirements, maintaining user confidence, and preventing data breaches or loss. How Do You Maintain Data Integrity? Data integrity issues can arise at multiple points across the data pipeline. We often refer to these issues as data freshness or stale data.

Data Validation 52

Data Validation Data Integration Data Cleanse Data Pipeline 52

Rockset

MAY 31, 2023

This technical blog highlights the architecture that paved the way for this accomplishment. Workload Isolation: When real-time data ingestion and query workloads run on the the same compute units, they directly compete for resources. It uses the open-source storage engine called RocksDB to store and access the Converged Index.

Data Ingestion 52

Data Ingestion Datasets Architecture Retail 52

Cloudera

APRIL 20, 2021

It can access data from inside the business, like ERP and asset management, outside sources, like edge devices and external assets, and correlate them for real-time predictive maintenance. A modern streaming architecture consists of critical components that provide data ingestion, security and governance, and real-time analytics.

Hospitality 92

Hospitality Kafka Retail Data Ingestion 92

Data Engineering Weekly

OCTOBER 30, 2022

I found the blog helpful in understanding the generative model’s historical development and the path forward. link] Sponsored- [New eBook] The Ultimate Data Observability Platform Evaluation Guide Considering investing in a data quality solution? The author explains how to dump the history of blockchains into S3.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Snowflake

APRIL 20, 2023

Data Pipelines Snowpipe Streaming – public preview While data generated in real time is valuable, it is more valuable when paired with historical data that helps provide context. Read our blog to learn more. Learn more about what these new functions are and how they work in our recent blog post.

Medical 57

Medical Retail Python Pharmaceutical 57

Cloudera

JUNE 7, 2021

It calls out that Cloudera DataFlow “ includes streaming flow and streaming data processing unified with Cloudera Data Platform ”. Hundreds of customers across multiple industry verticals are leveraging Cloudera DataFlow today for various streaming use cases like Clickstreams, log ingestion/analysis, social stream analysis, etc.

Kafka 101

Kafka Data Ingestion Architecture Cloud 101

Snowflake

MAY 22, 2023

Databases, shares, role-based access control, governance policies, compute resources, network policies, and all forms of user authentication—including password, federated authentication with single sign-on (SSO), and OAuth—failover seamlessly. Read our announcement blog post for more. Visit our documentation page to learn more.

Healthcare 52

Healthcare Scala Medical Transportation 52

phData: Data Engineering

SEPTEMBER 19, 2023

By employing robust data modeling techniques, businesses can unlock the true value of their data lake and transform it into a strategic asset. With many data modeling methodologies and processes available, choosing the right approach can be daunting. Want to learn more about data governance?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Pinterest Engineering

AUGUST 18, 2022

The HBase Ecosystem, though having various advantages like strong consistency at row level in high volume requests, flexible schema, low latency access to data, Hadoop integration, etc. In this blog post, we will first learn the various approaches considered for data migration with their trade offs.

Data Ingestion 52

Data Ingestion Hadoop Database Kafka 52

Data Engineering Weekly

NOVEMBER 20, 2022

With Upsolver SQLake, you build a pipeline for data in motion simply by writing a SQL query defining your transformation. The blog narrates the European Commission’s updated version of the European Standard Contractual Clauses (EU SCCS) and how to prepare to handle the privacy laws. Kudos to the author and the Atlassian team.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Knowledge Hut

MARCH 19, 2024

I will explore the top 10 AWS applications and their use cases in this blog. Security : AWS promotes security and compliance, providing comprehensive security features and controls to protect data and applications. These include encryption, identity and access management , network security, and compliance certifications.

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Cloudera

NOVEMBER 17, 2020

The data and the techniques presented in this prototype are still applicable as creating a PCA feature store is often part of the machine learning process. . The process followed in this prototype covers several steps that you should follow: Data Ingest – move the raw data to a more suitable storage location.

Deep Learning 110

Deep Learning Machine Learning Raw Data Data Ingestion 110