Data Engineering Podcast

MARCH 22, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Cloudera

JULY 14, 2023

While these instructions are carried out for Cloudera Data Platform (CDP), Cloudera Data Engineering, and Cloudera Data Warehouse, one can extrapolate them easily to other services and other use cases as well. In all the use cases we are trying to migrate a table named “events.”

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Monte Carlo

MAY 18, 2023

Depending on your requirements (more on that in the next section), it is prudent to evaluate open source data lineage tools alongside the several other types of platforms that provide data lineage capabilities. You may even already have access to one of them. Why do you need data lineage? “I

Metadata 64

Metadata Data Lake Government Data Governance 64

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

AltexSoft

AUGUST 29, 2023

Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. This structure is made efficient by data engineering practices that include object storage. Data warehouse vs. data lake in a nutshell.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

ProjectPro

FEBRUARY 16, 2023

With so much riding on the efficiency of ETL processes for data engineering teams, it is essential to take a deep dive into the complex world of ETL on AWS to take your data management to the next level. ETL has typically been carried out utilizing data warehouses and on-premise ETL tools.

AWS 52

AWS Data Management ETL Tools Management 52

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Cloudera

SEPTEMBER 17, 2020

By enabling their event analysts to monitor and analyze events in real time, as well as directly in their data visualization tool, and also rate and give feedback to the system interactively, they increased their data to insight productivity by a factor of 10. . Our solution: Cloudera Data Visualization.

Unstructured Data 80

Unstructured Data Data Warehouse Pharmaceutical MySQL 80

Databand.ai

JUNE 28, 2023

This might include processes like data extraction from different sources, data cleansing, data transformation (like aggregation), and loading the data into a database or a data warehouse. Having a bigger and more specialized data team can help, but it can hurt if those team members don’t coordinate.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. For example, grouping the ones about metadata, discoverability, and column naming might have made a lot of sense.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Striim

JANUARY 4, 2023

Companies need to analyze large volumes of datasets, leading to an increase in data producers and consumers within their IT infrastructures. These companies collect data from production applications and B2B SaaS tools (e.g., This data makes its way into a data repository, like a data warehouse (e.g.,

PostgreSQL 52

PostgreSQL Data Warehouse Data Lake Data 52

dbt Developer Hub

JANUARY 24, 2023

Ensures developers can work quickly by shortening the CI feedback loop Reduce costs in your data warehouse by running only what's been modified I think we can all agree that increasing developer productivity while simultaneously reducing costs is an outcome that every company strives for.

Metadata 52

Metadata Python Cloud Data Warehouse 52

Afterpay Tech

SEPTEMBER 6, 2022

This pattern also enables us to move away from the dependency on the binlog replication for sending data to the data warehouse. PCI-DSS is a set of requirements to ensure any company or seller that handles cardholder data is able to safely and securely accept, store, process and transmit cardholder data during card transactions.

Kafka 52

Kafka AWS Metadata Data Warehouse 52

Hepta Analytics

FEBRUARY 14, 2022

This week, we got to think about our data ingestion design. We looked at the following: How do we ingest – ETL vs ELT Where do we store the data – Data lake vs data warehouse Which tool to we use to ingest – cronjob vs workflow engine NOTE : This weeks task requires good internet speed and good compute.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Cloudera

SEPTEMBER 15, 2022

July brings summer vacations, holiday gatherings, and for the first time in two years, the return of the Massachusetts Institute of Technology (MIT) Chief Data Officer symposium as an in-person event. A key area of focus for the symposium this year was the design and deployment of modern data platforms. What is a data fabric?

Data Lake 79

Data Lake Analytics Application Cloud Storage Architecture 79

Monte Carlo

SEPTEMBER 20, 2022

The complexity of Prefect’s data pipelines underscores the value—and challenges—of achieving data reliability at scale. But a growing company means growing data needs. After some initial cheerleading, Dylan convinced the Prefect team to stand up a Monte Carlo instance for their BigQuery data warehouse.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Data Engineering Weekly

NOVEMBER 6, 2022

link] Pinterest: How Pinterest Leverages Realtime User Actions in Recommendation to Boost Homefeed Engagement Volume Looping in real-time user interaction events with the recommendation engine can significantly improve the user experience. Every data processing engine has one metadata store to integrate.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. Quality problems lead to first responders unable to check into disaster sites or parents unable to access ESA funds. So our solution was to start using an intentional contract: Events.

Kafka 52

Kafka ETL Tools BI Database 52

Monte Carlo

AUGUST 17, 2021

“What excites me about Monte Carlo is their vision for making the delivery of data more reliable and transparent through observability. Data is integral to Kolibri Games’ DNA, powering our product roadmap, marketing strategy, and growth operations.

Insurance 40

Insurance Retail Data Pipeline Portfolio 40

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Metadata 69

Metadata Java PostgreSQL Data Warehouse 69

Data Engineering Podcast

MAY 18, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Interview Introduction How did you get involved in the area of data management?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Towards Data Science

FEBRUARY 9, 2024

In truth, the synergy between batch and streaming pipelines is essential for tackling the diverse challenges posed to your data platform at scale. The key to seamlessly addressing these challenges lies, unsurprisingly, in data orchestration. Their robust core offering seamlessly integrates data warehouses with data-hungry applications.

Building 82

Building Transportation Data Lake Metadata 82

Precisely

JANUARY 25, 2024

There are three potential approaches to mainframe modernization: Data Replication creates a duplicate copy of mainframe data in a cloud data warehouse or data lake, enabling high-performance analytics virtually in real time, without negatively impacting mainframe performance. Best Practice 2. Best Practice 3.

Metadata 69

Metadata Data Governance Relational Database Government 69

ProjectPro

FEBRUARY 8, 2023

It offers users a data integration tool that organizes data from many sources, formats it, and stores it in a single repository, such as data lakes, data warehouses, etc., Glue uses ETL jobs for extracting data from various AWS cloud services and integrating it into data warehouses and lakes.

AWS 98

AWS Scala Metadata Data Lake 98

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

Cloudera

JANUARY 21, 2021

While cloud-native, point-solution data warehouse services may serve your immediate business needs, there are dangers to the corporation as a whole when you do your own IT this way. Cloudera Data Warehouse (CDW) is here to save the day! CDW is an integrated data warehouse service within Cloudera Data Platform (CDP).

IT 93

IT Data Lake Data Warehouse Cloud Storage 93

Data Engineering Weekly

MARCH 24, 2023

Data engineering starts to add value to the business by capturing events at each step of the business process. The events are then further enriched and analyzed to bring visibility to business operations. Architectural patterns for Data Creation There are three types of architecture patterns in data creation.

Engineering 67

Engineering Data Transportation Database 67

Monte Carlo

MAY 31, 2022

Metadata from the data warehouse/lake and from the BI tool of record can then be used to map the dependencies between the tables and dashboards. It also becomes outdated virtually the moment it’s mapped as your environment continues to ingest more data and you continue to layer on additional solutions.

Data Lake 52

Data Lake Building Scala Metadata 52

Snowflake

MAY 22, 2023

Cross-Cloud Snowgrid Account Replication expands replication beyond databases – general availability Account Replication, now generally available, expands replication beyond databases to account metadata and integrations, making business continuity truly turnkey. Learn more.

Healthcare 52

Healthcare Scala Medical Transportation 52

Monte Carlo

FEBRUARY 8, 2024

Robust role based access controls? How does it impact data warehouse/lakehouse performance? Data observability tools should offer both broad automated metadata monitoring across all the tables once they have been added to your selected schemas, as well as deep monitoring for issues inherent in the data itself.

BI 59

BI Data Pipeline Data Warehouse Data 59

Monte Carlo

AUGUST 10, 2023

Data lineage provides the answer by telling you which upstream sources and downstream ingestors were impacted, as well as which teams are generating the data and who is accessing it. At its core, there are three pillars of observability data : Metrics refer to a numeric representation of data measured over time.

Data Pipeline 98

Data Pipeline Software Engineer Software Engineering Machine Learning 98

Rockset

NOVEMBER 28, 2022

Today, I’m excited to share a few product updates we’ve been working on related to real-time Change Data Capture (CDC), including early access for popular templates and 3rd-party CDC platforms. Data warehouses were built for batch jobs, so we shouldn’t be surprised by this. What Is CDC And Why Is It Useful? The upshot?

Data Warehouse 52

Data Warehouse PostgreSQL MongoDB Data Pipeline 52

AltexSoft

NOVEMBER 7, 2022

Metadata database. A metadata database stores information about user permissions, past and current DAG and task runs, DAG configurations, and more. By default, Airflow handles metadata with SQLite which is meant for development only. Full REST API: easy access for third parties. Since the 2.0

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Cloudera

FEBRUARY 8, 2021

The first blog introduced a mock connected vehicle manufacturing company, The Electric Car Company (ECC), to illustrate the manufacturing data path through the data lifecycle. Having completed the Data Collection step in the previous blog, ECC’s next step in the data lifecycle is Data Enrichment.

Data Pipeline 114

Data Pipeline Building Manufacturing Data Warehouse 114