Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Join us at the top event for the global data community, Data Council Austin.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Join us at the top event for the global data community, Data Council Austin.

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Monte Carlo

APRIL 24, 2023

The first is referred to as “events” data, which consists of clickstream events, logs, and other user activity events; while the second type— known as “transactional” data—captures all the upstream services. The scale of data events depends entirely on the product. Missed Nishith’s 5 considerations?

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Data Engineering Podcast

MARCH 24, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Data lakes are notoriously complex. In the event of these different cluster errors, what are the strategies for mitigating and recovering from those failures?

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Netflix Tech

FEBRUARY 9, 2022

JAR) form to be executed as part of the user defined data pipeline. data pipeline ?—?a DAG) for the purpose of transforming data using some business logic. Netflix homegrown CLI tool for data pipeline management. workflow ?—?see SQL) or compiled (e.g. Dataflow ?—?Netflix namespace ?

Data Pipeline 80

Data Pipeline Scala Management Python 80

Data Engineering Podcast

NOVEMBER 26, 2023

Developing event-driven pipelines is going to be a lot easier - Meet Functions! Data lakes are notoriously complex. Memphis Logo]([link] Developing event-driven pipelines is going to be a lot easier - Meet Functions! Developing event-driven pipelines is going to be a lot easier - Meet Functions!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. Cliff Crosland co-founded Scanner to provide fast querying of high scale log data for security auditing.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Netflix Tech

NOVEMBER 14, 2023

Now, let’s explore the state of our pipelines after incorporating Psyberg. Pipelines After Psyberg Let’s explore how different modes of Psyberg could help with a multistep data pipeline. Psyberg Initialization The workflow starts with the Psyberg initialization (init) step.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Ascend.io

DECEMBER 11, 2023

Here’s the deal: for data to truly drive your business forward, you need a reliable and scalable system to keep it moving without hiccups. In other words, you need data orchestration. In this article, we’ll break down what data orchestration is, its significance, and how it differs from data pipeline orchestration.

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. RudderStack’s smart customer data pipeline is warehouse-first.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Databand.ai

AUGUST 30, 2023

It emphasizes the importance of collaboration between different teams, such as data engineers, data scientists, and business analysts, to ensure that everyone has access to the right data at the right time. This includes data ingestion, processing, storage, and analysis.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. We want interoperability for any data stored versus we have to think how to store the data in a specific node to optimize the processing. ” He/She is managing triggers, he/she needs to check conditions (event type ?

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

AUGUST 28, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

APRIL 5, 2021

Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Netflix Tech

NOVEMBER 20, 2023

The need for backfilling could be due to a variety of factors, e.g. (1) upstream data sets got repopulated due to changes in business logic of its data pipeline, (2) business logic was changed in a data pipeline, (3) anew metric was created that needs to be populated for historical time ranges, (4) historical data was found missing, etc.

Process 84

Process Data Pipeline Datasets Aggregated Data 84

Data Engineering Podcast

OCTOBER 29, 2021

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Pipeline 100

Workfall

JUNE 12, 2023

Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention. In real-time data streaming and event-driven architectures, CDC captures data changes to trigger actions or workflows.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. Your newly mimicked datasets are safe to share with developers, QA, data scientists—heck, even distributed teams around the world.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 15, 2021

Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days.

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

OCTOBER 19, 2020

For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home. Go to dataengineeringpodcast.com/conferences to check out the upcoming events being offered by our partners and get registered today!

Machine Learning 100

Machine Learning Data Engineering Data Engineer Data 100

Knowledge Hut

NOVEMBER 2, 2023

An Azure Data Engineer is a professional who is in charge of designing, implementing, and maintaining data processing systems and solutions on the Microsoft Azure cloud platform. A Data Engineer is responsible for designing the entire architecture of the data flow while taking the needs of the business into account.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Monte Carlo

NOVEMBER 14, 2023

This is the world that data orchestration tools aim to create. Data orchestration tools minimize manual intervention by automating the movement of data within data pipelines. This data orchestration tool manages task orchestration, cluster management, monitoring, and error reporting.

Pipeline-centric 40

Pipeline-centric Google Cloud Data Workflow Python 40

AltexSoft

MARCH 14, 2023

Moreover, over 20 percent of surveyed companies were found to be utilizing 1,000 or more data sources to provide data to analytics systems. These sources commonly include databases, SaaS products, and event streams. Databases store key information that powers a company’s product, such as user data and product data.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Netflix Tech

OCTOBER 18, 2022

With the high growth of workflows in the past few years?—?increasing increasing at > 100% a year, the need for a scalable data workflow orchestrator has become paramount for Netflix’s business needs. For example, the daily workflow should process the data when the data partition is ready, not always at midnight.

Java 81

Java Data Machine Learning Systems 81

DataKitchen

OCTOBER 25, 2022

And when these stakeholders can see a shared view of a single pipeline, they know everything that’s happening during a run of that pipeline. The journey reveals all of the complex steps, toolchains, and actions within the data workflow?most That data then fills several database tables.

Data Pipeline 52

Data Pipeline Data Workflow Coding Building 52

Data Engineering Weekly

JANUARY 15, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Knowledge Hut

MAY 23, 2024

Through MS Excel, data science professionals can represent the data simply through rows and columns. BigML: BigML is an online, cloud-based, event-driven tool that helps in data science and machine learning operations. Users can also leverage it for generating interactive visualizations over data.

Data Science 52

Data Science MongoDB Programming Language Unstructured Data 52

Monte Carlo

SEPTEMBER 20, 2022

Here’s how Prefect , Series B startup and creator of the popular data orchestration tool, harnessed the power of data observability to preserve headcount, improve data quality and reduce time to detection and resolution for data incidents. But a growing company means growing data needs. Image courtesy of Prefect.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

AltexSoft

NOVEMBER 7, 2022

The platform went live in 2015 at Airbnb, the biggest home-sharing and vacation rental site, as an orchestrator for increasingly complex data pipelines. It still remains a leading workflow management tool adopted by thousands of companies, from tech giants to startups. How data engineering works. What is Apache Airflow?

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

phData: Data Engineering

JANUARY 3, 2022

How do I know where this data came from or how it’s being used? How do I maintain all my data pipelines? How do I recreate the environment and data sets from scratch? How do I build confidence and trust in the data products I create? One of the worst things that can happen in a system is missing data.

IT 52

IT AWS Software Engineer Software Engineering 52