Data Engineering Podcast

DECEMBER 3, 2023

Andrei Tserakhau has dedicated his careeer to this problem, and in this episode he shares the lessons that he has learned and the work he is doing on his most recent data transfer system at DoubleCloud. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

DECEMBER 31, 2023

In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Designing 130

Designing Data Lake High Quality Data SQL 130

Knowledge Hut

DECEMBER 28, 2023

Today, organizations seek skilled professionals who can harness data’s power to drive informed decisions. As technology evolves, cloud platforms have emerged as the cornerstone of modern data management. Its comprehensive suite of services can handle data at scale. Who is an Azure Data Engineer?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Cloudera

FEBRUARY 11, 2019

The January 2019 “Magic Quadrant for Data Management Solutions for Analytics” provides valuable insights into the status, direction, and players in the DMSA market. 3. Expansion beyond core data management. It accomplishes this while avoiding issues around interoperability among the workloads.

Data Management 43

Data Management Management Metadata Government 43

Ascend.io

DECEMBER 19, 2023

Snowpipe : Automates data loading in near real-time, significantly reducing the latency in data availability for processing and analysis. Streams and Tasks : Enables Change Data Capture (CDC) and automates SQL execution, streamlining the process of tracking and responding to data modifications in your tables.

ETL Tools 52

ETL Tools IT Data Pipeline Data Warehouse 52

Knowledge Hut

SEPTEMBER 29, 2023

You must have a solid grasp of ideas in parallel processing, data architecture, and data computation languages like SQL, Python, or Scala in order to become a Microsoft Certified Azure Data Engineer. Why Should You Get an Azure Data Engineer Certification? Then, you can create analytical layer serving designs.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. To start, can you share your definition of what constitutes a "Data Lakehouse"?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication. How is Elementary designed/implemented?

Project 130

Project Data Lake High Quality Data Data Workflow 130

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. From a product perspective, what are the data challenges that are posed by email?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Podcast

MARCH 24, 2024

In this episode Pete Hunt, CEO of Dagster labs, outlines these new capabilities, how they reduce the burden on data teams, and the increased collaboration that they enable across teams and business units. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

Data Engineering Podcast

DECEMBER 17, 2023

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. In this episode founder Tarush Aggarwal explains how the realities of the modern data stack are impacting data teams and the work that they are doing to accelerate time to value.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. What were the design goals and constraints that led you to this architecture?

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. SIEM) A query engine is useless without data to analyze. What are the data acquisition paths/sources that you are designed to work with?-

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

MARCH 10, 2024

In this episode Alex Merced explains how the branching and merging functionality in Nessie allows you to use the same versioning semantics for your data lakehouse that you are used to from Git. Data lakes are notoriously complex. What are the core problems/complexities that Nessie is designed to solve?

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Data Engineering Podcast

MARCH 3, 2024

Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about how to conduct an AI program for your organization. Data lakes are notoriously complex. What are the foundational/fundamental principles that you need to design around to allow for future flexibility?

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

NOVEMBER 5, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Data lakes are notoriously complex. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Monte Carlo

MAY 14, 2024

Let’s revisit how several of those key table formats have emerged and developed over time: Apache Avro : Developed as part of the Hadoop project and released in 2009, Apache Avro provides efficient data serialization with a schema-based structure. For example, Starburst’s Icehouse implementation pairs Iceberg with open query engine Trino.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

NOVEMBER 19, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data projects are notoriously complex. With multiple stakeholders to manage across varying backgrounds and toolchains even simple reports can become unwieldy to maintain. Data lakes are notoriously complex.

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 12, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. In this episode Satish Jayanthi explains how he is building a framework to allow enterprises to move quickly while maintaining guardrails for data workflows.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Data Engineering Podcast

MAY 22, 2022

Summary The interfaces and design cues that a tool offers can have a massive impact on who is able to use it and the tasks that they are able to perform. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Scala 100

Scala SQL Data Data Workflow 100

Data Engineering Podcast

OCTOBER 29, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Pipeline 100

Data Engineering Podcast

JANUARY 1, 2022

In this episode Emily Riederer shares her work to create a controlled vocabulary for managing the semantic elements of the data managed by her team and encoding it in the schema definitions in her data warehouse. star/snowflake schema, data vault, etc.) Who should be involved in that design process?

Data Warehouse 100

Data Warehouse BI Data Workflow Coding 100

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Can you describe what your conception of a data contract is?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

NOVEMBER 14, 2021

In this episode Michael Harper advocates for proactive data quality and starting with the source, rather than being reactive and having to work backwards from when a problem is found. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Warehouse 100

Data Warehouse BI Data Workflow Data 100

Data Engineering Weekly

JANUARY 15, 2023

SiliconANGLE theCUBE: Analyst Predictions 2023 - The Future of Data Management By far one of the best analyses of trends in Data Management. The author writes an exciting article about switching to Github actions for dbt to save $65,000 [link] Sponsored: Upsolver - Write a SQL Query, Get a Data-in-Motion Pipeline!

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. This job requires a handful of skills, starting from a strong foundation of SQL and programming languages like Python , Java , etc.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Meltano

OCTOBER 5, 2022

We’ll explain what a data engineer is, what the job entails, and how to become a data engineer. Plus, we’ll explain how data engineers use Meltano, our DataOps platform, for efficient data management. What Is Data Engineering? A data engineer’s focus is mainly on raw data, formats, security, and storage.

Data Engineering 52

Data Engineering Data Engineer Engineering Raw Data 52

Knowledge Hut

SEPTEMBER 27, 2023

Its interface is designed to be straightforward, making it easy for individuals familiar with Excel to transition into the world of data analysis and reporting. This characteristic minimizes the initial training required, enabling users to quickly harness the power of data.

BI 52

BI Business Intelligence Certification Data Security 52

Monte Carlo

NOVEMBER 14, 2023

But even as the modern data stack continues to evolve, Airflow maintains its title as a perennial data orchestration favorite—and for good reason. Built by an early major Airflow committer as a first-principles rewrite of Airflow, Prefect was designed to remove much of the friction common to interacting with an orchestrator.

Pipeline-centric 40

Pipeline-centric Google Cloud Data Workflow Python 40

DataKitchen

APRIL 13, 2021

Piperr.io — Pre-built data pipelines across enterprise stakeholders, from IT to analytics, tech, data science and LoBs. Prefect Technologies — Open-source data engineering platform that builds, tests, and runs data workflows. Genie — Distributed big data orchestration service by Netflix. Telm.ai — Telm.ai

Consulting 123

Consulting Machine Learning Data Science Data Pipeline 123

Meltano

OCTOBER 5, 2022

Think about data operations as a factory assembly line where a warehouse engineer optimizes and automates processes to increase productivity and product quality. In the same way, a DataOps engineer designs the data assembly line that enables data scientists to derive insights from data analytics faster and with fewer errors.

Engineering 52

Engineering Raw Data Data Pipeline ETL Tools 52

Netflix Tech

FEBRUARY 9, 2022

SQL) or compiled (e.g. JAR) form to be executed as part of the user defined data pipeline. data pipeline ?—?a DAG) for the purpose of transforming data using some business logic. workflow ?—?see The design of a cron job is simple, you take some system command, you pick the schedule to run it on and you are done.

Data Pipeline 80

Data Pipeline Management Scala Python 80