Data Engineering Podcast

OCTOBER 9, 2022

In this episode Vusal Dadalov explains how the platform is implemented, the motivation for a truly open architecture, and how they have invested in integrating with the broader ecosystem to make it easy for you to get started. Can you describe how the Iomete platform is implemented? When is Iomete the wrong choice?

Metadata 100

Metadata AWS MongoDB MySQL 100

Data Engineering Podcast

JANUARY 25, 2021

In this episode they describe the data integration challenges facing many B2B companies, how their work on the Hotglue platform simplifies their efforts, and how they have designed the platform to make these ETL workloads embeddable and self service for end users. How are they currently handling solutions for those problems?

Data Integration 100

Data Integration IT Python BI 100

Data Engineering Podcast

FEBRUARY 22, 2021

In this episode co-founders John Lafleur and Michel Tricot share the story of how and why they created Airbyte, discuss the project’s design and architecture, and explain their vision of what an open soure data integration platform should offer. Interview Introduction How did you get involved in the area of data management?

Data Integration 100

Data Integration Data Warehouse Data Pipeline BI 100

Data Engineering Podcast

MARCH 22, 2021

Unfortunately, this is a non-trivial undertaking, particularly for teams that don’t have extensive experience working with streaming data and complex distributed systems. In this episode Raghu Murthy, founder and CEO of Datacoral, does a deep dive on how he and his team manage change data capture pipelines in production.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Towards Data Science

JUNE 19, 2023

I use Compose daily at work and for my personal projects to build and manage my data pipelines, and today I’d like to show how you can build your ETL projects with Compose. The second variable is an example of how to define a custom FooConnector, which you could create using the @make_connector decorator in a plugin.

PostgreSQL 79

PostgreSQL SQL Python Project 79

Meltano

DECEMBER 7, 2022

Luckily we have a community full of data experts that have worked with Meltano, Stitch, Airbyte, Fivetran, and all the big extract and load tools on the market. How do you know that orders got modified, and if so, which ones? The post 5 Helpful Extract & Load Practices for High-Quality Raw Data appeared first on Meltano.

Raw Data 52

Raw Data Metadata Data Database 52

Monte Carlo

SEPTEMBER 12, 2022

New frameworks, new challenges, and new technologies are constantly shifting how engineers think about their work and their roles within their organizations. With the success of companies like Databricks, Confluent, and Elastic, we’ve proven that the model works very well, and then we see an influx of venture capital.”

BI 52

BI Data Governance ETL Tools Data Warehouse 52

Grouparoo

DECEMBER 14, 2021

Companies like Fivetran , Meltano , and Airbyte are leading the ELT trend. How the data is loaded often depends on its intended use case. Incremental loads are usually more complicated than full loads since they only load new or updated records and have to work around issues with ordering, schema evolution, and monitorability.

Process 52

Process Raw Data Data Warehouse Data Pipeline 52

Data Engineering Podcast

JULY 2, 2021

In this episode the members of the Meltano project share the work they are doing to improve the discovery, quality, and capabilities of Singer taps and targets. They explain their work on the Meltano Hub and the Singer SDK and their long term goals for the Singer community. What are the main goals of the Meltano Hub?

Data Integration 100

Data Integration SQL BI Data Warehouse 100

Christophe Blefari

MARCH 2, 2024

Mistral ( credits ) Hello all, this is the Data News, this week edition might be smaller than usual in term of comments as I'm working on a Data News related project that takes me a bit of time, which will probably lead to a series of articles. Let's not forget Meltano. It's built on top of dlt. Written in Go.

Data 162

Data Python Database BI 162

Data Engineering Podcast

JULY 13, 2020

The Meltano project is aiming to provide a solution to that situation. In this episode, project lead Douwe Maan shares the history of how Meltano got started, the motivation for the recent shift in focus, and how it is implemented. Interview Introduction How did you get involved in the area of data management?

Data Integration 100

Data Integration Data Engineering Data Engineer Data 100

Meltano

OCTOBER 5, 2022

We’ll explain what a data engineer is, what the job entails, and how to become a data engineer. Plus, we’ll explain how data engineers use Meltano, our DataOps platform, for efficient data management. But what is data engineering exactly and what does a data engineer do? Interested in a data platform? What Is Data Engineering?

Data Engineering 52

Data Engineering Data Engineer Engineering Raw Data 52

Meltano

DECEMBER 19, 2022

In this article, you’ll explore all of these aspects in depth and learn how your team can benefit from focusing on code reviews. Say that you’ve just completed the implementation for an API that you’ve been working on. How else did the author test these changes (i.e. What is a Code Review? end-to-end tests)?

Coding 52

Coding PostgreSQL Data Database 52

Meltano

NOVEMBER 29, 2022

If you’re working with dbt , you need to check these five packages out. They supercharge the dbt workflows of the meltano community members. The Meltano community is full of ELT & dbt experts who actively contribute dbt packages, so we asked them for their favorite dbt packages!

SQL 52

SQL Utilities Coding Database 52

Data Engineering Podcast

DECEMBER 21, 2021

In this episode Artom Keydunov and Pavel Tiunov share their work on Cube.js Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values.

Data Analytics 130

Data Analytics BI Computer Science SQL 130

Meltano

OCTOBER 5, 2022

Get Started With DataOps In this guide, we’ll explain all you need to know about a DataOps engineer, including their key responsibilities, and how these experts use a DataOps platform, like Meltano , to optimize internal processes. How does this work? DataOps engineers. What Is A DataOps Engineer?

Engineering 52

Engineering Raw Data Data Pipeline ETL Tools 52

Data Engineering Podcast

JUNE 5, 2022

However, they are still challenging to work with and manage for anyone who isn’t in a technical role. How would you characterize your positioning in the data ecosystem? Who are the personas that you optimize for and how does that manifest in your product design and feature priorities? How is the Y42 platform implemented?

MongoDB 100

MongoDB MySQL Data Lake Data Warehouse 100

Data Engineering Podcast

DECEMBER 25, 2022

Summary Five years of hosting the Data Engineering Podcast has provided Tobias Macey with a wealth of insight into the work of building and operating data systems at a variety of scales and for myriad purposes. You can fix things yourself and better direct the community's work than just filing a ticket and hoping with a vendor.

Building 100

Building Metadata Business Intelligence Data Lake 100

Meltano

OCTOBER 19, 2022

In this PostgreSQL vs. MySQL comparison, we analyze crucial differences between the two database management systems to discover how they work and when to use them. It’s possible to work in both RDMS with: C/C++ PHP Delphi Java JavaScript (Node.js) Python .NET Every server works independently from others.

PostgreSQL 52

PostgreSQL MySQL Database SQL 52

Grouparoo

SEPTEMBER 7, 2021

Tools like Fivetran, Airbyte, Meltano, and others are focused on getting data into a data warehouse, and tools like dbt are focused on transforming that data into understandable and uniform data for teams to analyze. Here is a deep-dive example of how Operational Analytics drives growth.

ETL Tools 52

ETL Tools Data Warehouse Business Intelligence Datasets 52

Data Engineering Podcast

DECEMBER 26, 2021

In this episode Maxime Beauchemin returns to revisit what it means to be a data engineer and how the role has changed over the past 5 years. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Warehouse 100

Towards Data Science

NOVEMBER 4, 2023

We will discuss how to use this knowledge to power advanced analytics pipelines and operational excellence. Does your DE work well enough to fuel advanced data pipelines and Business intelligence (BI)? How to Become a Data Engineer As a data engineer, I am tasked to design efficient data processes almost every day. What is it?

Data Engineering 85

Data Engineering Data Engineer Engineering BI 85