Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Use cases change, needs change, technology changes – and therefore data infrastructure should be able to scale and evolve with change.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

Towards Data Science

NOVEMBER 4, 2023

Platform Specific Tools and Advanced Techniques Photo by Christopher Burns on Unsplash The modern data ecosystem keeps evolving and new data tools emerge now and then. In this article, I want to talk about crucial things that affect data engineers. Are your data pipelines efficient? Data warehouse exmaple.

Data Engineering 83

Data Engineering Data Engineer Engineering BI 83

Data Engineering Podcast

MAY 13, 2021

Summary There is a lot of attention on the database market and cloud data warehouses. While they provide a measure of convenience, they also require you to sacrifice a certain amount of control over your data. Firebolt is the fastest cloud data warehouse. Visit dataengineeringpodcast.com/firebolt to get started.

PostgreSQL 100

PostgreSQL Data Warehouse Building MySQL 100

Data Engineering Podcast

JANUARY 1, 2022

Summary Communication and shared context are the hardest part of any data system. In recent years the focus has been on data catalogs as the means for documenting data assets, but those introduce a secondary system of record in order to find the necessary information.

Data Warehouse 100

Data Warehouse BI Data Workflow Data 100

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

FEBRUARY 8, 2021

Summary With all of the tools and services available for building a data platform it can be difficult to separate the signal from the noise. In this episode Zeeshan Qureshi and Michelle Ark share their experiences using DBT to manage the data warehouse for Shopify.

Data Warehouse 100

Data Warehouse Building BI SQL 100

Data Engineering Podcast

MAY 1, 2022

Summary The predominant pattern for data integration in the cloud has become extract, load, and then transform or ELT. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

APRIL 23, 2023

Meroxa was created to enable teams of all sizes to deliver real-time data applications. In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Data Engineering Podcast

AUGUST 31, 2020

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. The current trends in data warehousing are oriented around cloud native architectures that take advantage of dynamic scaling and the separation of compute and storage.

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Data Engineering Podcast

DECEMBER 8, 2019

Summary Data warehouses have gone through many transformations, from standard relational databases on powerful hardware, to column oriented storage engines, to the current generation of cloud-native analytical engines. How does it compare to the other available platforms for data warehousing?

Data Warehouse 100

Data Warehouse Cloud AWS Relational Database 100

Data Engineering Podcast

DECEMBER 26, 2021

Summary Data Engineering is still a relatively new field that is going through a continued evolution as new technologies are introduced and new requirements are understood. In this episode Maxime Beauchemin returns to revisit what it means to be a data engineer and how the role has changed over the past 5 years.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Warehouse 100

Data Engineering Podcast

MAY 27, 2021

Summary The data warehouse has become the focal point of the modern data platform. With increased usage of data across businesses, and a diversity of locations and environments where data needs to be managed, the warehouse engine needs to be fast and easy to manage.

Data Warehouse 100

Data Warehouse Cloud PostgreSQL Kafka 100

Knowledge Hut

MARCH 28, 2024

Data science has become one of the most trending fields today. Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 This demonstrates the increasing need for Microsoft Certified Data Engineers. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

JANUARY 18, 2021

Summary The data warehouse has become the central component of the modern data stack. This is an interesting conversation about the importance of the data warehouse and how it can be used beyond just internal analytics. And don’t forget to thank them for their continued support of this show!

Data Warehouse 100

Data Warehouse BI Data Data Pipeline 100

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Engineering at Meta

FEBRUARY 20, 2024

We’ve partnered with Voltron Data and the Arrow community to align and converge Apache Arrow with Velox , Meta’s open source execution engine. This new convergence helps Meta and the larger community build data management systems that are unified, more efficient, and composable.

Data Management 98

Data Management Bytes Management Datasets 98

Monte Carlo

APRIL 4, 2023

Data mesh vs data warehouse is an interesting framing because it is not necessarily a binary choice depending on what exactly you mean by data warehouse (more on that later). Despite their differences, however, both approaches require high-quality, reliable data in order to function. What is a Data Mesh?

Data Warehouse 52

Data Warehouse Data Governance Data Architecture 52

Data Engineering Podcast

JULY 17, 2022

Summary There are extensive and valuable data sets that are available outside the bounds of your organization. Whether that data is public, paid, or scraped it requires investment and upkeep to acquire and integrate it with your systems. Atlan is the metadata hub for your data ecosystem.

Data Management 130

Data Management Management Metadata MongoDB 130

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Data Engineering Weekly

MARCH 24, 2024

link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024. link] Mercado Libre Tech: Data Mesh @ MELI - Building Highways for Thousands of Data Producers Ok, Data Mesh is still alive!!

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Data Engineering Podcast

APRIL 9, 2023

Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. Now that the data warehouse has taken center stage a new approach of composable customer data platforms is emerging.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. ClickHouse is an open source, column-oriented database engine built for interactive analytics with linear scalability. Integrating data across the enterprise has been around for decades – so have the techniques to do it.

Data Warehouse 100

Data Warehouse MySQL Data Lake Hadoop 100

Data Engineering Podcast

SEPTEMBER 19, 2021

This brings with it a unique set of challenges for data collection, data management, and analytical capabilities. In this episode Jillian Rowe shares her experience of working in the field and supporting teams of scientists and analysts with the data infrastructure that they need to get their work done. Missing data?

Data Engineering 100

Data Engineering Data Engineer Engineering Data Lake 100

Monte Carlo

AUGUST 25, 2023

Modern companies are ingesting, storing, transforming, and leveraging more data to drive more decision-making than ever before. Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. But, the options for data storage are evolving quickly. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Data Engineering Podcast

JUNE 25, 2021

Summary Data Engineering is a broad and constantly evolving topic, which makes it difficult to teach in a concise and effective manner. In this episode they reflect on the lessons that they learned while teaching the first cohort of their bootcamp how to be effective data engineers. No more scripts, just SQL.

Data Engineering 100

Data Engineering Data Engineer Engineering BI 100

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is Data Science? What are the roles and responsibilities of a Data Engineer? What is Data Science?

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Maxime Beauchemin

AUGUST 28, 2017

This post follows up on The Rise of the Data Engineer , a recent post that was an attempt at defining data engineering and described how this new role relates to historical and modern roles in the data space. Understanding and exposing the adversity that the role is facing is a first step towards finding solutions.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

Knowledge Hut

MARCH 13, 2024

Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Knowledge Hut

MARCH 13, 2024

Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Knowledge Hut

DECEMBER 21, 2023

Data Science is the world's most rapidly growing sector and data engineers are at the forefront. In this article, we will understand the promising data engineer career outlook and what it takes to succeed in this role. What is Data Engineering? What are the Data Engineer Career Opportunities?

Data Engineering 52

Data Engineering Data Engineer Engineering MongoDB 52

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. Structuring data refers to converting unstructured data into tables and defining data types and relationships based on a schema. What is Data Warehouse? .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Data Engineering Weekly

NOVEMBER 5, 2023

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. The blog is a classic case study for data engineers who like to build SQL-like abstractions. Visit rudderstack.com to learn more.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Weekly

DECEMBER 21, 2022

The Rise of Data Modeling Data modeling has been one of the hot topics in Data LinkedIn. Hadoop put forward the schema-on-read strategy that leads to the disruption of data modeling techniques as we know until then. Let’s reference what the data world looked like before the Hadoop era. Why is it so?

Data Engineering 57

Data Engineering Data Engineer Engineering Hadoop 57

Data Engineering Podcast

NOVEMBER 16, 2020

Summary The core mission of data engineers is to provide the business with a way to ask and answer questions of their data. This often takes the form of business intelligence dashboards, machine learning models, or APIs on top of a cleaned and curated data set.

Data Management 100

Data Management Management BI Business Intelligence 100