Data Engineering Podcast

JANUARY 30, 2022

Summary Pandas is a powerful tool for cleaning, transforming, manipulating, or enriching data, among many other potential uses. As a result it has become a standard tool for data engineers for a wide range of applications. What are the main tasks that you have seen Pandas used for in a data engineering context?

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

Data Engineering Podcast

MAY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex. My thanks to the team at Code Comments for their support.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Engineering at Meta

FEBRUARY 20, 2024

This new convergence helps Meta and the larger community build data management systems that are unified, more efficient, and composable. Meta’s Data Infrastructure teams have been rethinking how data management systems are designed.

Data Management 98

Data Management Bytes Management Datasets 98

Data Engineering Podcast

DECEMBER 26, 2021

Summary Data Engineering is still a relatively new field that is going through a continued evolution as new technologies are introduced and new requirements are understood. In this episode Maxime Beauchemin returns to revisit what it means to be a data engineer and how the role has changed over the past 5 years.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Warehouse 100

Data Engineering Podcast

JULY 10, 2022

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Weekly

MAY 5, 2024

link] LinkedIn: LakeChime - A Data Trigger Service for Modern Data Lakes LinkedIn points out two critical flaws in a partitioned approach to data management. The granularity of partition creation constrained data consumption.

Data Engineering 67

Data Engineering Data Engineer Engineering Google Cloud 67

Data Engineering Weekly

MARCH 3, 2024

Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup. There will be food, networking, and real-world talks around data engineering.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Your newly mimicked datasets are safe to share with developers, QA, data scientists—heck, even distributed teams around the world. What do you have planned for the future of Profisee?

Data Management 100

Data Management Management MongoDB Scala 100

Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data.

Data Management 130

Data Management Management Metadata MongoDB 130

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is Data Science? What are the roles and responsibilities of a Data Engineer? And many more. And many more.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Data Engineering Weekly

FEBRUARY 25, 2024

Are you ready to take Europe's data engineering scene to the next level? If you want to shape the future of data in Europe, join our core working group. They developed a new data structure with explicit aggregation layers, leading to a threefold increase in dashboard usage and more efficient data analysis processes.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Podcast

SEPTEMBER 19, 2021

This brings with it a unique set of challenges for data collection, data management, and analytical capabilities. In this episode Jillian Rowe shares her experience of working in the field and supporting teams of scientists and analysts with the data infrastructure that they need to get their work done. Missing data?

Data Engineering 100

Data Engineering Data Engineer Engineering Data Lake 100

Data Engineering Podcast

JUNE 25, 2021

Summary Data Engineering is a broad and constantly evolving topic, which makes it difficult to teach in a concise and effective manner. In this episode they reflect on the lessons that they learned while teaching the first cohort of their bootcamp how to be effective data engineers.

Data Engineering 100

Data Engineering Data Engineer Engineering BI 100

Data Engineering Weekly

DECEMBER 25, 2023

Welcome to another insightful edition of Data Engineering Weekly. As we approach the end of 2023, it's an opportune time to reflect on the key trends and developments that have shaped the field of data engineering this year. In conclusion, 2023 has been a year of significant developments and shifts in data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Data Engineering Podcast

JULY 24, 2022

Summary Data engineering is a difficult job, requiring a large number of skills that often don’t overlap. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.

Data Engineering 100

Data Engineering Data Engineer Lambda Architecture Engineering 100

Data Engineering Podcast

JULY 17, 2022

Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Weekly

MARCH 31, 2024

[link] Sponsored: Accelerate Your Data Apps to Sub-Second Query Performance with Cube Cloud Learn strategies to overcome slow queries in this guide on optimizing your data reporting for speed and responsiveness, aligning perfectly with your users' expectations.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Data Engineering Podcast

AUGUST 17, 2020

Summary Most databases are designed to work with textual data, with some special purpose engines that support domain specific formats. TileDB is a data engine that was built to support every type of data by using multi-dimensional arrays as the foundational primitive. What was your motivation for building it?

Data Engineering 100

Data Engineering Data Engineer Engineering Database Design 100

Data Engineering Weekly

OCTOBER 29, 2023

The explosion of modern data stacks made it much harder to build. Criteo shares its experience building data lineage from data stewards manually mapping the relationship to automation with pattern recognition. The sheer volume of data making it impossible to manage cataloging at scale.

Data Engineering 52

Data Engineering Data Engineer Engineering Algorithm 52

Data Engineering Weekly

JUNE 12, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 57

Data Engineering Data Engineer Engineering AWS 57

Knowledge Hut

MARCH 13, 2024

Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Knowledge Hut

MARCH 13, 2024

Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Knowledge Hut

MARCH 28, 2024

Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 All these numbers point to one thing–increased job roles and careers, especially when we talk about data engineering jobs in Azure, which are on the rise every year. Let’s get started.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Weekly

MARCH 24, 2024

Mercado Libre writes about its adoption of Data Mesh, which, necessitated by the company's rapid growth and diverse ecosystem, decentralizes data production across domains, leveraging cultural shifts and technological advancements for independent data management.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Data Engineering Weekly

JULY 23, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. See how it works today. While on travel Aswin & I were busy preparing for DewCon.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Data Engineering Podcast

JUNE 8, 2020

Summary The landscape of data management and processing is rapidly changing and evolving. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering?

Data Management 100

Data Management Management Machine Learning Portfolio 100

Cloudera

NOVEMBER 28, 2023

With its rise in popularity generative AI has emerged as a top CEO priority, and the importance of performant, seamless, and secure data management and analytics solutions to power those AI applications is essential. This means you can expect simpler data management and drastically improved productivity for your business users.

Certification 82

Certification Data Management Management Cloud 82

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Maxime Beauchemin

AUGUST 28, 2017

This post follows up on The Rise of the Data Engineer , a recent post that was an attempt at defining data engineering and described how this new role relates to historical and modern roles in the data space. In my experience, it’s rare to find any sort of decent dev or test environments in the big data world.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

Ascend.io

FEBRUARY 28, 2024

The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How does AI recalibrate the workload and priorities of data teams? How can data engineers harness the power of AI?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Weekly

DECEMBER 10, 2023

Capital One: Insights on building a data strategy to drive business value One of the hotly debated and many companies struggling with is to build an agile data strategy to drive business value. There is no shortage of data, but data management is still hard, which requires skillful execution and alignment across the org.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Data Engineering Weekly

NOVEMBER 5, 2023

The blog is a classic case study for data engineers who like to build SQL-like abstractions. ” [link] Meta: Automating data removal Deleting is the hardest part of data management, and Meta writes about its internal system design to automate the data removal process.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Podcast

DECEMBER 2, 2019

Citadel is no exception, and in this episode Michael Watson and Robert Krzyzanowski share their experiences managing and leading the data engineering teams that power the business. This was a great conversation for an inside look at how to build and maintain a data driven culture.

Data Engineering 100

Data Engineering Data Engineer Engineering Machine Learning 100

Data Engineering Podcast

JANUARY 30, 2022

Summary The Data Engineering Podcast has been going for five years now and has included conversations and interviews with a huge number of guests, covering a broad range of topics. In this episode he shares some reflections on producing the podcast, compiling the book, and relevant trends in the ecosystem of data engineering.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

Data Engineering Podcast

APRIL 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer Engineering IT 100

Data Engineering Podcast

FEBRUARY 18, 2024

In this episode Dain Sundstrom, CTO of Starburst, explains how the combination of the Trino query engine and the Iceberg table format offer the ease of use and execution speed of data warehouses with the infinite storage and scalability of data lakes. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262