Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Towards Data Science

OCTOBER 25, 2023

As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. You probably work closely with the colleague who builds the data pipeline that you were consuming. There was only one data team, two at most.

Data Pipeline 82

Data Pipeline SQL Data Architecture Data 82

Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Atlan is the metadata hub for your data ecosystem.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

OCTOBER 16, 2022

In this episode Adrian Kosowski explains how the Pathway product got started, how its design simplifies the creation of data products that support supply chain operations, and how developers can help to build an ecosystem of applications that allow businesses to accelerate their time to insight.

Database 100

Database Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 2, 2022

In this episode Roambee CEO, Sanjay Sharma, shares the types of questions that companies are asking about their logistics, the technical work that they do to provide ways to answer those questions, and how they approach the challenge of data quality in its many forms. Atlan is the metadata hub for your data ecosystem.

Metadata 100

Metadata Electronics MongoDB MySQL 100

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Tom Baeyens explains their reasons for creating a new syntax for expressing and validating checks for data assets and processes, as well as how to incorporate it into your own projects. Atlan is the metadata hub for your data ecosystem. For more information on Prefect, visit dataengineeringpodcast.com/prefect.

Building 100

Building Metadata MongoDB MySQL 100

Knowledge Hut

MAY 9, 2024

Responsibilities Responsibilities of data modelers include validating data models, evaluating existing systems, ensuring data consistency, and optimizing metadata. Skills Required Data modelers must be proficient in SQL, metadata management, data modeling, interpersonal communication, and statistical analysis.

Data Science 52

Data Science Data Mining Data Architect Programming Language 52

Data Engineering Podcast

OCTOBER 9, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata AWS MongoDB MySQL 100

Data Engineering Podcast

AUGUST 21, 2022

In this episode Tracy Daniels, CDO of Truist, shares her journey into the position, her responsibilities, and her relationship to the data professionals in her organization. Atlan is the metadata hub for your data ecosystem. RudderStack helps you build a customer data platform on your warehouse or data lake.

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Data Engineering Podcast

JULY 10, 2022

In this episode Inbar Yogev and Lior Winner share the journey that they and their teams at Riskified have been on for their data platform. They also discuss how they have established a guild system for training and supporting data professionals in the organization. Atlan is the metadata hub for your data ecosystem.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

JUNE 12, 2022

Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth. In this episode she shares her thoughts and insights on how to be intentional about establishing your own data team. Atlan is the metadata hub for your data ecosystem.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Data Engineering Podcast

JULY 31, 2022

In this episode Ernie Ostic shares the approach that he and his team at Manta are taking to build a complete view of data lineage across the various data systems in your organization and the useful applications of that information in the work of every data stakeholder. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode founders Barr Moses and Lior Gavish rejoin the show to reflect on the evolution and adoption of data observability technologies and the capabilities that are being introduced as the broader ecosystem adopts the practices. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

In this episode Paolo Platter shares the lessons they have learned in that process, the Data Mesh Boost platform that they have built to reduce some of the boilerplate required to make it successful, and some of the considerations to make when deciding if a data mesh is the right choice for you.

Metadata 100

Metadata MongoDB MySQL Business Intelligence 100

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Atlan is the metadata hub for your data ecosystem.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

NOVEMBER 20, 2021

In this episode he shares the journey that he and his team at Elementl have taken to understand the state of the ecosystem and how they can provide a foundational layer for a holistic data platform. Start trusting your data with Monte Carlo today! If this resonates with you, you’re not alone.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Ascend.io

AUGUST 31, 2023

In this article, we assess: The role of the data warehouse on one hand, and the data lake on the other; The features of ETL and ELT in these two architectures; The evolution to EtLT; The emerging role of data pipelines. This unchanging schema forms the foundation for all queries and business intelligence.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

AltexSoft

FEBRUARY 11, 2023

It serves as a foundation for the entire data management strategy and consists of multiple components including data pipelines; , on-premises and cloud storage facilities – data lakes , data warehouses , data hubs ;, data streaming and Big Data analytics solutions ( Hadoop , Spark , Kafka , etc.);

Data Architect 87

Data Architect Certification Generalist Big Data 87

Monte Carlo

JANUARY 16, 2024

And with so many data teams across functions, how does Fox approach data governance? But, when a business has scaled to the level of Fox, it’s common for data silos to occur, especially as pipelines become more complex. The complexity of a modern data pipeline. Image from Castor.

Government 52

Government High Quality Data Data Governance Entertainment 52

Monte Carlo

JUNE 28, 2022

Traditionally, data lakes held raw data in its native format and were known for their flexibility, speed, and open source ecosystem. By design, data was less structured with limited metadata and no ACID properties. Unity Catalog The Unity Catalog unifies metastores, catalogs, and metadata within Databricks.

Data Lake 52

Data Lake Metadata AWS Data Warehouse 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

DataKitchen

AUGUST 3, 2021

The past decades of enterprise data platform architectures can be summarized in 69 words. First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. Data professionals are not perfectly interchangeable.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Monte Carlo

AUGUST 17, 2021

And your entire data engineering team is all in on this “data as code” movement. The flip side of this data-driven coin: your stakeholders (CEO and CTO included) ping you nearly every other hour to ask you: “is my data up-to-date?” To achieve this data trust, they turned to Monte Carlo.

Insurance 40

Insurance Retail Data Pipeline Portfolio 40

Data Engineering Podcast

DECEMBER 29, 2022

In this episode she shares the strategic and tactical elements of how to make more effective use of the technical and organizational resources that are available to you for getting work done with data. Atlan is the metadata hub for your data ecosystem. Struggling with broken pipelines? Missing data? Missing data?

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

DECEMBER 25, 2022

Summary Five years of hosting the Data Engineering Podcast has provided Tobias Macey with a wealth of insight into the work of building and operating data systems at a variety of scales and for myriad purposes. Atlan is the metadata hub for your data ecosystem. Struggling with broken pipelines? Missing data?

Building 100

Building Metadata Business Intelligence Data Lake 100

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is the metadata hub for your data ecosystem. Struggling with broken pipelines? Missing data?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? What is data pipeline architecture?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

AltexSoft

OCTOBER 30, 2021

If you’re an executive who has a hard time understanding the underlying processes of data science and get confused with terminology, keep reading. We will try to answer your questions and explain how two critical data jobs are different and where they overlap. Data science vs data engineering.

Data Engineering 98

Data Engineering Data Engineer Engineering Machine Learning 98

Data Engineering Podcast

OCTOBER 23, 2022

In this episode Shane Gibson shares practical advice and insights from his years of experience as a consultant and engineer working in data about how to adopt agile principles in your data work so that you can move faster and provide more value to the business, while building systems that are maintainable and adaptable.

Project 130

Project Metadata MongoDB MySQL 130

Cloudera

FEBRUARY 8, 2021

Below is the entire set of steps in the data lifecycle, and each step in the lifecycle will be supported by a dedicated blog post(see Fig. 1): Data Collection – data ingestion and monitoring at the edge (whether the edge be industrial sensors or people in a vehicle showroom). 2 ECC data enrichment pipeline.

Data Pipeline 118

Data Pipeline Building Manufacturing Data Warehouse 118

Data Engineering Podcast

NOVEMBER 27, 2022

In this episode Wes McKinney shares the ways that Arrow and its related projects are improving the efficiency of data systems and driving their next stage of evolution. Atlan is the metadata hub for your data ecosystem. Struggling with broken pipelines? Missing data? Start trusting your data with Monte Carlo today!

Data Process 100

Data Process Process Metadata Business Intelligence 100

Data Engineering Podcast

JUNE 26, 2022

In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

ProjectPro

SEPTEMBER 26, 2021

According to a survey, big data engineering job interviews increased by 40% in 2020 compared to only a 10% rise in Data science job interviews. Table of Contents Big Data Engineer - The Market Demand Who is a Big Data Engineer? Most of these are performed by Data Engineers.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52