Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Atlan is the metadata hub for your data ecosystem.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

JULY 31, 2022

In this episode Ernie Ostic shares the approach that he and his team at Manta are taking to build a complete view of data lineage across the various data systems in your organization and the useful applications of that information in the work of every data stakeholder. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Weekly

DECEMBER 29, 2022

Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise.

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Data Engineering Podcast

FEBRUARY 20, 2022

In this episode Guy Yachdav, director of software engineering for ImmunAI, shares the complexities that are inherent to managing data workflows for bioinformatics. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Workfall

JUNE 12, 2023

In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Data, the lifeblood of organisations, holds the key to unlocking untapped potential and propelling businesses forward. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. Data As Code is a very strong choice : we do not want any UI because it is an heritage of the ETL period. What you have to code is this workflow ! We want to have our hands free and be totally devoted to devops principles.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps. data scientist or data analyst). data scientist or data analyst).

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

APRIL 5, 2021

She explains how the design of the platform is informed by the needs of managing data projects for large and small teams across her previous roles, how it integrates with your existing systems, and how it can work to bring everyone onto the same page. What portions of the data workflow is Atlan responsible for? Here’s Why.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

AUGUST 5, 2019

Announcements Welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Monte Carlo

JANUARY 16, 2024

Factor in the advertising strategies, media production, partner programming, audience analytics…and you’re looking at an ocean of data that would fill even the deepest trench (we’d like a television show about that too, please!). So how does Fox’s data strategy support these complex data workflows? Image from Castor.

Government 52

Government High Quality Data Data Governance Entertainment 52

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Weekly

JANUARY 15, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. 2023 predictions from the panel are; Unified metadata becomes kingmaker.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Monte Carlo

SEPTEMBER 20, 2022

Here’s how Prefect , Series B startup and creator of the popular data orchestration tool, harnessed the power of data observability to preserve headcount, improve data quality and reduce time to detection and resolution for data incidents. But Monte Carlo doesn’t stop at the “most important” tables.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. When is JSON-LD the wrong choice?

Building 189

Building BI SQL Python 189

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is the metadata hub for your data ecosystem. Missing data? Missing data?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Data lakes are notoriously complex.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Databand.ai

AUGUST 30, 2023

Here are some of the reasons why DataOps tools are important: Improved Collaboration DataOps tools enable better collaboration between data teams, including data engineers, data scientists, and data analysts. This enables data teams to quickly and easily find the data they need for their analytics projects.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

AltexSoft

NOVEMBER 7, 2022

You can hardly compare data engineering toil with something as easy as breathing or as fast as the wind. The platform went live in 2015 at Airbnb, the biggest home-sharing and vacation rental site, as an orchestrator for increasingly complex data pipelines. How data engineering works. Metadata database.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Databand.ai

AUGUST 30, 2023

Unified DataOps represents a fresh approach to managing and synchronizing data operations across several domains, including data engineering, data science, DevOps, and analytics. Technical Challenges Choosing appropriate tools and technologies is critical for streamlining data workflows across the organization.

Data Governance 40

Data Governance Data Cleanse Government Data Pipeline 40

Netflix Tech

NOVEMBER 20, 2023

These challenges are currently addressed in suboptimal and less cost efficient ways by individual local teams to fulfill the needs, such as Lookback: This is a generic and simple approach that data engineers use to solve the data accuracy problem. Users configure the workflow to read the data in a window (e.g.

Process 84

Process Data Pipeline Datasets SQL 84

Ascend.io

MAY 19, 2023

Picture a data engineering leader, like a captain at the helm of a ship, navigating the vast ocean of data. The modern data stack promised a vessel with cutting-edge navigation tools to sail smoothly through this ocean. What are the key characteristics of the post-modern data stack?

Data Pipeline 52

Data Pipeline Data Engineering Data Engineer Media 52

Ascend.io

DECEMBER 11, 2023

Data orchestration is the process of efficiently coordinating the movement and processing of data across multiple, disparate systems and services within a company. So, why is data orchestration a big deal? It automates and optimizes data processes, reducing manual effort and the likelihood of errors.

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

Monte Carlo

JUNE 14, 2023

For example, data may be extracted from multiple sources, reorganized, and joined at different times before ultimately being served to its final destination. Why is data pipeline architecture important? Data then, and even today for some organizations, was primarily hosted in on-premises databases with non-scalable storage.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Ascend.io

MARCH 17, 2023

As a result, stakeholders limit their reliance on data, making decisions based on gut instinct rather than facts. To shift this harmful approach, companies need to make fundamental changes to their data engineering function and start running at speed and with agility. And that leaves them with one choice.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineer 52

Ascend.io

MARCH 17, 2023

As a result, stakeholders limit their reliance on data, making decisions based on gut instinct rather than facts. To shift this harmful approach, companies need to make fundamental changes to their data engineering function and start running at speed and with agility. And that leaves them with one choice.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineer 52

Data Engineering Podcast

SEPTEMBER 18, 2022

In this episode he discusses the challenge of maintaining shared visibility and understanding of data work across the various stakeholders and his efforts to make it a seamless experience. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Building 100

Building Metadata MongoDB MySQL 100

AltexSoft

JUNE 7, 2021

A HDFS Master Node, called a NameNode , keeps metadata with critical information about system files (like their names, locations, number of data blocks in the file, etc.) and keeps track of storage capacity, a volume of data being transferred, etc. No real-time data processing. HDFS master-slave structure.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

Netflix Tech

OCTOBER 18, 2022

With the high growth of workflows in the past few years?—?increasing increasing at > 100% a year, the need for a scalable data workflow orchestrator has become paramount for Netflix’s business needs. A workflow instance is an execution of a workflow, similarly, an execution of a step is called a step instance.

Java 81

Java Data Machine Learning Systems 81

Databand.ai

DECEMBER 29, 2022

She regularly contributes to IBM’s Journey to AI blog and shares her advice on LinkedIn around data strategy, data science, women in AI, data and analytics, data governance, and artificial intelligence. On LinkedIn, he posts regularly about AI, data, data science, data engineering, and machine learning.

BI 52

BI Consulting Data Science Data Governance 52

AltexSoft

MARCH 14, 2023

That’s why some MDS tools are commercial distributions designed to be low-code or even no-code, making them accessible to data practitioners with minimal technical expertise. This means that companies don’t necessarily need a large data engineering team. Data democratization.

IT 59

IT Data Warehouse Data Governance Data Lake 59