Data Engineering Podcast

JANUARY 30, 2022

He recently wrote a book on effective patterns for Pandas code, and in this episode he shares advice on how to write efficient data processing routines that will scale with your data volumes, while being understandable and maintainable. You can observe your pipelines with built in metadata search and column level lineage.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps. How does this differ from "business as usual" in the data industry?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

APRIL 5, 2021

She explains how the design of the platform is informed by the needs of managing data projects for large and small teams across her previous roles, how it integrates with your existing systems, and how it can work to bring everyone onto the same page. What portions of the data workflow is Atlan responsible for?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Data Engineering Podcast

AUGUST 5, 2019

Announcements Welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Knowledge Hut

SEPTEMBER 29, 2023

Why Should You Get an Azure Data Engineer Certification? Becoming an Azure data engineer allows you to seamlessly blend the roles of a data analyst and a data scientist. One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

SEPTEMBER 17, 2023

Linked data technologies provide a means of tightly coupling metadata with raw information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. What is the overlap between knowledge graphs and "linked data products"?

Building 189

Building BI SQL Python 189

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is the metadata hub for your data ecosystem. Missing data? Stale dashboards?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Data lakes are notoriously complex. Data lakes are notoriously complex.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. These tools help organizations implement DataOps practices by providing a unified platform for data teams to collaborate, share, and manage their data assets.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Data Engineering Weekly

JANUARY 15, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. 2023 predictions from the panel are; Unified metadata becomes kingmaker.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Ascend.io

DECEMBER 11, 2023

Data orchestration is the process of efficiently coordinating the movement and processing of data across multiple, disparate systems and services within a company. So, why is data orchestration a big deal? It automates and optimizes data processes, reducing manual effort and the likelihood of errors.

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

Databand.ai

AUGUST 30, 2023

Integrating these principles with data operation-specific requirements creates a more agile atmosphere that supports faster development cycles while maintaining high quality standards. Technical Challenges Choosing appropriate tools and technologies is critical for streamlining data workflows across the organization.

Data Governance 40

Data Governance Data Cleanse Government Data Pipeline 40

AltexSoft

JUNE 7, 2021

A HDFS Master Node, called a NameNode , keeps metadata with critical information about system files (like their names, locations, number of data blocks in the file, etc.) and keeps track of storage capacity, a volume of data being transferred, etc. Data management and monitoring options. Apache Hadoop ecosystem.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

Data Engineering Podcast

SEPTEMBER 18, 2022

In this episode he discusses the challenge of maintaining shared visibility and understanding of data work across the various stakeholders and his efforts to make it a seamless experience. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Building 100

Building Metadata MongoDB MySQL 100

AltexSoft

SEPTEMBER 21, 2023

The Elastic Stacks Elasticsearch is integral within analytics stacks, collaborating seamlessly with other tools developed by Elastic to manage the entire data workflow — from ingestion to visualization. Each document has unique metadata fields like index , type , and id that help identify its storage location and nature.

Engineering 52

Engineering NoSQL Programming Language Java 52

Databand.ai

DECEMBER 29, 2022

He believes in making data do the work through proper data management based on strategic rationale and business alignment. On LinkedIn, he posts frequently about big data, master data, data science, data management, and data storytelling.

BI 52

BI Consulting Data Science Data Governance 52

AltexSoft

MARCH 14, 2023

This development has paved the way for a suite of cloud-native data tools that are user-friendly, scalable, and affordable. Known as the Modern Data Stack (MDS) , this suite of tools and technologies has transformed how businesses approach data management and analysis.

IT 59

IT Data Warehouse Data Governance Data Lake 59