Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. You can observe your pipelines with built in metadata search and column level lineage.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Workfall

JUNE 12, 2023

It facilitates data synchronisation, replication, real-time analytics, and event-driven processing, empowering data-driven decision-making and operational efficiency. These additional columns store metadata like timestamps, user IDs, and change types, ensuring granular change tracking and auditability.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Podcast

JANUARY 30, 2022

You can observe your pipelines with built in metadata search and column level lineage. Finally, if you have existing workflows in AbInitio, Informatica or other ETL formats that you want to move to the cloud, you can import them automatically into Prophecy making them run productively on Spark.

Data Engineering 100

Data Engineering Data Engineer Engineering Python 100

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

APRIL 5, 2021

What portions of the data workflow is Atlan responsible for? What components of the data stack might Atlan replace? How would you characterize Atlan’s position in the current data ecosystem? What makes Atlan stand out from other systems for data cataloguing, metadata management, or data governance?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Weekly

MARCH 11, 2023

[link] Data Engineering Weekly Data Catalog - A Broken Promise Data catalogs are the most expensive data integration systems you never intended to build.

Data Engineering 52

Data Engineering Data Engineer Engineering High Quality Data 52

Data Engineering Podcast

AUGUST 5, 2019

Finding the data that you need is tricky, and Amundsen will help you solve that problem. And as your data grows in volume and complexity, there are foundational principles that you can follow to keep data workflows streamlined. Finding the data that you need is tricky, and Amundsen will help you solve that problem.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Monte Carlo

JANUARY 16, 2024

Factor in the advertising strategies, media production, partner programming, audience analytics…and you’re looking at an ocean of data that would fill even the deepest trench (we’d like a television show about that too, please!). So how does Fox’s data strategy support these complex data workflows?

Government 52

Government High Quality Data Data Governance Entertainment 52

Hepta Analytics

FEBRUARY 14, 2022

Disadvantages of a data lake are: Can easily become a data swamp data has no versioning Same data with incompatible schemas is a problem without versioning Has no metadata associated It is difficult to join the data Data warehouse stores processed data, mostly structured data.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Cloudera

FEBRUARY 11, 2019

Cloudera provides a unified platform with multiple data apps and tools, big data management, hybrid cloud deployment flexibility, admin tools for platform provisioning and control, and a shared data experience for centralized security, governance, and metadata management. 3. Expansion beyond core data management.

Data Management 43

Data Management Management Metadata Government 43

Knowledge Hut

SEPTEMBER 29, 2023

Why Should You Get an Azure Data Engineer Certification? Becoming an Azure data engineer allows you to seamlessly blend the roles of a data analyst and a data scientist. One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Cloudera

AUGUST 21, 2020

The governance aspect is perhaps even more important and businesses need to be able to understand where the data comes from. Data lineage, personally identifiable information or PPI and metadata all fall under a broad data governance banner which is critically important in terms of what needs to be protected and mapped out.

Banking 59

Banking Kafka Cloud Storage Government 59

Monte Carlo

SEPTEMBER 20, 2022

Here’s how Prefect , Series B startup and creator of the popular data orchestration tool, harnessed the power of data observability to preserve headcount, improve data quality and reduce time to detection and resolution for data incidents. But Monte Carlo doesn’t stop at the “most important” tables.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. Linked data technologies provide a means of tightly coupling metadata with raw information.

Building 189

Building BI SQL Python 189

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is the metadata hub for your data ecosystem. Missing data? Missing data?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

NOVEMBER 26, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Weekly

JANUARY 15, 2023

SiliconANGLE theCUBE: Analyst Predictions 2023 - The Future of Data Management By far one of the best analyses of trends in Data Management. 2023 predictions from the panel are; Unified metadata becomes kingmaker. The author walked through various strategies, from sync to async job submission and batch job submission strategy.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Databand.ai

AUGUST 30, 2023

DataOps tools should provide a comprehensive data cataloging solution that allows organizations to create a centralized repository of their data assets, complete with metadata, data lineage information, and data samples.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Ascend.io

DECEMBER 11, 2023

Data orchestration is the process of efficiently coordinating the movement and processing of data across multiple, disparate systems and services within a company. So, why is data orchestration a big deal? It automates and optimizes data processes, reducing manual effort and the likelihood of errors.

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

AltexSoft

NOVEMBER 7, 2022

DevOps tasks — for example, creating scheduled backups and restoring data from them. Airflow is especially useful for orchestrating Big Data workflows. Airflow is not a data processing tool by itself but rather an instrument to manage multiple components of data processing. Metadata database.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

dbt Developer Hub

OCTOBER 14, 2021

It uses the dbt Cloud Metadata API to surface metadata from dbt right in Hex, letting you quickly get the context you need on things like data freshness without juggling multiple apps and browser tabs. Hex just launched an integration with dbt! Get started here. Things to Watch ? What's missing? Spreadsheets?

Metadata 52

Metadata BI Software Engineer Software Engineering 52

Netflix Tech

NOVEMBER 20, 2023

This enables auto propagation of backfill data in multi-stage pipelines. Netflix Maestro Maestro is the Netflix data workflow orchestration platform built to meet the current and future needs of Netflix. As we know, an iceberg table contains a list of snapshots with a set of metadata data.

Process 84

Process Data Pipeline Datasets SQL 84

Databand.ai

AUGUST 30, 2023

Integrating these principles with data operation-specific requirements creates a more agile atmosphere that supports faster development cycles while maintaining high quality standards. Technical Challenges Choosing appropriate tools and technologies is critical for streamlining data workflows across the organization.

Data Governance 40

Data Governance Data Cleanse Government Data Pipeline 40

AltexSoft

JUNE 7, 2021

A HDFS Master Node, called a NameNode , keeps metadata with critical information about system files (like their names, locations, number of data blocks in the file, etc.) and keeps track of storage capacity, a volume of data being transferred, etc. Among solutions facilitation data management are. Apache Hadoop ecosystem.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

Netflix Tech

OCTOBER 27, 2020

Netflix Scheduler is built on top of Meson which is a general purpose workflow orchestration and scheduling framework to execute and manage the lifecycle of the data workflow. Bulldozer makes data warehouse tables more accessible to different microservices and reduces each individual team’s burden to build their own solutions.

Data Warehouse 97

Data Warehouse Datasets Data Big Data 97

Netflix Tech

OCTOBER 18, 2022

With the high growth of workflows in the past few years?—?increasing increasing at > 100% a year, the need for a scalable data workflow orchestrator has become paramount for Netflix’s business needs. A workflow instance is an execution of a workflow, similarly, an execution of a step is called a step instance.

Java 81

Java Data Machine Learning Systems 81

Ascend.io

MAY 19, 2023

The holistic approach of the post-modern data stack translates into numerous benefits: First, it accelerates pinpointing and troubleshooting pipeline hotspots with a single console that observes the entire data pipeline and all its processes. Second, it enhances governance and security. Lastly, it improves efficiency and scalability.

Data Pipeline 52

Data Pipeline Data Engineering Data Engineer Media 52

AltexSoft

SEPTEMBER 21, 2023

The Elastic Stacks Elasticsearch is integral within analytics stacks, collaborating seamlessly with other tools developed by Elastic to manage the entire data workflow — from ingestion to visualization. Each document has unique metadata fields like index , type , and id that help identify its storage location and nature.

Engineering 52

Engineering NoSQL Programming Language Java 52

Data Engineering Podcast

SEPTEMBER 18, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. What are the contributing factors that lead to fragmentation of visibility for data workflows at different stages?

Building 100

Building Metadata MongoDB MySQL 100

Monte Carlo

JUNE 14, 2023

Apache Spark – Labeled as a unified analytics engine for large scale data processing, many leverage this open source solution for streaming use cases, often in conjunction with Databricks. Data orchestration Airflow : Airflow is the most common data orchestrator used by data teams.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Ascend.io

MARCH 17, 2023

For data pipeline automation, these previous approaches have foundered on one important barrier: the need to scale. This can be done with unique digital fingerprints that map not just every snippet of data, but the data engineers’ code as well. Value Catching data problems in real-time avoids costly reruns and delays.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineer 52

Ascend.io

MARCH 17, 2023

For data pipeline automation, these previous approaches have foundered on one important barrier: the need to scale. This can be done with unique digital fingerprints that map not just every snippet of data, but the data engineers’ code as well. Value Catching data problems in real-time avoids costly reruns and delays.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineer 52

Databand.ai

DECEMBER 29, 2022

Follow Sudhir on LinkedIn 13) Benjamin Rogojan Data Science And Data Engineering Consultant at Acheron Analytics Benjamin is a data science and data engineering consultant with nearly a decade of experience working with companies like Healthentic, Facebook, and Acheron Analytics.

BI 52

BI Consulting Data Science Data Governance 52

AltexSoft

MARCH 14, 2023

Also, Databricks are pioneering the lakehouse concept that makes it possible to use data management features inherent in data warehousing on the raw data stored in a low-cost data lake owing to its metadata layer. Data orchestration involves managing the scheduling and execution of data workflows.

IT 59

IT Data Warehouse Data Governance Data Lake 59