Data Engineering Podcast

SEPTEMBER 23, 2018

Summary As your data needs scale across an organization the need for a carefully considered approach to collection, storage, organization, and access becomes increasingly critical. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Silectis

FEBRUARY 3, 2021

Below we’ll cover the basics of data lineage, why it is important, and how Magpie enables teams to trust their data with this important new release. What is Data Lineage? Data lineage refers to the entire lifecycle of a dataset from its sources of origin all the way to its current state.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

The Modern Data Company

JANUARY 17, 2023

Companies Can Finally End Marketing Silos CDPs tend to be siloed in marketing departments — data comes through marketing channels, and there is very little connection with the company’s larger data ecosystem. CDP data remains cut off from data in other departments that could enrich the customer picture.

Metadata 59

Metadata Government Data Lake Data 59

Ascend.io

MAY 23, 2023

Already in 2016, IBM estimated the cost of bad data to be over three trillion dollars, and that was before the chaos of data lakes emerged and orphaned datasets began to swamp the land. This strategy minimizes clutter and storage space, but could lose some data that is no longer useful going forward.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Cloudera

AUGUST 4, 2021

Today, modern data governance programs must address not only an exponential increase in data volumes, but also cloud, hybrid architectures, and more reliance on advanced analytics. Trends toward more self-service data access and real-time applications have made data governance more visible and critical than ever before.

Data Governance 85

Data Governance Government Metadata Data Security 85

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Google Cloud 64

AltexSoft

NOVEMBER 7, 2022

Metadata database. A metadata database stores information about user permissions, past and current DAG and task runs, DAG configurations, and more. By default, Airflow handles metadata with SQLite which is meant for development only. Full REST API: easy access for third parties. Since the 2.0

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

AltexSoft

OCTOBER 8, 2021

They are applied to retrieve data from the source systems, perform transformations when necessary, and load it into a target system ( data mart , data warehouse, or data lake). So, why is data integration such a big deal? Oracle Data Integrator, IBM InfoSphere, Snaplogic, Xplenty, and.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52