Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Atlan is the metadata hub for your data ecosystem.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Christophe Blefari

JUNE 16, 2023

The power of pre-commit and SQLFluff —SQL is a query programming language used to retrieve information from data storages, and like any other programming language, you need to enforce checks at all times. Malloy's Near Term Roadmap — I've shared recently Malloy demo , which was awesome. but I missed it).

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Precisely

MAY 31, 2023

Users cannot find the data they need, gain access in a timely fashion, rely on data being in a useable format, or understand its impact on the business – meaning that it is unusable to them. The end result is that users can’t trust their data because it lacks integrity. In many cases, that means real-time availability.

Data Integration 52

Data Integration Metadata Data Governance Data 52

Data Engineering Podcast

SEPTEMBER 23, 2018

He also explains which layers are useful for the different members of the business, and which pitfalls to look out for along the path to a mature and flexible data platform. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Monte Carlo

APRIL 24, 2023

Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption. Databricks Data Catalog and AWS Lake Formation are examples in this vein. AWS is one of the most popular data lake vendors.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Ascend.io

MAY 23, 2023

Orphaned datasets occur due to several reasons, such as changes in data relationships, ad-hoc archiving of deleted records, restarts from system failures and errors, or design flaws in data management strategies. This strategy minimizes clutter and storage space, but could lose some data that is no longer useful going forward.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Cloudera

AUGUST 31, 2021

Running on CDW is fully integrated with streaming, data engineering, and machine learning analytics. It has a consistent framework that secures and provides governance for all data and metadata on private clouds, multiple public clouds, or hybrid clouds. Consideration of both data & metadata in the migration.

Data Warehouse 81

Data Warehouse Database-centric Metadata Cloud 81

phData: Data Engineering

DECEMBER 16, 2021

As a result, it is important to know what is happening to your data. Is your data just being scanned, or is it being moved? Where is your metadata stored? How easy is it to offboard your data if you choose another tool? It can let the user know about the quality and age of the data as well as any policies attached to it.

Metadata 52

Metadata Datasets ETL Tools Cloud 52

Data Engineering Podcast

JUNE 26, 2022

In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Atlan is the metadata hub for your data ecosystem. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

MAY 18, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Monte Carlo

MAY 25, 2023

That’s why we’re excited to announce that uniting diverse data team personas to collectively ensure data quality just got easier, thanks to a new integration between Monte Carlo’s data observability platform and Atlan’s active metadata platform.

Metadata 52

Metadata Data Governance Government Business Intelligence 52

The Modern Data Company

MARCH 7, 2023

A data taxonomy uses a system of unambiguous metadata terms (such as a filename or tags attached to a file) that allow an enterprise to classify a file or dataset into important business categories. How Does Taxonomy Aid Data Governance? Incompleteness – Data may be missing important attributes.

Data Governance 52

Data Governance Government Metadata Datasets 52

AltexSoft

MARCH 2, 2023

However, decentralized models may result in inconsistent and duplicate master data. There’s a centralized structure that provides a framework, which is then used by autonomous departments that own their data and metadata. Data management is the overall process of collecting, storing, organizing, maintaining, and using data.

Data Governance 52

Data Governance Government Healthcare Programming 52

The Modern Data Company

JANUARY 17, 2023

These include internal data lakes, apps (1st or 3rd party), web SDK, internal or 3rd-party databases, and even spreadsheets generated within the marketing department. Without lineage or governance metadata, the quality of this data is anyone’s guess. New data is added to the catalog automatically upon ingestion.

Metadata 59

Metadata Government Data Lake Data 59

Cloudera

JUNE 25, 2019

Cloudera shared a comprehensive overview and demonstration of the all-new Cloudera Data Platform (CDP). This industry-first enterprise data cloud offering will include cloud-native services for data warehousing, machine learning, streaming ingest, and operational data stores.

Cloud 87

Cloud Entertainment Government Machine Learning 87

Data Engineering Podcast

JANUARY 28, 2018

Your host is Tobias Macey and today I’m interviewing Danielle Robinson and Joe Hand about Dat Project, a distributed data sharing protocol for building applications of the future Interview Introduction How did you get involved in the area of data management? What is the Dat project and how did it get started?

Data 100

Data Project Electronics Data Management 100

Cloudera

MARCH 15, 2023

In this post I am going to set up a demo environment with a Spring Boot microservice and a streaming cluster using Cloudera Public Cloud. The Outbox Pattern The general idea behind this pattern is to have an “outbox” table in the service’s data store.

PostgreSQL 89

PostgreSQL Kafka Database Data 89

AltexSoft

OCTOBER 8, 2021

The bad news is, integrating data can become a tedious task, especially when done manually. Luckily, there are various data integration tools that support automation and provide a unified data view for more efficient data management. Data integration process. Data profiling and cleansing.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

Data Engineering Weekly

JULY 23, 2023

link] Piethein Strengholt: Data Management at Scale Data Mesh is a widely discussed and debated topic on the possibility of decentralized data management at scale. The author unpacks the federated design for data management in a phase-by-phase manner by adding more components as expanding.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

The Modern Data Company

JANUARY 22, 2024

The Modern Story: Navigating Complexity and Rethinking Data in The Business Landscape Enterprises face a data landscape marked by the proliferation of IoT-generated data, an influx of unstructured data, and a pervasive need for comprehensive data analytics.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

The Modern Data Company

JANUARY 22, 2024

The Modern Story: Navigating Complexity and Rethinking Data in The Business Landscape Enterprises face a data landscape marked by the proliferation of IoT-generated data, an influx of unstructured data, and a pervasive need for comprehensive data analytics.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

The Modern Data Company

MARCH 1, 2024

Data Products work similarly, offering pre-packaged, actionable data units tailored to specific business goals. We will delve into the world of Data Products, exploring their transformative potential for CDOs and data teams. Data Products simplify access and processing, empowering faster decision-making.

Database-centric 40

Database-centric Data Lake Government Retail 40

Monte Carlo

APRIL 20, 2023

I can surface ownership metadata and alert the relevant owners to make sure the appropriate changes are made so these breakages never happen. This way no decisions get made on bad data and our team becomes a proactive part of the solution,” said then Senior Director of Data at Freshly, Vitaly Lilich.

Data Warehouse 52

Data Warehouse BI Government Data 52

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Google Cloud 64