Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Data Engineering Podcast

JANUARY 23, 2022

He also provides some excellent insights into the factors that play into the build vs. buy decision at different organizational sizes. You can observe your pipelines with built in metadata search and column level lineage. You can observe your pipelines with built in metadata search and column level lineage.

Building 100

Building Management Data Pipeline Metadata 100

Databand.ai

AUGUST 30, 2023

Building a Winning Data Quality Strategy: Step by Step Eitan Chazbani August 30, 2023 What Is a Data Quality Strategy? This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. This starts with building a strong business case for your data quality strategy.

Building 52

Building Data Cleanse Data Governance Datasets 52

Data Engineering Podcast

FEBRUARY 19, 2023

It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

IT 147

IT Data Lake Metadata Data Warehouse 147

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 4, 2022

The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. Zing Data is building a mobile native platform for business intelligence. In this episode Sabin Thomas shares his experiences building the platform and the interesting ways that it is being used.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

François Nguyen

MARCH 22, 2021

How do we build data products ? How to build Data Products or never call me Data Pipeline any more You have this interesting schema in her second article on Data Mesh by Zhamak Dehghani : “Data mesh introduces the concept of data product as its architectural quantum. .” What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Towards Data Science

FEBRUARY 6, 2024

Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Accessibility : I could easily request access to these data products. Additionally, it is really hard to identify the owners, one of them has even recently left the company.

Systems 83

Systems Raw Data Metadata Data Cleanse 83

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

LinkedIn Engineering

MARCH 21, 2023

That’s why we believe a skills-first approach to hiring talent will help companies gain access to wider talent pools to find the skilled workers their businesses need, especially in sectors that are aggressively looking for talent. What is the skills taxonomy?

Building 85

Building Recruitment Machine Learning Deep Learning 85

Data Engineering Podcast

AUGUST 21, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Data Engineering Podcast

JULY 31, 2022

In this episode Ernie Ostic shares the approach that he and his team at Manta are taking to build a complete view of data lineage across the various data systems in your organization and the useful applications of that information in the work of every data stakeholder. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

LinkedIn Engineering

JUNE 2, 2023

This data must be accessible and available in near real-time in order to accurately reflect current trends and conditions. Discoverability: Consumers must be able to easily discover and access the correct data sources for their needs, whether it is stored in a centralized repository or scattered throughout the wider data landscape.

Building 75

Building Banking Datasets Media 75

Towards Data Science

MAY 7, 2024

meetings with sales reps from all of the SaaS data integration tooling companies and are granted 14 day access to try their wares. This will help create an overall picture of logging, metadata, latency, CDC, and all the other things that come with a pipeline. Gather the team together and determine what these things are.

ETL Tools 84

ETL Tools Metadata Data Integration Data Science 84

Data Engineering Podcast

AUGUST 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata MongoDB MySQL Business Intelligence 100

Data Engineering Podcast

MARCH 18, 2023

Data Council attendees are amazing founders, data scientists, lead engineers, CTOs, heads of data, investors and community organizers who are all working together to build the future of data. Visit: dataengineeringpodcast.com/data-council today RudderStack makes it easy for data teams to build a customer data platform on their own warehouse.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents. These features further enhance the protection of customers’ sensitive data.

Transportation 83

Transportation Data Governance Government Consulting 83

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. A data observability platform?

IT 100

IT Metadata MongoDB MySQL 100

Precisely

JUNE 6, 2023

Atif Salam noted that as recently as a year ago, the primary focus in many organizations was on ingesting data and building data lakes. Building on that thought, Sanjeev Mohan noted that the building blocks of modern data architecture remain more or less the same, but today’s businesses are focus on optimization.

Data Architecture 52

Data Architecture Architecture Metadata Data Lake 52

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. It also included metadata about ads, such as ad placement and impression-tracking events. As we were gearing up for launch, we wanted to ensure it would go as smoothly as possible.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

ProjectPro

FEBRUARY 25, 2022

Efficient Scheduling and Runtime Increased Adaptability and Scope Faster Analysis and Real-Time Prediction Introduction to the Machine Learning Pipeline Architecture How to Build an End-to-End a Machine Learning Pipeline? This makes it easier for machine learning pipelines to fit into any model-building application.

Machine Learning 52

Machine Learning Building Amazon Web Services AWS 52

Data Engineering Weekly

MARCH 24, 2024

al6z: 16 Changes to the Way Enterprises Are Building and Buying Generative AI This report has a lot of interesting insight into the enterprise adoption of Gen AI. The APIs support emitting unstructured log lines and typed metadata key-value pairs (per line). The extracted key-value pairs are written to the line’s metadata.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Edureka

MAY 6, 2024

However, building and deploying these models is just the beginning. It involves collaboration between data scientists, ML engineers, and IT professionals to automate and optimize the end-to-end process of building, deploying, and maintaining machine learning applications. Why do we need MLOps?

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Monte Carlo

MAY 18, 2023

You may even already have access to one of them. We’ve written extensively on how to analyze the build vs buy decision across the modern data stack, but it’s important to speak specifically about open source data lineage options. The Spark data lake lineage example is helpful for context here. Ok, that is enough caveat emptor.

Metadata 64

Metadata Data Lake Government Data Governance 64

Data Engineering Weekly

DECEMBER 29, 2022

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. There are not many sources to pull the metadata. The modern(?)

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Rockset

APRIL 18, 2023

Join me and Rockset VP of Engineering Louis Brandy for a tech talk, From Spam Fighting at Facebook to Vector Search at Rockset: How to Build Real-Time Machine Learning at Scale , on May 17th at 9am PT/ 12pm ET. Fast Search: Combine vector search and selective metadata filtering to deliver fast, efficient results. Why use vector search?

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). The data is protected but it is also accessible by the people who need it.

Government 79

Government Metadata Data Lake Data 79

Precisely

MAY 31, 2023

When attempting to build a data strategy, the primary obstacle organizations face is a lack of resources. Teams are building complex, hybrid, multi-cloud environments, moving critical data workloads to the cloud, and addressing data quality challenges. The end result is that users can’t trust their data because it lacks integrity.

Data Integration 52

Data Integration Metadata Data Governance Data 52

Netflix Tech

MARCH 5, 2019

While our engineering teams have and continue to build solutions to lighten this cognitive load (better guardrails, improved tooling, …), data and its derived products are critical elements to understanding, optimizing and abstracting our infrastructure. How can we automatically provision or de-provision access privileges?

Cloud 73

Cloud Building Amazon Web Services Metadata 73

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Data engineers build the systems that store and process sensitive information.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Monte Carlo

FEBRUARY 21, 2023

In this post, we’ll discuss what, exactly, a data fabric is, how other companies have used it, and how you can build one at your company. It’s a very good template to actually think through how to build some of these [governance standards].” Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

In this post, we’ll discuss what, exactly, a data fabric is, how other companies have used it, and how you can build one at your company. It’s a very good template to actually think through how to build some of these [governance standards].” Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Towards Data Science

DECEMBER 1, 2023

However, we will pay particular attention to Access Control (this will be used for dbt access). After the initial sync, you can access your data from the Snowflake UI. There are two ways to access dbt: dbt Cloud and dbt Core. Choose your preferred method and set up the Google Calendar connector.

Data Engineering 73

Data Engineering Data Engineer Project Engineering 73

Data Engineering Weekly

JUNE 29, 2023

The quest to simplify data access is there forever, but with the advancement in LLM, I think it will become a reality. Databricks and Snowflake are better places to index the data and its metadata to enable natural language query capabilities. On top of it, it does support access control for queries and maintains the permission model.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76