Data Engineering Podcast

FEBRUARY 20, 2022

He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers. You can observe your pipelines with built in metadata search and column level lineage.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 85

Systems Metadata Media Kafka 85

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. This rule empowers microservices to change their database schemas however they like without worrying about breaking other systems. What are Events?

Kafka 52

Kafka ETL Tools BI Database 52

Data Engineering Podcast

APRIL 24, 2022

WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Machine Learning 100

Machine Learning Systems Data Lake Metadata 100

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. An OS snapshot is a collection of bootfiles (initrd, vmlinuz), RPMs, and a few extra metadata.

Systems 55

Systems Metadata Datasets Designing 55

Netflix Tech

NOVEMBER 14, 2023

Late-arriving data is essentially delayed data due to system retries, network delays, batch processing schedules, system outages, delayed upstream workflows, or reconciliation in source systems. Using fixed lookback windows to always reprocess data, assuming that most late-arriving events will occur within that window.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Data Engineering Podcast

JULY 5, 2021

SaaSGlue is a managed service that lets you connect all of your systems, across clouds and physical infrastructure, and spanning all of your programming languages. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

Systems 100

Systems Management Data Warehouse Programming Language 100

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. We used Elasticsearch dashboards to analyze results.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Atlan is the metadata hub for your data ecosystem. What are some of the unique characteristics of that information?

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

JUNE 17, 2023

In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible for downstream use cases. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Data Lake 130

Data Lake Metadata Data Architecture Machine Learning 130

Knowledge Hut

JANUARY 2, 2024

You will learn how to build up Kube-state-metrics system, pull and collect metrics, deploy a Prometheus server and metrics exporters, configure alerts with Alertmanager, and create Grafana dashboards. Simply put, every item in a Kubernetes Prometheus store is a metric event that comes with a timestamp.

Architecture 98

Architecture Metadata Utilities Data Collection 98

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. As data pipelines sprawl across disparate systems, it becomes harder to trace data lineage, track data quality, and quickly pinpoint issues when they arise.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents.

Transportation 81

Transportation Data Governance Government Consulting 81

Rockset

MARCH 18, 2024

For instance, GPT-4’s cutoff date was April 2023, so it would not be aware of any events or developments happening beyond that point of time. Real-time metadata filtering Streaming data on products in a catalog is used to generate vector embeddings as well as provide additional contextual information. What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Podcast

MARCH 18, 2023

He also explains why data security is distinct from application security and some methods for reducing the challenge of working across different data systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Lyft Engineering

OCTOBER 3, 2023

For example, we can almost instantly validate that each record is well-formed and complete during event generation. Our Analytic Event Lifecycle below demonstrates the workflow of how much of our data gets to Hive. We log these events asynchronously at the order of millions per second.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. Yeah…I can do that already!

Process 80

Process Kafka Datasets SQL 80

Data Engineering Podcast

MARCH 27, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms.

Data Governance 130

Data Governance Government Cloud Building 130

Workfall

JUNE 12, 2023

It bridges gaps in data ecosystems, ensuring consistency and synchronisation across systems. Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Podcast

JULY 10, 2022

They also discuss how they have established a guild system for training and supporting data professionals in the organization. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

LinkedIn Engineering

DECEMBER 19, 2023

The framework incorporates proactive measures to monitor critical events, provides deployment insights, and issues proper notifications to enable seamless deployments. The system maintains a robust deployment history, seamlessly integrating with tools like version drift to detect deployed version differences.

Big Data 92

Big Data Hadoop Metadata Data 92

Data Engineering Podcast

SEPTEMBER 17, 2023

Linked data technologies provide a means of tightly coupling metadata with raw information. what are the characteristics that distinguish a knowledge graph from What are the layers/stages of applications and data that can/should incorporate JSON-LD as the representation for records and events?

Building 189

Building BI SQL Python 189

Towards Data Science

FEBRUARY 9, 2024

Eventing In many cases, the data platform itself needs to be responsible for, or at the very least inform, the generation of first party data. I break down eventing into two categories: Change Data Capture  — CDC The basic gist of CDC is using your database’s CRUD commands as a stream of data itself.

Building 81

Building Transportation Data Lake Metadata 81

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. The system can fail or apply a circuit breaker in the pipeline if the data contract fails. Event Routers typically don’t alter the payload.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Podcast

FEBRUARY 27, 2022

Time-series data is time stamped so you can measure how a system is changing. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. What other tools/systems will need to integrate with it?

Designing 100

Designing Metadata Data Lake Relational Database 100

Data Engineering Podcast

SEPTEMBER 4, 2022

Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

IT 100

IT Metadata MongoDB MySQL 100

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

ProjectPro

FEBRUARY 8, 2023

Users can schedule ETL jobs, and they can also choose the events that will trigger them. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. AWS Glue then creates data profiles in the catalog, a repository for all data assets' metadata, including table definitions, locations, and other features.

AWS 98

AWS Scala Metadata Data Lake 98

Monte Carlo

MARCH 12, 2024

In order to find and fix data incidents quickly, data observability solutions must provide visibility into the behavior of the data, systems, and code. That context for investigating and resolving these system failures is now much richer, greatly accelerating the resolution process. How Does It Work?

BI 52

BI Metadata Data Pipeline Data Engineering 52

Snowflake

AUGUST 31, 2023

Because of its leading ecosystem of diverse adopters, contributors and commercial offerings, Iceberg helps prevent storage lock-in and eliminates the need to move or copy tables between different systems, which often translates to lower compute and storage costs for your overall data stack.

Metadata 81

Metadata AWS Data Lake Datasets 81

Striim

FEBRUARY 5, 2024

It is not a standalone product but comprises key elements, including data integration, ensuring the smooth merging of data; data quality, maintaining high data standards; metadata management, organizing and understanding data context; and security, safeguarding data integrity.

Pharmaceutical 52

Pharmaceutical Data Cleanse Metadata Medical 52

DataKitchen

OCTOBER 20, 2023

.’ It assigns unique identifiers to each data item—referred to as ‘payloads’—related to each event. This transparent system effectively answers real-time data location and status questions, thus enhancing customer trust and satisfaction.

Insurance 70

Insurance Pharmaceutical Data Data Ingestion 70

Data Engineering Podcast

MARCH 22, 2021

For analytical systems, the only way to provide this reliably is by implementing change data capture (CDC). Unfortunately, this is a non-trivial undertaking, particularly for teams that don’t have extensive experience working with streaming data and complex distributed systems. What are the alternatives to CDC?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100