Data Engineering Podcast

FEBRUARY 20, 2022

You can observe your pipelines with built in metadata search and column level lineage. Your host is Tobias Macey and today I’m interviewing Guy Yachdav, Director of Software Engineering at Immunai, about his work at Immunai to wrangle biological data for advancing research into the human immune system. regulatory, security, etc.)

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 85

Systems Metadata Media Kafka 85

Netflix Tech

MARCH 10, 2021

ConsoleMe: A Central Control Plane for AWS Permissions and Access By Curtis Castrapel , Patrick Sanders , and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. Groups beyond software engineering teams are standing up their own systems and automation.

AWS 100

AWS Accessible Accessibility Cloud 100

Data Engineering Podcast

DECEMBER 4, 2022

The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. This opens the door for busy employees to access and analyze their company information away from their desk, but it has the more powerful effect of bringing first-class support to companies operating in mobile-first economies.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents. These features further enhance the protection of customers’ sensitive data.

Transportation 79

Transportation Data Governance Government Consulting 79

Data Engineering Podcast

JULY 31, 2022

Because of its centrality to your data systems it is valuable for debugging, governance, understanding context, and myriad other purposes. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 30, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Many conversations around data and analytics are focused on self-service access. Atlan is the metadata hub for your data ecosystem.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Weekly

MARCH 24, 2024

The author goes beyond comparing the tools to various offerings from streaming vendors in stream processing and Kafka protocol-supported systems. The logging engine to debug AI workflow logs is an excellent system design study if you’re interested in it. The extracted key-value pairs are written to the line’s metadata.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Knowledge Hut

MARCH 7, 2024

Syncing Across Data Sources Once you import data into Big Data platforms you may also realize that data copies migrated from a wide range of sources on different rates and schedules can rapidly get out of the synchronization with the originating system. Two, it creates a commonality of data definitions, concepts, metadata and the like.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Data Engineering Podcast

AUGUST 6, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Can you describe the technical elements of that system? How does that activity map to data systems/products?

Metadata 100

Metadata MongoDB MySQL Business Intelligence 100

Edureka

MAY 6, 2024

MLOps is an emerging discipline that aims to unify and streamline the machine learning system development (Dev) and operations (Ops) lifecycle. MLOps refers to the practice and discipline within machine learning that aims to unify and streamline machine learning system development (Dev) and operations (Ops).

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Ascend.io

FEBRUARY 28, 2024

The foundation of any AI system is high-quality data. Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. And crucially, what does the future hold for data engineering in an AI-driven world?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Podcast

MARCH 27, 2022

You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. How have the design and goals of the system changed or evolved since you started working on it? Can you describe what Privacera is and the story behind it?

Data Governance 130

Data Governance Government Cloud Building 130

Knowledge Hut

APRIL 4, 2024

Chef Solo) Users write system configuration files that are called ‘Recipes’, which are then organized into ‘Cookbooks’. Nodes, or the systems needing the configuration, are the final part. Keeping the Chef repository in sync with the source code revision control system. Validating Chef's source code. Nodes can be of any type, i.e:

Architecture 52

Architecture Amazon Web Services Metadata AWS 52

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Weekly

DECEMBER 29, 2022

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. There are not many sources to pull the metadata. The modern(?)

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Monte Carlo

MARCH 4, 2024

Table of Contents How Data Observability Works Made For The Lakehouse: Iceberg and Delta Transform in place: Dynamic Tables and Delta Live Tables Why This Matters How Data Observability Works Data observability provides full visibility into your data, systems, and code so you can find and fix bad data quickly.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

MARCH 18, 2023

He also explains why data security is distinct from application security and some methods for reducing the challenge of working across different data systems. How has the scope and complexity of implementing security controls on data systems changed in recent years? Don't miss out on our only event this year!

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Monte Carlo

JANUARY 29, 2024

Snowflake recently announced the launch of Snowflake Horizon —a unified collection of compliance, security, privacy, interoperability, and access capabilities—that make it easy for organizations to govern and take action on their data and apps across clouds, teams, partners, and customers. Volume: Are there too many or too few rows?

Metadata 64

Metadata High Quality Data Data Pipeline Machine Learning 64

Cloudera

NOVEMBER 29, 2023

It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. You can track each iteration, compare changes, and ensure that you always have access to the version that suits your needs. Its development is geared towards easing the transition between different cloud systems.

Machine Learning 85

Machine Learning Metadata Cloud Project 85

LinkedIn Engineering

DECEMBER 19, 2023

The historical upgrade system couldn’t adapt to architectural changes like the introduction of an observer namenode (Now handling a massive influx of read requests – 150K QPS – from services such as Trino ), ZKFC auto-failover, HDFS federation , etc.

Big Data 92

Big Data Hadoop Metadata Data 92

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. This culminated in the creation of GenOS, an operating system for developing GenAI-powered features.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Cloudera

AUGUST 10, 2021

Apache Ozone enhancements deliver full High Availability providing customers with enterprise-grade object storage and compatibility with Hadoop Compatible File System and S3 API. . Impala Row Filtering to set access policies for rows when reading from a table. Figure 1: sales group SELECT access.

Cloud 97

Cloud Kafka Metadata SQL 97

Monte Carlo

MAY 18, 2023

You may even already have access to one of them. Data lineage tools help you uncover and map the complex relationships between data elements across various sources, systems, and processes. There are a few reasons for this, but it’s mainly because data lineage involves so many layers–system, code, users– that are constantly evolving.

Metadata 64

Metadata Data Lake Government Data Governance 64

phData: Data Engineering

SEPTEMBER 19, 2023

A data lake is a centralized repository containing extensive storage for raw, unfiltered data coming into a company’s data storage system. Data modeling methodologies are systematic approaches used to design and define the structure and relationships of data within a system. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudera

MARCH 3, 2022

Data analytics and machine learning can become a business and a compliance risk if data security, governance, lineage, metadata management, and automation are not holistically applied across the entire data lifecycle and all environments. One possible solution is to adopt a hybrid cloud strategy. .

Data Governance 97

Data Governance Government Metadata Medical 97

Rockset

APRIL 18, 2023

We’re excited to introduce vector search on Rockset to power fast and efficient search experiences, personalization engines, fraud detection systems and more. Vector search has seen an explosion in popularity due to improvements in accuracy and broadened accessibility to the models used to generate embeddings.

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Data Engineering Weekly

JUNE 29, 2023

The quest to simplify data access is there forever, but with the advancement in LLM, I think it will become a reality. Databricks and Snowflake are better places to index the data and its metadata to enable natural language query capabilities. On top of it, it does support access control for queries and maintains the permission model.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Knowledge Hut

APRIL 25, 2024

The overwhelming Internet traffic to the target or its surrounding infrastructure locks up the system and forces it to temporarily stay unavailable. Malware Malware is the short form for “malicious software”, describing a wide variety of malicious software or code used to infect and/or damage a system. It comes in all shapes and sizes.

Electronics 52

Electronics Utilities Banking Programming 52

Precisely

MAY 31, 2023

In the same way that big cloud-platform providers offer simplified access to infrastructure, and data cloud providers like Databricks and Snowflake have vastly simplified access to data and analytics, modern data integrity tools must streamline and automate data integrity processes.

Data Integration 52

Data Integration Metadata Data Governance Data 52

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 89

Datasets Metadata Algorithm Architecture 89

Cloudera

DECEMBER 10, 2020

Some data warehousing solutions such as appliances and engineered systems have attempted to overcome these problems, but with limited success. . In the above example, CDW could have cut costs by 18% for the duration that the system was idle. In a multi-tenant environment, many users access the same data sources. For example, .

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Knowledge Hut

JULY 24, 2023

In the world of databases, data independence plays a vital role in making sure the flexibility and adaptability of database systems. It allows changes to be made at various levels of a database system without causing disruptions or requiring extensive modifications to the applications that rely on the data.

Database Design 52

Database Design Relational Database Metadata Database 52