Data Engineering Weekly

FEBRUARY 18, 2024

One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. See how recent product updates make it easier for developers and data engineers to debug data models, work in a BI-like UI, and fine-tune deployments for better query performance. Was this simply too ambitious?

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

dbt Developer Hub

SEPTEMBER 14, 2021

Give Jeremy a win and check out the blog he just posted on why this matters even more leading up to ?dbt Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! Have you subscribed to the Analytics Engineering Roundup ? Anna's latest reflection? Things to Listen To ? ​

Metadata 52

Metadata Data Pipeline Python Cloud 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. So, let's shape the future of Data Engineering together. See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 97

Building Metadata Cloud Storage AWS 97

Data Engineering Weekly

MAY 16, 2023

I won’t bore you with the importance of data quality in the blog. Event Routers can add additional metadata to the envelope of the event. The same goes for the popular orchestration engine Airflow, where the Airflow has separate operators ( SQLCheckOperators ) for the data quality check. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Weekly

OCTOBER 23, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Hello Data Friends, Welcome to another edition of Data Engineering Weekly.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Data Engineering Weekly

OCTOBER 30, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Til about [link] Thank you, Anton Bryzgalov, for contributing the blog to the Data Engineering Weekly.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. But unfortunately, very little that happens in data engineering today is observable. Most data pipelines are built to move but not monitor.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? At the same time, data engineering was the slightly younger sibling, but it was going through something similar. I wasn’t promoted or assigned to this new role.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Big Data Tools

JUNE 8, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

JUNE 8, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 90

Data Process Process Metadata Finance 90

Edureka

MAY 6, 2024

It involves collaboration between data scientists, ML engineers, and IT professionals to automate and optimize the end-to-end process of building, deploying, and maintaining machine learning applications. But before we begin, Let’s have a look at what we will be covering in this blog: What is MLOPS? Why do we need MLOPS?

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Data Engineering Weekly

NOVEMBER 6, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. I plan to write a series of blogs on Schemata and Data Contract in the coming weeks.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Data Engineering Podcast

FEBRUARY 26, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Are you tired of dealing with the headache that is the 'Modern Data Stack'? Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Building 147

Building Metadata Machine Learning Data Integration 147

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. What is your take?

IT 100

IT Metadata MongoDB MySQL 100

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Knowledge Hut

SEPTEMBER 29, 2023

This growth is creating a strong demand for data experts, especially Azure data engineers. But who are Azure data engineers, and what do they do? Moreover, what benefits can you expect from a career in Azure Data Engineering? Join us on this journey through the exciting realm of Azure Data Engineering.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. The transformation step takes place in a separate engine and the transformed data is stored in a data store. This was used to test our setup. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Workfall

JUNE 12, 2023

In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Enter Change Data Capture (CDC), a transformative concept that seamlessly weaves its way into the very fabric of data engineering. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Cloudera

SEPTEMBER 24, 2020

However, there is a secret I am keeping to the end of the blog, which makes the decision even easier for the user: so easy in fact, you do not even have to decide yourself. Written in C++, which is very CPU efficient, with a very fast query planner and metadata caching, Impala is optimized for low latency queries. So, why choose?

Data Warehouse 68

Data Warehouse SQL Engineering Metadata 68

Netflix Tech

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. and based on content characteristics and/or metadata signaled in the bitstream. A vital aspect of such development is subjective testing with HDR encodes in order to generate training data. Krasula, A. Choudhury, S. Malfait, A.

Metadata 100

Metadata Electronics Algorithm Software Engineer 100

Lyft Engineering

SEPTEMBER 6, 2023

Routing overrides metadata: embed metadata in API request headers defining which offloaded deployment the request will get routed to. Context propagation : routing overrides metadata is propagated throughout the service mesh. Routing overrides metadata with ProxyApp Staging overrides was well-received by our product engineers.

Building 80

Building Metadata Electronics Engineering 80

Pinterest Engineering

NOVEMBER 7, 2023

Jeff Xiang | Software Engineer, Logging Platform Vahid Hashemian | Software Engineer, Logging Platform Jesus Zuniga | Software Engineer, Logging Platform At Pinterest, data is ingested and transported at petabyte scale every day, bringing inspiration for our users to create a life they love.

Kafka 98

Kafka Java Software Engineer Software Engineering 98

Lyft Engineering

OCTOBER 3, 2023

This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Christophe Blefari

DECEMBER 3, 2022

Joe Reis launched his Substack — Joe is the co-author of the great The Fundamentals of Data Engineering and his blog already have 2 articles I deeply recommend: No extra credit for complexity & Groundhog Days. This time Apple engineers shared at Trino—good old Presto for lost people—Summit how they use it.

Kafka 130

Kafka BI Finance Data 130

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Knowledge Hut

MAY 9, 2024

In this blog post, we will look at some of the world's highest paying data science jobs, what they entail, and what skills and experience you need to land them. Responsibilities Responsibilities of data modelers include validating data models, evaluating existing systems, ensuring data consistency, and optimizing metadata.

Data Science 52

Data Science Data Mining Data Architect Programming Language 52

Data Council

MARCH 25, 2021

How to Build a Community (Fishtown Analytics) Claire Carroll's first personal blog post on community-building is a must-read. Here's our March 2021 roundup of links from across the web that we selected for you: 1. As Fishtown Analytics' community manager for the last 2.5

Metadata 52

Metadata Management Building IT 52

Data Engineering Podcast

JANUARY 23, 2022

Summary Data engineering is a relatively young and rapidly expanding field, with practitioners having a wide array of experiences as they navigate their careers. Ashish Mrig currently leads the data analytics platform for Wayfair, as well as running a local data engineering meetup.

Building 100

Building Management Data Pipeline Metadata 100

Cloudera

AUGUST 10, 2021

We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Figure 8: Data lineage based on Kafka Atlas Hook metadata. x, and 6.3.x,

Cloud 96

Cloud Kafka Metadata SQL 96

Cloudera

JULY 14, 2023

In this blog, I will describe a few strategies one could undertake for various use cases. While these instructions are carried out for Cloudera Data Platform (CDP), Cloudera Data Engineering, and Cloudera Data Warehouse, one can extrapolate them easily to other services and other use cases as well.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The CSP engine is powered by Apache Flink, which is the best-in-class processing engine for stateful streaming pipelines.

Process 112

Process SQL Kafka Database 112

Cloudyard

APRIL 4, 2024

Read Time: 2 Minute, 13 Second Consider a scenario, you are part of data engineering team at a retail company. This blog post explores how Snowpark, a powerful tool for data processing within Snowflake, can be used to perform RFM segmentation and unlock actionable customer insights. Frequency: How often a customer makes purchases.

Retail 40

Retail Data Ingestion Metadata Datasets 40

Lyft Engineering

JUNE 28, 2023

While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort. In this blog post, we will discuss what we built in support of that goal and some of the lessons we learned along the way.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Data Engineering Podcast

MARCH 22, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. How do you handle observability of CDC flows?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Cloudera

FEBRUARY 14, 2024

It offers several benefits such as schema evolution, hidden partitioning, time travel, and more that improve the productivity of data engineers and data analysts. This blog discusses a few problems that you might encounter with Iceberg tables and offers strategies on how to optimize them in each of those scenarios.

Bytes 56

Bytes Metadata Data Lake SQL 56