Data Engineering Weekly

OCTOBER 23, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Hello Data Friends, Welcome to another edition of Data Engineering Weekly.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Data Engineering Weekly

OCTOBER 30, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? Data science as a discipline was going through its adolescence of self-affirming and defining itself. We were pioneers.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Jesse Anderson

NOVEMBER 14, 2023

A Guest Post by Ole Olesen-Bagneux In this blog post I would like to describe a new data team, that I call ‘the data discovery team’. In an enterprise data reality, searching for data is a bit of a hassle. 1) The data discovery team must work on discovering the IT landscape.

Metadata 147

Metadata Data Science Big Data Data 147

Big Data Tools

JUNE 8, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

JUNE 8, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Knowledge Hut

SEPTEMBER 29, 2023

This growth is creating a strong demand for data experts, especially Azure data engineers. But who are Azure data engineers, and what do they do? Moreover, what benefits can you expect from a career in Azure Data Engineering? Why Should You Get an Azure Data Engineer Certification?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Weekly

NOVEMBER 6, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. I plan to write a series of blogs on Schemata and Data Contract in the coming weeks.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

FEBRUARY 26, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Are you tired of dealing with the headache that is the 'Modern Data Stack'? It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. We feel your pain.

Building 147

Building Metadata Machine Learning Data Integration 147

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode founders Barr Moses and Lior Gavish rejoin the show to reflect on the evolution and adoption of data observability technologies and the capabilities that are being introduced as the broader ecosystem adopts the practices. Atlan is the metadata hub for your data ecosystem. What is your take?

IT 100

IT Metadata MongoDB MySQL 100

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 90

Data Process Process Metadata Finance 90

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Your other engines that may be writing to the table, such as Apache Spark or Apache Flink, can continue to write, and Snowflake can read.

Building 90

Building Metadata Cloud Storage AWS 90

Workfall

JUNE 12, 2023

In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Data, the lifeblood of organisations, holds the key to unlocking untapped potential and propelling businesses forward. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Weekly

MAY 16, 2023

In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive? I won’t bore you with the importance of data quality in the blog. Event Routers can add additional metadata to the envelope of the event. How to Fix It?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Podcast

JANUARY 23, 2022

Summary Data engineering is a relatively young and rapidly expanding field, with practitioners having a wide array of experiences as they navigate their careers. Ashish Mrig currently leads the data analytics platform for Wayfair, as well as running a local data engineering meetup.

Building 100

Building Management Data Pipeline Metadata 100

Edureka

MAY 6, 2024

It involves collaboration between data scientists, ML engineers, and IT professionals to automate and optimize the end-to-end process of building, deploying, and maintaining machine learning applications. But before we begin, Let’s have a look at what we will be covering in this blog: What is MLOPS? Why do we need MLOPS?

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Christophe Blefari

DECEMBER 3, 2022

Joe Reis launched his Substack — Joe is the co-author of the great The Fundamentals of Data Engineering and his blog already have 2 articles I deeply recommend: No extra credit for complexity & Groundhog Days. This post tries to show how data team should work with finance team to avoid this spat.

Kafka 130

Kafka BI Finance Data 130

Knowledge Hut

MAY 9, 2024

From Silicon Valley to Wall Street, from healthcare to e-commerce, data scientists are highly valued and well-compensated in various industries and sectors. According to Glassdoor, the average annual pay of a data scientist is USD 126,683. What is Data Science? They manage data storage and the ETL process.

Data Science 52

Data Science Data Mining Data Architect Programming Language 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Lyft Engineering

OCTOBER 3, 2023

It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. As such, we have reached a point where the quantity of data is no longer a boundary. Yet this has come at the cost of quality.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Data Engineering Podcast

MARCH 22, 2021

In this episode Raghu Murthy, founder and CEO of Datacoral, does a deep dive on how he and his team manage change data capture pipelines in production. e.g. APIs and third party data sources How can we integrage CDC into metadata/lineage tooling? How can we build data quality checks into CDC workflows?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Cloudyard

APRIL 4, 2024

Read Time: 2 Minute, 13 Second Consider a scenario, you are part of data engineering team at a retail company. However, the volume of daily transaction data poses challenges in effectively segmenting customers and optimizing engagement. Traditional segmentation methods fall short in scalability and real-time insights.

Retail 40

Retail Data Ingestion Metadata Datasets 40

AltexSoft

FEBRUARY 11, 2023

If you are not familiar with the above-mentioned concepts, we suggest you to follow the links above to learn more about each of them in our blog posts. Bad data management be like, Source: Makeameme Data architects are sometimes confused with other roles inside the data science team. Feel free to enjoy it.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Cloudera

JULY 14, 2023

Depending on the size and usage patterns of the data, several different strategies could be pursued to achieve a successful migration. In this blog, I will describe a few strategies one could undertake for various use cases. This will be discussed in a later blog. Relatively fast as the underlying data files are kept in place.

Metadata 70

Metadata Data Warehouse Big Data Ecosystem Java 70

phData: Data Engineering

DECEMBER 16, 2021

Welcome to part two of our trilogy on data catalogs. If you missed our first blog on what a data catalog is , be sure to check it out! In this blog, we’ll explore what the ideal team to support your data catalog looks like. Who Are the Users of a Data Catalog? How will Human-Driven Metadata be Captured?

Metadata 52

Metadata Datasets Data Education 52

Data Engineering Podcast

JUNE 29, 2020

This was a great conversation about the complexities of working in a niche domain of data analysis and how to build a pipeline of high quality data from collection to analysis. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Collection 100

Data Collection Management High Quality Data Metadata 100

Databand.ai

SEPTEMBER 12, 2022

Unfortunately, most data engineers don’t realize the schema has changed until someone else downstream tells them. Databand helps fix this problem by capturing the metadata from your datasets and then alerting you when dataset operations change unexpectedly. Yeah, we hear this question a lot too.

Datasets 52

Datasets Metadata Data Engineering Data Engineer 52

Data Council

MARCH 25, 2021

How to Build a Community (Fishtown Analytics) Claire Carroll's first personal blog post on community-building is a must-read. Here's our March 2021 roundup of links from across the web that we selected for you: 1. As Fishtown Analytics' community manager for the last 2.5

Metadata 52

Metadata Management Building IT 52

Cloudera

FEBRUARY 14, 2024

It offers several benefits such as schema evolution, hidden partitioning, time travel, and more that improve the productivity of data engineers and data analysts. This blog discusses a few problems that you might encounter with Iceberg tables and offers strategies on how to optimize them in each of those scenarios.

Bytes 57

Bytes Metadata Data Lake SQL 57

Data Engineering Podcast

AUGUST 5, 2019

Announcements Welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Cloudera

DECEMBER 11, 2020

In a previous blog post on CDW performance, we compared Azure HDInsight to CDW. In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to EMR 6.0 (also powered by Apache Hive-LLAP) on Amazon using the TPC-DS 2.9 More on this later in the blog.

Data Warehouse 107

Data Warehouse Metadata Datasets Data 107

Cloudera

MARCH 17, 2023

I took the free version of ChatGPT on a test drive (in March 2023) and asked some simple questions on data lakehouse and its components. Hopefully this blog will give ChatGPT an opportunity to learn and correct itself while counting towards my 2023 contribution to social good. I thought this was a fairly comprehensive list.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Workfall

AUGUST 14, 2023

These tools, when combined, form a dynamic duo that streamlines and scales data engineering workflows. In this blog, we will explore the benefits of using DBT and Airflow together and how they make data pipeline management easier than ever before. In this blog, we will cover: Understanding DBT and Airflow What is DBT?

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Cloudera

SEPTEMBER 29, 2020

Performance is one of the key, if not the most important deciding criterion, in choosing a Cloud Data Warehouse service. In today’s fast changing world, enterprises have to make data driven decisions quickly and for that they rely heavily on their data warehouse service. . benchmark. Figure 2 – TPC-DS per query speedup.

Data Warehouse 117

Data Warehouse Cloud Storage Metadata Cloud 117

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.

Metadata 130

Metadata MongoDB Scala MySQL 130

phData: Data Engineering

SEPTEMBER 19, 2023

By employing robust data modeling techniques, businesses can unlock the true value of their data lake and transform it into a strategic asset. With many data modeling methodologies and processes available, choosing the right approach can be daunting. Want to learn more about data governance?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52