Jesse Anderson

JANUARY 29, 2024

The premier of my latest talk covering The State of Data Engineering. This starts with data warehousing and goes into data science. I finish off by showing how data engineering can avoid the same fate as data warehousing and data science.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Science 130

Data Engineering Weekly

APRIL 14, 2024

link] Arpit Choudhury: Growth needs to speak the language of Engineering and Data Software engineers and data engineers don’t speak the same language It reminds me of this famous quote The Author highlights that the growth team should be aware of the skill differences.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Waitingforcode

FEBRUARY 3, 2023

In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. We all have our habits and as programmers, libraries and frameworks are definitely a part of the group. The part for Python will follow next week!

Scala 130

Scala Java Data Engineering Data Engineer 130

Jesse Anderson

DECEMBER 12, 2022

Big data projects were given to data scientists and data warehouse teams, where the projects subsequently failed. As clearly evident as that sounds now, my writing about needing data engineering went heavily against the grain of everything that was written at the time.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Weekly

FEBRUARY 4, 2024

Joe Reis: Definition of Data Modeling & What Data Modeling Is not Joe raised a very fundamental question in data engineering. What is Data Modeling, and what is not? He rightly points out that many data engineers will name any of the modeling techniques as data modeling.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Weekly

DECEMBER 3, 2023

Github writes an excellent blog to capture the current state of the LLM integration architecture. link] Microsoft: Generative AI for Beginners Understanding Gen-AI becomes a mandatory skill for application developers and data engineers. I experienced similar drawbacks to what Lyft is talking about in Druid.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Data Engineering Weekly

MARCH 3, 2024

Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup. There will be food, networking, and real-world talks around data engineering. The article concludes with a look at data contracts as a concrete example of these principles in practice.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Weekly

FEBRUARY 18, 2024

Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. link] Sponsored: Data modeling and exploration in Playground 2.0 Elevate your data skills!

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Data Dependency : does the pipeline depend on the data of other tables?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

RandomTrees

FEBRUARY 2, 2024

GPT-based data engineering accelerators make the working of data more accessible. These accelerators use GPT models to do data tasks faster, fix any issues, and save a lot of time. GPT models change data in simple language and also provide summaries and explanations. One can rely on this information.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Data Engineering Weekly

MARCH 10, 2024

Furthermore, it presents a common lakehouse design, emphasizing data validation across bronze, silver, and gold layers, and suggests technologies and practices for maintaining high data quality, emphasizing continuous improvement and the critical role of standardization. link] NVIDIA: What Is Trustworthy AI?

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

KDnuggets

NOVEMBER 15, 2021

The October blogs that won KDnuggets Rewards include: How I Tripled My Income With Data Science in 18 Months; What Google Recommends You do Before Taking Their Machine Learning or Data Science Course; How to Build Strong Data Science Portfolio as a Beginner; Data Scientist vs Data Engineer Salary.

Portfolio 94

Portfolio Data Science Machine Learning Data Engineering 94

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. The blog specifically focuses on the following areas of LLM implementations.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . Prerequisites for deploying CDP Data Engineering on Azure can be found here.

Data Engineering 126

Data Engineering Data Engineer Engineering Pipeline-centric 126

Data Engineering Weekly

AUGUST 20, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. But 4 years later, in 2023 — where has the data mesh gotten us?

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Data Engineering Weekly

AUGUST 6, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. But 4 years later, in 2023 — where has the data mesh gotten us?

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Weekly

MARCH 19, 2023

The article discusses incremental processing strategy, handling late-arriving data, and backfilling with the design patterns explaining how Apache Hudi simplifies ETL processing. link] Data Engineering Weekly talks in detail about adopting functional data engineering principles, and Apache Hudi certainly supports it out of the box.

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Data Engineering Podcast

JULY 10, 2022

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

databricks

MAY 1, 2024

The generative AI revolution is transforming the way that teams work, and Databricks Assistant leverages the best of these advancements. It allows you.

Data Engineering 103

Data Engineering Data Engineer Engineering Data 103

Data Engineering Weekly

NOVEMBER 5, 2023

The blog is a classic case study for data engineers who like to build SQL-like abstractions. Google writes an excellent blog on an efficient grammar correction model based on the state-of-the-art EdiT5 model architecture. The blog is an excellent source for designing large-scale systems.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. So, let's shape the future of Data Engineering together.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Waitingforcode

MARCH 23, 2023

Welcome to the 2nd part of the series with great streaming and project organization blog posts summaries!

Data Engineering 147

Data Engineering Data Engineer Engineering Project 147

Data Engineering Weekly

AUGUST 13, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. But 4 years later, in 2023 — where has the data mesh gotten us?

Data Engineering 52

Data Engineering Data Engineer Engineering Food 52

Data Engineering Weekly

MARCH 12, 2023

link] Editor’s Note: Data Engineering Radio At Data Engineering Weekly, We strive to bring the best thought process around building and operating data. We call it Data Engineering Weekly Radio. link] Uber: D3- An Automated System to Detect Data Drifts No Data!

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Weekly

APRIL 23, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating! .

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Data Science Blog: Data Engineering

DECEMBER 23, 2022

Warum Data Engineering der Data Science in Bedeutung und Berufschancen längst die Show stiehlt, dabei selbst ebenso einem stetigen Wandel unterliegt. Was ein Data Engineer wirklich können muss Der Data Scientist als sexiest Job des 21. Data Engineer, der gefragteste Job des 21sten Jahrhunderts?

Data Engineering 52

Data Engineering Data Engineer Engineering Business Intelligence 52

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. Meta shares its ever-changing landscape of data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Waitingforcode

APRIL 30, 2023

Welcome to the 3rd part of the series with great streaming and project organization blog posts summaries!

Data Engineering 130

Data Engineering Data Engineer Engineering Project 130

Data Engineering Weekly

APRIL 2, 2023

Google announces DataPerf, the first community, and platform to build leaderboards for data benchmarks. The Data creation part of a big untapped market in data engineering. nHowever, High-Quality Data Creation and Data collaboration going to remain challenging. ","username":"ananthdurai","name":"at-ananth-at-data-folks

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Weekly

FEBRUARY 19, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Join our webinar to learn how you can replace GA with analytics on your data cloud.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability. To measure, but not track.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Data Engineering Weekly

MARCH 5, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Editor’s Note: Who Owns the Data Contract?

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Weekly

NOVEMBER 26, 2023

It’s the end of the year, and there will be a lot of buzz about what the next five years in data engineering might bring. The Battle for Supremacy: Inside the Fierce Lakehouse Architecture War One of the hot topics in the data industry is which LakeHouse format to choose. But I had my doubts. The categories are merging.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Data Engineering Weekly

JUNE 25, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. The blog narrates the essence of the asset model below.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Weekly

JUNE 18, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Data management is critical for any organization to succeed in this AI world.

Data Engineering 52

Data Engineering Data Engineer Engineering MySQL 52

Data Engineering Weekly

NOVEMBER 27, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52