Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data.

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

Analytics Vidhya

FEBRUARY 19, 2023

Introduction Data pipelines play a critical role in the processing and management of data in modern organizations. A well-designed data pipeline can help organizations extract valuable insights from their data, automate tedious manual processes, and ensure the accuracy of data processing.

Data Pipeline 197

Data Pipeline Data Designing Data Process 197

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently.

Data Engineering 162

Data Engineering Data Engineer Engineering Data Pipeline 162

Confessions of a Data Guy

JANUARY 13, 2023

Rust has been on my mind a lot lately, probably because of Data Engineering boredom, watching Spark clusters chug along like some medieval farm worker endlessly trudging through the muck and mire of life. appeared first on Confessions of a Data Guy. appeared first on Confessions of a Data Guy.

Data Pipeline 147

Data Pipeline Data Engineering Data Engineer Data 147

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community!

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Start Data Engineering

FEBRUARY 22, 2024

Data Pipeline Logging Best Practices 3.1. Metadata: Information about pipeline runs, & data flowing through your pipeline 3.2. Introduction 2. Setup & Logging architecture 3. Obtain visibility into the code’s execution sequence using text logs 3.3. Understand resource usage by tracking Metrics 3.4.

Metadata 130

Metadata Data Engineering Data Engineer Engineering 130

Start Data Engineering

MAY 13, 2021

What is an idempotent function Pre-requisites Why idempotency matters Making your data pipeline idempotent Conclusion Further reading References What is an idempotent function “Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the (..)

Data Pipeline 147

Data Pipeline Computer Science Data Data Engineering 147

Towards Data Science

MARCH 3, 2024

Data engineering techniques for robust and sustainable ETL Continue reading on Towards Data Science »

Data Pipeline 79

Data Pipeline Building Data Science Data 79

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is feature engineering is and why/to whom it matters?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Towards Data Science

APRIL 3, 2023

Data pipeline management done right simplifies deployment and increases the availability and accessibility of data for analytics Continue reading on Towards Data Science »

Data Pipeline 81

Data Pipeline Data Science Data Accessible 81

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineering Data Engineer Engineering 147

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions. What is late-arriving data? Let’s dive in!

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Data Engineering Weekly

MARCH 17, 2024

Compliance is mandatory, with strict penalties for violations, emphasizing the importance of data scientists familiarizing themselves with the law to avoid prohibited AI uses and ensure ethical, safe AI development. It discusses the significance of data governance, sharing history, and generative AI's impact on data economy standards.

Data Engineering 70

Data Engineering Data Engineer Engineering Data Governance 70

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 189

Data Engineering Data Engineer Engineering Project 189

Start Data Engineering

OCTOBER 12, 2021

Introduction Responsibilities of a data engineer 1. Move data between systems 2. Manage data warehouse 3. Schedule, execute, and monitor data pipelines 4. Serve data to the end-users 5. Data strategy for the company 6.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Warehouse 130

Start Data Engineering

DECEMBER 12, 2021

Introduction If you are building data pipelines from the ground up, the number of available data engineering tools to choose from can be overwhelming. Components 4. Choosing tools 4.1 Requirement x Component framework 4.2 Conclusion 6. Further reading 1.

Data Pipeline 130

Data Pipeline Data Data Engineering Data Engineer 130

RandomTrees

FEBRUARY 2, 2024

GPT-based data engineering accelerators make the working of data more accessible. These accelerators use GPT models to do data tasks faster, fix any issues, and save a lot of time. GPT models change data in simple language and also provide summaries and explanations. One can rely on this information.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Towards Data Science

MARCH 23, 2024

Data pipelines that would turn you into a decorated data professional Continue reading on Towards Data Science »

Data Engineering 96

Data Engineering Data Engineer Project Engineering 96

Snowflake

OCTOBER 3, 2023

We are excited to announce the availability of data pipelines replication, which is now in public preview. In the event of an outage, this powerful new capability lets you easily replicate and failover your entire data ingestion and transformations pipelines in Snowflake with minimal downtime.

Data Pipeline 74

Data Pipeline Management Data Ingestion Data 74

Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineering 85

Data Engineering Data Engineer Engineering SQL 85

Start Data Engineering

APRIL 8, 2021

Understand the current state of the data pipeline 1. Timelines & deliverables Deliver iteratively Conclusion Further reading References Introduction As data engineers, you will have to re-engineer legacy data pipelines. Think like the end user 2. Know the why 3. End user interviews 4.

Data Pipeline 130

Data Pipeline Engineering Data Data Engineering 130

Data Engineering Weekly

DECEMBER 3, 2023

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. link] Microsoft: Generative AI for Beginners Understanding Gen-AI becomes a mandatory skill for application developers and data engineers.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Data Engineering Podcast

OCTOBER 28, 2021

Summary Streaming data systems have been growing more capable and flexible over the past few years. Despite this, it is still challenging to build reliable pipelines for stream processing. Struggling with broken pipelines? Missing data? Start trusting your data with Monte Carlo today! Stale dashboards?

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Knowledge Hut

MARCH 15, 2024

At the same time, it has opened up a wealth of opportunities for data engineers. With businesses harnessing the power of Azure’s services, the need for skilled data engineers has topped the charts. Speaking from experience, the data engineers in this role are right in the thick of it all.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Governance 52

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . CDP data lifecycle integration and SDX security and governance. Easy job deployment.

Data Engineering 124

Data Engineering Data Engineer Engineering Pipeline-centric 124

Knowledge Hut

MARCH 20, 2024

At the same time, it has opened up a wealth of opportunities for data engineers. With businesses harnessing the power of Azure’s services, the need for skilled data engineers has topped the charts. Speaking from experience, the data engineers in this role are right in the thick of it all.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Governance 52

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineering 130

Data Engineering Data Engineer Engineering Cloud Computing 130

Towards Data Science

DECEMBER 11, 2023

Streaming data pipelines and real-time analytics Continue reading on Towards Data Science »

Data Engineering 80

Data Engineering Data Engineer Engineering Data Pipeline 80

Acceldata

FEBRUARY 16, 2023

Why Data Engineers Must Manage Data Pipelines, Not Data Warehouses

Data Warehouse 52

Data Warehouse Data Pipeline Data Engineering Data Engineer 52

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. What is the data behavior? See how it works today. Dropbox: Is this a date?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Towards Data Science

FEBRUARY 24, 2023

The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. While this idea isn’t completely foreign to data engineering, it can certainly be described as an extremely uncommon practice. Data is different. Let’s introduce some chaos.”

Data Engineering 98

Data Engineering Data Engineer Engineering Software Engineer 98

Start Data Engineering

OCTOBER 22, 2021

Data modeling 2.1.1. Data storage 2.2. Data transformation 2.2.1. Data pipeline 2.4. Data analytics 3. Introduction SQL is the bread and butter of data engineering. Introduction 2. SQL skills 2.1. Gathering requirements 2.1.2. Exploration 2.1.3. Modeling 2.1.4. Transformation types 2.2.1.1.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

Data Engineering Weekly

MARCH 10, 2024

Editor’s Note: Chennai Meetup Wrap-Up & Preparation work started for DEWCon I am so grateful for the enthusiastic participants who made our Chennai Data Heroes- Community for Data Folks meetup vibrant! Big thanks to our insightful speakers, Hareshkumar Selvakumar - Talks about his work on Data Products for PayPal.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineering 98

Data Engineering Data Engineer Coding Project 98