Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? Data science as a discipline was going through its adolescence of self-affirming and defining itself. We were pioneers.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Weekly

JANUARY 8, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Pipelines for data in motion can quickly turn into DAG hell.

Data Engineering 40

Data Engineering Data Engineer Engineering Pipeline-centric 40

Cloudera

SEPTEMBER 17, 2020

With growing disparate data across everything from edge devices to individual lines of business needing to be consolidated, curated, and delivered for downstream consumption, it’s no wonder that data engineering has become the most in-demand role across businesses — growing at an estimated rate of 50% year over year.

Data Pipeline 103

Data Pipeline Data Engineering Data Engineer Engineering 103

Grouparoo

OCTOBER 6, 2021

It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Monte Carlo

MARCH 28, 2024

As well as answering that question, we’ll be delving into how they differ from data scientists and data engineers. We’ll also be covering what they do day to day, what sort of salaries they command, and how individuals might go about pivoting to become a DataOps engineer. What does a DataOps engineer do?

Pipeline-centric 75

Pipeline-centric Engineering BI Google Cloud 75

Data Engineering Podcast

AUGUST 13, 2022

Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Dataquest

JANUARY 25, 2017

Data scientists have become extremely sought after , and for good reason — a skilled data scientist can add incredible value to a business. But what about data engineers? A data scientist is only as good as the data they have access to. A data engineer transforms data into a useful format for analysis.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Christophe Blefari

DECEMBER 19, 2022

But this article is not about the pricing which can be very subjective depending on the context—what is 1200$ for dev tooling when you pay them more than $150k per year, yes it's US-centric but relevant. It can be deployment in all environment or as a lot of data only in production, because only production exists.

Management 130

Management Pipeline-centric Database-centric SQL 130

Meltano

JANUARY 25, 2023

A curated list of the top 9 must read blogs on data. The data world is in turmoil and lots of exciting things happen every day, week and year. At Meltano we’re ourselves avid users of data, data engineers, data PMs, and data enthusiasts through and through. Happy reading!

Pipeline-centric 52

Pipeline-centric Database-centric Data Warehouse SQL 52

Pipeline Data Engineering

MARCH 2, 2022

Data engineering salon. News and interesting reads about the world of data. The Unbundling of Airflow Gorkem Yurtseven, Co-Founder, Features and Labels A diverse set of tools is unbundling Airflow and this diversity is causing substantial fragmentation in modern data stack. What the hell happened?

Pipeline-centric 52

Pipeline-centric BI Consulting Data Warehouse 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services.

Technology 130

Technology Architecture Google Cloud Metadata 130

DataKitchen

AUGUST 27, 2021

It implemented hundreds of schema and data set changes per week without introducing errors. Arguably the most agile and effective data analytics capability in the pharmaceutical industry was accomplished cost-effectively, with a data engineering team of seven and another 10-12 data analysts.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Lake Data Analytics 98

Ascend.io

OCTOBER 20, 2023

The Nuances of Snowflake Costing Snowflake’s pricing strategy is an exemplification of its user-centric approach: pay for what you use. The more tables you have, the kind of SQL queries you run, and the dimensions of your data warehouses are pivotal determinants. So Why Do Snowflake Costs Become Prohibitive?

Pipeline-centric 52

Pipeline-centric IT Data Pipeline Bytes 52

RandomTrees

NOVEMBER 29, 2023

So, here the argument on what to use and where to use become an important topic to be considered and here we can rather focus on some of the adaptable and sensitive models and we begin to consider the data vaults’ technique. So, what is a data vault model or modelling approach? post which is the ML model trainings.

Data Warehouse 52

Data Warehouse Data Lake Database-centric Data Cleanse 52

phData: Data Engineering

JULY 18, 2023

With the birth of cloud data warehouses, data applications, and generative AI , processing large volumes of data faster and cheaper is more approachable and desired than ever. This typically results in long-running ETL pipelines that cause decisions to be made on stale or old data.

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52

DataKitchen

NOVEMBER 4, 2021

It often takes months to progress from a data lake to the final delivery of insights. One data engineer called it the “last mile problem.” . In our many conversations about data analytics, data engineers, analysts and scientists have verbalized the difficulty of creating analytics in the modern enterprise.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Knowledge Hut

SEPTEMBER 26, 2023

Microsoft Azure's Azure Synapse, formerly known as Azure SQL Data Warehouse, is a complete analytics offering. Designed to tackle the challenges of modern data management and analytics, Azure Synapse brings together the worlds of big data and data warehousing into a unified and seamlessly integrated platform.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52

Monte Carlo

JANUARY 9, 2024

We first heard these terrifying anecdotes when we gathered a panel of data experts: Gable CEO and co-founder Chad Sanderson , Ternary Data CEO and author of O’Reilly Data Engineering Fundamentals Joe Reis , and Capital One Senior Manager of Analytics & Data Science Emily Riederer.

Pipeline-centric 52

Pipeline-centric Database-centric Data Manufacturing 52

Databand.ai

JULY 19, 2023

Learn more in our detailed guide to data lineage visualization (coming soon) Integration with Multiple Data Sources Data lineage tools are designed to integrate with a wide range of data sources, including databases, data warehouses, and cloud-based data platforms.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Monte Carlo

DECEMBER 7, 2022

Check out Rob’s full conversation on radical transparency in data engineering during our IMPACT 2022 conference, available on demand. The GitLab data stack Using a cloud-based and modular data stack makes it easy for the data team to scale while serving distributed stakeholders. Image courtesy of GitLab.

Building 52

Building Pipeline-centric Data Data Programming 52

Databand.ai

DECEMBER 20, 2022

He is also an open-source developer at The Apache Software Foundation and the author of Hysterical , a popular blog on tech careers and topics like data, coding, and engineering. On LinkedIn, he posts frequently about data engineering, data architecture, interview preparation, and career advice.

Data Analytics 52

Data Analytics Google Cloud Data Science Data Mining 52

Databand.ai

DECEMBER 29, 2022

She regularly contributes to IBM’s Journey to AI blog and shares her advice on LinkedIn around data strategy, data science, women in AI, data and analytics, data governance, and artificial intelligence. On LinkedIn, he posts regularly about AI, data, data science, data engineering, and machine learning.

BI 52

BI Consulting Data Science Data Governance 52

Airbnb Tech

JUNE 1, 2021

On the other hand, it burdened the centralized data engineering with the impossible task of gatekeeping and onboarding an endless stream of new datasets into new and existing core tables. Furthermore, pipelines built downstream of core_data created a proliferation of duplicative and diverging metrics.

Datasets 52

Datasets Pipeline-centric Metadata Data Science 52

DataKitchen

JUNE 24, 2021

In May 2021 at the CDO & Data Leaders Global Summit, DataKitchen sat down with the following data leaders to learn how to use DataOps to drive agility and business value. Kurt Zimmer, Head of Data Engineering for Data Enablement at AstraZeneca. Jim Tyo, Chief Data Officer, Invesco. Be business-centric.

Pipeline-centric 84

Pipeline-centric Manufacturing Education Data Cleanse 84