U-Next

AUGUST 17, 2022

Introduction to 2022 Data Engineer Roles and Responsibilities. Companies and enterprises, large and small, are built on data. Data Engineer roles and responsibilities include aiding in the collection of issues and the delivery of remedies addressing customer demand and product accessibility.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Knowledge Hut

NOVEMBER 2, 2023

For those aspiring to build a career within the Azure ecosystem, navigating the choices between Azure Data Engineers and Azure DevOps Engineers can be quite challenging. Azure Data Engineers and Azure DevOps Engineers are two critical components of the Azure ecosystem for different but interconnected reasons.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Weekly

APRIL 2, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real-time with your own JavaScript or Python code. Meta: Presto - A Decade of SQL Analytics at Meta Presto and Kafka are the two systems that greatly impacted data infrastructure in the last decade.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Ascend.io

MAY 7, 2024

to bring its cutting-edge automation platform that revolutionizes modern data engineering. .” to bring its cutting-edge automation platform that revolutionizes modern data engineering. will strengthen clients’ data security postures by quickly identifying and mitigating potential risks.

Pipeline-centric 52

Pipeline-centric Data Cleanse Data Security Data Pipeline 52

Snowflake

APRIL 8, 2024

The list of Top 10 semi-finalists is a perfect example: we have use cases for cybersecurity, gen AI, food safety, restaurant chain pricing, quantitative trading analytics, geospatial data, sales pipeline measurement, marketing tech and healthcare. Our sincere thanks go out to everyone who participated in this year’s competition.

Pipeline-centric 127

Pipeline-centric Food Healthcare Unstructured Data 127

Knowledge Hut

FEBRUARY 27, 2023

A data engineer is a key member of an enterprise data analytics team and is responsible for handling, leading, optimizing, evaluating, and monitoring the acquisition, storage, and distribution of data across the enterprise. Data Engineers indulge in the whole data process, from data management to analysis.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Towards Data Science

JUNE 27, 2023

The Top 5 Features for Efficient Data Manipulation This April, pandas 2.0.0 was officially launched , making huge waves across the data science community. Due to its extensive functionality and versatility, pandas has secured a place in every data scientist’s heart. Photo by Yancy Min on Unsplash. Yep, pandas 2.0

Pipeline-centric 98

Pipeline-centric Data Science Machine Learning Datasets 98

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Netflix Tech

NOVEMBER 3, 2023

Democratizing Stream Processing @ Netflix By Guil Pires , Mark Cho , Mingliang Liu , Sujay Jain Data powers much of what we do at Netflix. On the Data Platform team, we build the infrastructure used across the company to process data at scale. Furthermore, many pipelines needed to be composed of multiple Processors.

SQL 106

SQL Pipeline-centric Kafka Data 106

Monte Carlo

MAY 30, 2024

Generative AI has the potential to transform your business and your data engineering team, but only when it’s done right. So how can your data team actually drive value with your LLM or GenAI initiative? Data retrieval: Based on the query, the RAG system searches the database to find relevant data.

Pipeline-centric 52

Pipeline-centric Database-centric Datasets Data Pipeline 52

Christophe Blefari

APRIL 8, 2023

Data News entering in town ( credits ) Hey you, if I wasn't late in my newsletter writing it wouldn't be me. But here is your usual Data News. Data modeling Dear readers, I have to confess something. I did not care about data modeling for years. Data modeling should not be a required data engineer skill.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

Data News entering in town ( credits ) Hey you, if I wasn't late in my newsletter writing it wouldn't be me. But here is your usual Data News. Data modeling Dear readers, I have to confess something. I did not care about data modeling for years. Data modeling should not be a required data engineer skill.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Data Engineering Weekly

JANUARY 8, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Pipelines for data in motion can quickly turn into DAG hell.

Data Engineering 40

Data Engineering Data Engineer Engineering Pipeline-centric 40

Data Engineering Podcast

MAY 7, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How do you manage permissions/auditability of updating or amending profile data? Let us know if you have opinions there!

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Cloudera

SEPTEMBER 17, 2020

For enterprise organizations, managing and operationalizing increasingly complex data across the business has presented a significant challenge for staying competitive in analytic and data science driven markets. CDP data lifecycle integration and SDX security and governance. Enterprise Data Engineering From the Ground Up.

Data Pipeline 103

Data Pipeline Data Engineering Data Engineer Engineering 103

Data Engineering Podcast

AUGUST 13, 2022

Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Ascend.io

JUNE 8, 2023

Because of its transformative potential, data has graduated from being merely a by-product of business operations to a critical asset in its own right. Is it possible to treat data not just as a necessary operational output, but as a product that holds immense strategic value? Without them, data products can’t exist.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Towards Data Science

MAY 19, 2023

Data Entropy — More Data, More Problems? How to navigate and embrace complexity in a modern data organisation. Business users are unable to find and access data assets critical to their workflows. Data engineers spend countless hours troubleshooting broken pipelines. The list goes on.

Pipeline-centric 105

Pipeline-centric Data Software Engineer Software Engineering 105

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? Data science as a discipline was going through its adolescence of self-affirming and defining itself. We were pioneers.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

DataKitchen

FEBRUARY 21, 2023

On 24 January 2023, Gartner released the article “ 5 Ways to Enhance Your Data Engineering Practices.” Data team morale is consistent with DataKitchen’s own research. We surveyed 600 data engineers , including 100 managers, to understand how they are faring and feeling about the work that they are doing.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Grouparoo

OCTOBER 6, 2021

It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Ripple Engineering

JANUARY 29, 2024

Why do we need a Data Platform? At Ripple, different business units require a platform to quickly and easily turn their data into insights and compelling customer experiences.   A lack of a centralized system makes building a single source of high-quality data difficult. Why is it difficult to build a centralized platform?

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

Dataquest

JANUARY 25, 2017

From helping cars drive themselves to helping Facebook tag you in photos , data science has attracted a lot of buzz recently. Data scientists have become extremely sought after , and for good reason — a skilled data scientist can add incredible value to a business. But what about data engineers?

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Ascend.io

AUGUST 29, 2023

A star-studded baseball team is analogous to an optimized “end-to-end data pipeline” — both require strategy, precision, and skill to achieve success. Just as every play and position in baseball is key to a win, each component of a data pipeline is integral to effective data management.

Data Pipeline 52

Data Pipeline Pipeline-centric Database-centric Data Ingestion 52

ProjectPro

DECEMBER 16, 2021

Are you a newbie in the data science domain ready to embark on a rewarding journey but are confused between the roles of a Machine Learning Engineer vs Data Scientist? Data Science is an emerging discipline and so are the roles and job titles pretty much evolving.

Machine Learning 52

Machine Learning Engineering Pipeline-centric Database-centric 52

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Pipeline Data Engineering

MARCH 2, 2022

Data engineering salon. News and interesting reads about the world of data. The Unbundling of Airflow Gorkem Yurtseven, Co-Founder, Features and Labels A diverse set of tools is unbundling Airflow and this diversity is causing substantial fragmentation in modern data stack. What the hell happened?

Pipeline-centric 52

Pipeline-centric BI Consulting Data Warehouse 52

Snowflake

JUNE 28, 2023

We’ve talked about this opportunity before and at Summit 2023, we announced a number of capabilities that come together to help our customers bring generative AI and LLMs directly to their proprietary data, all delivered through a single, secure platform. Read on to learn more about these announcements. A win/win for both parties.

Pipeline-centric 75

Pipeline-centric Unstructured Data Government Data 75

Cloudera

DECEMBER 13, 2023

Implementing a cohesive data vision LGIM’s Global Chief Operating Office is responsible for the company’s technology, data, client servicing and the management of its strategic operating agenda. LGIM’s data ecosystem had become fragmented over time as it had grown.

Pipeline-centric 89

Pipeline-centric Professional Services Datasets BI 89

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? This was a fascinating conversation with someone who has spent his entire career working on simplifying complex data problems. They have built an easy to use platform that lets you leverage your company’s single sign on for your data platform.

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

Netflix Tech

JANUARY 10, 2024

The Netflix video processing pipeline went live with the launch of our streaming service in 2007. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Process 91

Process Pipeline-centric Media Metadata 91

Ascend.io

APRIL 27, 2023

Going into the Data Pipeline Automation Summit 2023, we were thrilled to connect with our customers and partners and share the innovations we’ve been working on at Ascend. The summit explored the future of data pipeline automation and the endless possibilities it presents.

Data Pipeline 52

Data Pipeline Pipeline-centric Data Validation Data Engineering 52

Meltano

JANUARY 25, 2023

A curated list of the top 9 must read blogs on data. The data world is in turmoil and lots of exciting things happen every day, week and year. At Meltano we’re ourselves avid users of data, data engineers, data PMs, and data enthusiasts through and through. Happy reading!

Pipeline-centric 52

Pipeline-centric Database-centric Data Warehouse SQL 52

DoorDash Engineering

NOVEMBER 28, 2023

DoorDash has been building an internal Machine Learning Workbench over the past year to enhance data operations and assist our data scientists, analysts, and AI/ML engineers. As shown in Figure 1, data science intersects ML in multiple ways and is paramount to DoorDash’s success.

Machine Learning 90

Machine Learning Pipeline-centric Data Science Designing 90

Christophe Blefari

DECEMBER 19, 2022

of the Data News. But this article is not about the pricing which can be very subjective depending on the context—what is 1200$ for dev tooling when you pay them more than $150k per year, yes it's US-centric but relevant. Last week dbt Labs decided to change the pricing of their Cloud offering.

Management 130

Management Pipeline-centric Database-centric SQL 130