Cloudera

JUNE 2, 2022

Since 2015, the Cloudera DataFlow team has been helping the largest enterprise organizations in the world adopt Apache NiFi as their enterprise standard data movement tool. This need has generated a market opportunity for a universal data distribution service. Why does every organization need it when using a modern data stack?

Systems 101

Systems Data Lake Google Cloud Data Collection 101

Rockset

AUGUST 11, 2022

I had the pleasure of recently hosting a data engineering expert discussion on a topic that I know many of you are wrestling with – when to deploy batch or streaming data in your organization’s data stack. Andreas also produces the well-known ‘Plumbers of Data Science’ blog and podcast , as well as a YouTube channel.

Bytes 52

Bytes Consulting Kafka MongoDB 52

Data Engineering Weekly

APRIL 2, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real-time with your own JavaScript or Python code. Meta: Presto - A Decade of SQL Analytics at Meta Presto and Kafka are the two systems that greatly impacted data infrastructure in the last decade.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Podcast

SEPTEMBER 22, 2019

Summary Object storage is quickly becoming the unifying layer for data intensive applications and analytics. Modern, cloud oriented data warehouses and data lakes both rely on the durability and ease of use that it provides. And for your machine learning workloads, they just announced dedicated CPU instances.

AWS 100

AWS Google Cloud Cloud Storage Data Lake 100

Pinterest Engineering

OCTOBER 31, 2023

At Pinterest, we are evolving our compute platform to provide an application-centric and fully managed compute API for the 90th percentile of use cases. Architecture PinCompute is a regional Platform-as-a-Service (PaaS) that builds on top of Kubernetes. The member clusters are zonal, and are used for the actual workload executions.

Architecture 95

Architecture Pipeline-centric Accessible Accessibility 95

Monte Carlo

JULY 7, 2022

Editor’s Note : We ran into Andrew at our London IMPACT event in early 2022. At the time, he was one of a very few people using the term “data contract.” Not only was he using the term, but his implementation was generating results. Forward: A Data Engineering Movement Afoot? Or is it a passing fad?

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

Netflix Tech

JULY 26, 2021

This happens at an unprecedented scale and introduces many interesting challenges; one of the challenges is how to provide visibility of Studio data across multiple phases and systems to facilitate operational excellence and empower decision making. tactical) in nature.

Data 101

Data Data Pipeline MySQL Data Warehouse 101

Data Engineering Weekly

MARCH 31, 2024

Intuit: How Intuit data analysts write SQL 2x faster with the internal GenAI tool The productivity increase with GenAI is undeniable, and several startups are trying to solve the Text2SQL generation problem. My key highlight is that Excellent data documentation and “clean data” improve results.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Rockset

FEBRUARY 24, 2023

Introduction Let’s get this out of the way at the beginning: understanding effective streaming data architectures is hard, and understanding how to make use of streaming data for analytics is really hard. Stream processing or an OLAP database? What Is Streaming Data? Kafka or Kinesis ?

Kafka 52

Kafka AWS Amazon Web Services Programming Language 52

Data Engineering Weekly

OCTOBER 29, 2023

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. LinkedIn shares its success story of using Apache Beam across business use cases. The explosion of modern data stacks made it much harder to build.

Data Engineering 52

Data Engineering Data Engineer Engineering Algorithm 52

Picnic Engineering

JULY 26, 2022

Data Mesh, or Hub-and-Spoke? and other reflections on building data governance. Since the publication of the first blog post in this series, we have received numerous questions via social media, direct messages, public posts, and meet-up discussions. How to succeed with an analytics strategy? Any advice?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Weekly

NOVEMBER 13, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. I'm a software engineer; how can I transition to a Machine Learning engineer?

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Cloudera

AUGUST 20, 2021

How CDP Enables and Accelerates Data Product Ecosystems. A multi-purpose platform focused on diverse value propositions for data products. As a result, CDP-enabled data products can meet multiple and varying functional and non-functional requirements that correspond to product attributes, each fulfilling specific customer needs.

Data Warehouse 96

Data Warehouse Data Architecture Cloud 96

Monte Carlo

SEPTEMBER 6, 2022

Say it with me: bad data is inevitable. It doesn’t care about how proactive you are at writing dbt tests, how perfectly your data is modeled, or how robust your architecture is. The possibility of a major data incident (Null value? Errant schema change? Failed model?) Check it out. Here’s why.

Software Engineer 52

Software Engineer Software Engineering Retail Data 52

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Introduction to the Data Mesh Architecture and its Required Capabilities. Introduction.

Architecture 123

Architecture Metadata Government Kafka 123

Cloudera

JUNE 16, 2022

The ability to perform analytics on data as it is created and collected (a.k.a. real-time data streams) and generate immediate insights for faster decision making provides a competitive edge for organizations. . CSP was recently recognized as a leader in the 2022 GigaOm Radar for Streaming Data Platforms report.

Kafka 85

Kafka Manufacturing Data Lake SQL 85

Data Engineering Weekly

AUGUST 27, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. See how it works today. Does that will data teams shrink in size?

Data Engineering 52

Data Engineering Data Engineer Engineering Machine Learning 52

Snowflake

JULY 13, 2023

As we continue to revolutionize the way businesses operate, allowing them to solve their most pressing problems and drive revenue through the Data Cloud, the insights, expertise, and experiences we offer at Summit have continued to grow. Whether you missed the event this year (there’s always Summit 2024 !)

Entertainment 57

Entertainment Manufacturing Retail Media 57

Data Engineering Weekly

JULY 23, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. See how it works today. How to make a cost-efficient choice?

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Data Engineering Weekly

JULY 30, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. See how it works today. We handle Petabytes of data!!! If you’re a.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Christophe Blefari

NOVEMBER 18, 2023

Still I've accumulated a lot of articles that I think should fit in the Data News so this week might be a huge recap of content that has been produce in the last month. I think it will not be live but the recording will be published after the event on YouTube I think. I hope you will enjoy the selection. It was a brutal decision.

Food 147

Food Data Data Lake Retail 147