DoorDash Engineering

JANUARY 16, 2024

As the number of microservices and back-ends has grown, however, DoorDash has observed an uptick in cross-availability zone (AZ) data transfer costs. These data transfer costs — incurred on both send and receive — allow DoorDash to provide its end users a highly available service that can withstand degradations of one or more AZs.

Bytes 84

Bytes Cloud Management PostgreSQL 84

Data Engineering Podcast

DECEMBER 26, 2021

Summary The data mesh is a thesis that was presented to address the technical and organizational challenges that businesses face in managing their analytical workflows at scale. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

BI 130

BI Data Warehouse Data Engineering Data Engineer 130

Monte Carlo

FEBRUARY 19, 2024

And there are few data industry buzzwords that have been more hyped over the last four years than the self-service darling, “the data mesh.” Now, let me just say on the front end, the data mesh is not an Avatar 2 situation. (14 Table of Contents What’s a data mesh again? Newsletters. It was a lot.

Architecture 59

Architecture Government Data Data Architecture 59

Data Engineering Weekly

APRIL 7, 2024

Poor data quality and unlcear data ownership remains the top challenges for the data teams. Data Mesh continuously gaining popularity among the enterprises. It is a stark difference from the Gartner report about data mesh. I guess only the time will tell who wins in the data mesh vs data fabric war.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Jesse Anderson

APRIL 13, 2022

Data mesh is quickly becoming a way for companies to roll out their data strategy. If you haven’t already learned about data mesh , I suggest doing so. I think a crucial part of your data mesh revolves around the choice of publish/subscribe technologies. They both have similar publish APIs.

Kafka 124

Kafka Data Cloud Technology 124

Striim

FEBRUARY 19, 2024

Organizations struggle with managing vast amounts of complex data. Traditional methods often fall short, leading to data silos and security vulnerabilities. Innovations like Data Mesh and Data Fabric have emerged as solutions, offering new ways to manage data effectively and derive actionable insights.

Data Integration 52

Data Integration Government Data Data Governance 52

Christophe Blefari

APRIL 5, 2024

Lost between ideas ( credits ) Hey, new Data News edition. MDS Fest is a free virtual 5 days conference about Modern Data Stack topics, a lot of awesome speakers, there are a few talks I can't wait to watch. It shows a few RAGs, Agents and parsers on documents to retrieve the data you need. But here we are.

SQL 130

SQL Data Python Datasets 130

Towards Data Science

JANUARY 16, 2024

My personal take on justifying the existence of Data Mesh A senior stakeholder at one my projects mentioned that they wanted to decentralise their data platform architecture and democratise data across the organisation. When I heard the words ‘decentralised data architecture’, I was left utterly confused at first!

Data Warehouse 66

Data Warehouse Data Architecture Relational Database NoSQL 66

Christophe Blefari

OCTOBER 14, 2023

( credits ) Hello, dear Data News reader, I hope you'll enjoy this new edition. Then Marc Lamberti gave a huge update about Airflow but done differently — It wasn't about slides with a list of new features but rather about how you can write, in 2023, a data pipeline with Airflow.

Python 130

Python Datasets Data Banking 130

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. In this article, we will explore the advantages and limitations of data mesh, while also providing best practices for building and optimizing a data mesh with Striim.

Architecture 52

Architecture Generalist Government Datasets 52

François Nguyen

MARCH 7, 2021

I discovered Zhamak Dehghani’s first article about Data Mesh in August 2020. I will try to answer to this question “Could you illustrate your journey towards a Data Mesh ?” Solving the right problem Let’s say that your starting point is having multiple data teams and multiple data platforms.

Government 130

Government Data Governance Data Metadata 130

Ascend.io

JULY 17, 2023

If you work with data, you’ll have come across the term data mesh by now. This decentralized but interconnected approach to structuring data has become increasingly popular since the term was coined by Zhamak Dehghani 4 years ago. Data Mesh: 101 First, a quick primer on the data mesh concept.

Datasets 52

Datasets Telecommunication Architecture Government 52

Data Engineering Podcast

JANUARY 23, 2022

Summary Data platforms are exemplified by a complex set of connections that are subject to a set of constantly evolving requirements. The recent move toward the data mesh as a formalized architecture that builds on this design provides the language that data teams need to make this a more organized effort.

Data Integration 100

Data Integration Data Pipeline Data Architecture 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? How can we interoperate between the data domains ?

Architecture 130

Architecture Technology Google Cloud Metadata 130

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly. Apache Kafka came in 2011 and gave the industry a much better way to move real-time data.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Towards Data Science

NOVEMBER 4, 2023

Platform Specific Tools and Advanced Techniques Photo by Christopher Burns on Unsplash The modern data ecosystem keeps evolving and new data tools emerge now and then. In this article, I want to talk about crucial things that affect data engineers. Are your data pipelines efficient? Data warehouse exmaple.

Data Engineering 85

Data Engineering Data Engineer Engineering BI 85

Snowflake

AUGUST 21, 2023

With demonstrable success across a range of industries, organizations are increasingly pursuing cutting-edge data mesh architectures to enhance self-service data use. Data-as-a-product: By considering data resources through a product lens, teams can adopt practices centered around quality and ease of use.

Pipeline-centric 52

Pipeline-centric Architecture Government Data Architect 52

Data Engineering Podcast

JULY 3, 2022

Summary The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform architectures for managing data and analytical workflows. Atlan is the metadata hub for your data ecosystem.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Integration 162

Data Integration BI SQL Python 162

Ascend.io

APRIL 20, 2023

Parsing current data stacks is like playing the longest-lasting game of Jenga. Only the most experienced data engineers can extract pieces of it. But there’s a better way to play the game: data mesh. Data mesh decentralizes and democratizes data across the organization. Why Is Data Mesh Important?

Government 52

Government Architecture Data Architecture Data Governance 52

Cloudera

JUNE 7, 2022

We live in a hybrid data world. In the past decade, the amount of structured data created, captured, copied, and consumed globally has grown from less than 1 ZB in 2011 to nearly 14 ZB in 2020. Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB.

IT 108

IT Unstructured Data Data Architecture Government 108

Monte Carlo

JANUARY 23, 2023

When it comes to the data community, there’s always a debate broiling about something— and right now “data mesh vs data lake” is right at the top of that list. In this post we compare and contrast the data mesh vs data lake to illustrate the benefits of each and help discover what’s right for your data platform.

Data Lake 52

Data Lake Business Intelligence Unstructured Data Architecture 52

Cloudera

SEPTEMBER 15, 2022

Cloudera Contributor: Mark Ramsey, PhD ~ Globally Recognized Chief Data Officer. July brings summer vacations, holiday gatherings, and for the first time in two years, the return of the Massachusetts Institute of Technology (MIT) Chief Data Officer symposium as an in-person event. Luke: What is a modern data platform?

Data Lake 79

Data Lake Analytics Application Cloud Storage Architecture 79

Christophe Blefari

NOVEMBER 25, 2022

Capturing the news ( credits ) Hello you, I hope this data news finds you well. I've launched in a rush an Advent of Data. The goal is simple, in December: 24 data people will produce 24 data gems. I know it's a late notice thing, but this is a good occasion to contribute to the data community.

Data 130

Data Java Data Engineering Data Engineer 130

Ascend.io

JULY 5, 2023

As more companies ingest and leverage ever-increasing amounts of data, a new problem has come to light: it turns out that monolithic data platforms built and managed by centralized teams are rarely the best engine with which to create value. At the same time, the practice of building data products has also grown in popularity.

Government 52

Government Data Data Pipeline Architecture 52

Data Engineering Weekly

SEPTEMBER 3, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. See how it works today. 📣 Exciting news! . 📩 Looking forward to seeing you all!

Data Engineering 73

Data Engineering Data Engineer Engineering Data 73

Monte Carlo

JUNE 22, 2023

At our latest IMPACT summit, Roche shared their data mesh strategy for creating reliable data products. The goal of the Roche data team is to maximize the outcomes of customers and patients through data and analytics products. So they agreed to leverage Data Vault 2.0

Building 52

Building Data Ingestion Data Business Analyst 52

DataKitchen

DECEMBER 13, 2021

Gartner – Top Trends and Data & Analytics for 2021: XOps. What is a Data Mesh? DataOps Data Architecture. A Guide to Understanding DataOps Solutions. DataOps is Not Just a DAG for Data. Data Observability and Monitoring with DataOps. DataOps is NOT Just DevOps for Data.

Data Governance 98

Data Governance Data Architecture Government Data Engineering 98

Ascend.io

MARCH 14, 2023

Data represents our present and our future, and therein lies a significant problem: the more data you’re dealing with, the more challenging it will be to scale your company in a sustainable and standardized way. So, what’s the solution? What Is a Data Mesh?

Government 52

Government Architecture Data Lake Data 52

Ascend.io

MARCH 14, 2023

Data represents our present and our future, and therein lies a significant problem: the more data you’re dealing with, the more challenging it will be to scale your company in a sustainable and standardized way. So, what’s the solution? What Is a Data Mesh?

Government 52

Government Architecture Data Lake Data 52

Snowflake

JANUARY 25, 2023

Find out why its Head of Data and Information Management uses data mesh—and Snowflake—to make it a reality. It’s on a mission to drive the shift towards more sustainable transport solutions that benefit the environment and meet the needs of people and society throughout the transport ecosystem.

Transportation 52

Transportation Cloud Manufacturing High Quality Data 52

Monte Carlo

DECEMBER 28, 2022

Data mesh is a complex socio-technological data engineering concept, but it doesn’t change too much. Data mesh is one of those rare transformative concepts that emerged relatively fully formed as a result of creator Zhamak Dhegani’s years of consulting experience captured in a comprehensive 384 page book.

Retail 52

Retail Consulting Architecture Data 52

Monte Carlo

APRIL 4, 2023

Data mesh vs data warehouse is an interesting framing because it is not necessarily a binary choice depending on what exactly you mean by data warehouse (more on that later). Despite their differences, however, both approaches require high-quality, reliable data in order to function. What is a Data Mesh?

Data Warehouse 52

Data Warehouse Data Governance Data Architecture 52

Data Engineering Podcast

JULY 22, 2019

Summary The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and access.

Data Lake 100

Data Lake Hadoop Data Architecture 100

Monte Carlo

JULY 25, 2022

There’s a lot of content out there about why a data mesh is (or isn’t) the best thing since sliced bread. But one thing’s for sure: if you can’t trust the data powering your analytics architecture, it’s hard to justify the investment. The data mesh contains multitudes. What does that mean, exactly?

Government 52

Government Data Architecture Insurance 52

Data Engineering Weekly

AUGUST 13, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. But 4 years later, in 2023 — where has the data mesh gotten us?

Data Engineering 52

Data Engineering Data Engineer Engineering Food 52

Data Engineering Weekly

AUGUST 6, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. One of the standard challenges in a data pipeline is de-duplication.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52