The Modern Data Company

FEBRUARY 28, 2023

However, knowing that the cloud is an important piece of the puzzle does not mean that companies aren’t making a lot of mistakes with cloud migrations and implementations. This blog post will discuss some of the common causes, which have nothing to do with technology and everything to do with poor planning.

Cloud 98

Cloud Hadoop Technology Coding 98

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. I'm open to all honest feedbacks. Is it Kafka?

Kafka 130

Kafka Data Lake Data Storage Data 130

Christophe Blefari

DECEMBER 31, 2023

I'd also like to say hello to all the newcomers who arrived in December, thank you for your trust. I'd also like to say hello to all the newcomers who arrived in December, thank you for your trust. Before moving on to the Data News, a bit of personal news, in December, I took part in the MotherDuck meetup in Berlin.

Data 100

Data Cloud Storage Datasets Python 100

Data Engineering Weekly

FEBRUARY 4, 2024

link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. It opens some old memory; try to solve this problem first with Presto-Kafka connector and then using OLAP engines like Druid & Apache Pinot. Visit rudderstack.com to learn more.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. This is crucial for use cases like market signaling and forecasting which benefit from, and depend upon, the most up-to-date information. Written by Ritesh Varyani and Jeana Choi at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Cloudera

JULY 14, 2023

However, as there are already 25 million terabytes of data stored in the Hive table format, migrating existing tables in the Hive table format into the Iceberg table format is necessary for performance and cost. In this blog, I will describe a few strategies one could undertake for various use cases.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Data Engineering Weekly

NOVEMBER 26, 2023

This year still feels like a fresh newbie, especially since we just bounced back from the pandemic rollercoaster 🎢 of last year. It's like piecing together a puzzle from history to get a clearer picture of the future. Databricks is already doing a lot of these move-up stack products. But I had my doubts.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Christophe Blefari

MAY 19, 2023

One day, I decided to save the links on a blog created for the occasion, a few days later, 3 people subscribed. One day, I decided to save the links on a blog created for the occasion, a few days later, 3 people subscribed. If you had not joined so early, I would never have realized that people would like to read my content.

Data 130

Data Data Engineering Data Engineer Hadoop 130

Pinterest Engineering

OCTOBER 17, 2023

Charles Wu | Software Engineer; Isabel Tallam | Software Engineer; Kapil Bajaj | Engineering Manager Overview In this blog, we present a pragmatic way of integrating analytics, written in Python, with our distributed anomaly detection platform, written in Java. Background Warden is the distributed anomaly detection platform at Pinterest.

Algorithm 87

Algorithm Java Python Software Engineer 87

Pinterest Engineering

JANUARY 9, 2024

Advertisers can choose conversions as a campaign objective and inspire people to take specific actions like online checkouts, increased signups, or stronger leads. Advertisers can choose conversions as a campaign objective and inspire people to take specific actions like online checkouts, increased signups, or stronger leads.

Machine Learning 76

Machine Learning Architecture Engineering Utilities 76

Snowflake

AUGUST 30, 2023

In this blog post, we will delve into six such capabilities – comprehensive cross-cloud replication, zero copy database and schema clone, collation support, stored procedures, multi-table transactions, and transparent online upgrade – that every enterprise must consider while choosing their data platforms. What makes it truly effective?

Scala 88

Scala Government Database Cloud 88

DoorDash Engineering

FEBRUARY 7, 2023

DoorDash’s expansion from food delivery into new verticals like convenience and grocery introduced a number of new business challenges that would need to be supported by our technical stack. This article will take a deep dive into the steps we took to migrate over to CockroachDB and how we ensured that migration was smooth and successful.

PostgreSQL 119

PostgreSQL Retail SQL Database 119

Data Engineering Weekly

DECEMBER 3, 2023

Github writes an excellent blog to capture the current state of the LLM integration architecture. The blog is an excellent read to understand late-arriving data, backfilling, and incremental processing complications. I like the 3G model with Guardrails, Guidelines & Gadget, which I’m sure I will use more often :-).

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Knowledge Hut

MARCH 28, 2024

However, we as users may also build their own virtual PCs using operating systems like Windows 10. In this blog we will see what virtualization is and how to enable virtualization in W indows 11 and 10. It enables fast virtual machine migration across hosts and multi-OS operation on the same host.

Systems 52

Systems Utilities Cloud Management 52

Data Engineering Weekly

MARCH 21, 2023

From editor #121, we took the following articles Oda: Data as a product at Oda Oda writes an exciting blog about “Data as a Product,” describing why we must treat data as a product, dashboard as a product, and the ownership model for data products. link] The blog highlights six key principles of the value creation of data.

Data 52

Data Utilities Data Engineering Data Engineer 52

Databand.ai

AUGUST 30, 2023

7 Data Testing Methods, Why You Need Them & When to Use Them Helen Soloveichik August 30, 2023 What Is Data Testing? Data testing involves the verification and validation of datasets to confirm they adhere to specific requirements. This is part of a series of articles about data quality. In this article: Why Is Data Testing Important?

Data Validation 52

Data Validation Data Integration Data Database 52

LinkedIn Engineering

MARCH 27, 2023

In this blog, we’ll provide a high-level overview of the platform and its evolution over time. At LinkedIn, one of our most important goals is to offer a top-notch experience for all of our customers. Purchasing is a big part of that customer experience, and it is not as straightforward as many people would think.

Architecture 78

Architecture Coding Project Systems 78

Data Engineering Weekly

MARCH 19, 2023

The blog discusses implementing Type-2 SCD modeling and strategies to generate surrogate keys and bridge tables to handle many-to-many relationships. What happens in some core models like “customers” where more than one dimension change requires tracking? link] Sanjeev Mohan: What Exactly is a Data Product?

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Cloudera

NOVEMBER 1, 2022

In the private sector, excluding highly regulated industries like financial services, the migration to the public cloud was the answer to most IT modernization woes, especially those around data, analytics, and storage. It’s no secret that IT modernization is a top priority for the US federal government.

Cloud 86

Cloud Government Cloud Computing Data Architecture 86

Data Engineering Weekly

APRIL 2, 2023

Come and hear talks from companies like StarTree, Confluent, LinkedIn, DoorDash, Imply, and Uber on how they are advancing the state-of-the-art in user-facing analytics delivered instantly. Now you can win $1,000 cash by contributing a Transformation to our open-source library. Go to rtasummit.com and register with DEW30 for 30% off.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Weekly

OCTOBER 1, 2023

link] Agoda: How to Design and Maintain a High-Performing Data Pipeline I liked this article because it covers the basic 101 of data pipelines on maintaining a healthy infrastructure. The blog talks about the current state of the recsys, ongoing work, and the basic user model recommendation. See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

LinkedIn Engineering

APRIL 25, 2023

In our previous blog post on GraphQL, we explained how LinkedIn uses GraphQL to expedite the process of onboarding new use-cases for external API partners. In this blog post, we will cover how the GraphQL layer is architected for use by our internal engineers to build member and customer facing applications.

Architecture 109

Architecture Metadata Java Transportation 109

DoorDash Engineering

JANUARY 23, 2024

DoorDash has grown from executing simple restaurant deliveries to working with a wide variety of businesses, ranging from grocery and retail to parcels and pet supplies. Each business faces its own set of constraints as it strives to meet its goals.

Architecture 74

Architecture Metadata Bytes Systems 74

Robinhood

FEBRUARY 7, 2024

In this blog, we explore the evolution of our in-house batch processing infrastructure and how it helps Robinhood work smarter. In this blog, we explore the evolution of our in-house batch processing infrastructure and how it helps Robinhood work smarter. Authored by: Grace L., and Sreeram R. Authored by: Grace L., and Sreeram R.

Process 75

Process Hadoop Architecture Accessible 75

Knowledge Hut

APRIL 26, 2024

Dataset files may be in the form of excel sheets containing rows and columns, bunch of images, videos and audios, in the form of Text like words, sentences and paragraphs, in the form of numbers or values, messages, chats, statuses and in the form of different files like word, txt, pdf, xml and s o on.

Datasets 98

Datasets Machine Learning Deep Learning Finance 98

Tweag

FEBRUARY 8, 2023

Like many members of the Nix community – more than 25% according to the Nix community survey – I rely on macOS to provide me a stable shiny desktop environment. Like many members of the Nix community – more than 25% according to the Nix community survey – I rely on macOS to provide me a stable shiny desktop environment.

Systems 100

Systems Building Cloud IT 100

Cloudera

AUGUST 22, 2021

So, what do you like most about the cloud? It seems like there’s always something new to look out for every day, be it new use cases, technology or ways that organizations can benefit. Retirement of legacy tech and migration to new services can be challenging for any business, but greater for enterprises. That’s so interesting!

Cloud 104

Cloud Telecommunication Insurance Manufacturing 104

Tweag

DECEMBER 6, 2023

We had to move it to the main stage to provide enough room for all attendees. We had to move it to the main stage to provide enough room for all attendees. Gunnar Wagenknecht (Salesforce) gave us a detailed account of the scale of the Bazel migration at Salesforce. It was really good actually.

Bytes 56

Bytes Coding Building Systems 56

DoorDash Engineering

JANUARY 16, 2024

In this blog post, we describe the journey DoorDash took using a service mesh to realize data transfer cost savings without sacrificing service quality. For singleton services or those not migrated to the cell architecture, deployment occurs in a singular global cell. All microservice pods are deployed in multiple isolated cells.

Bytes 84

Bytes Cloud Management PostgreSQL 84

Cloudera

SEPTEMBER 28, 2020

Cloudera has found that customers have spent many years investing in their big data assets and want to continue to build on that investment by moving towards a more modern architecture that helps leverage the multiple form factors. Sentry to Ranger migration tools. Navigator to atlas migration, Improved performance and scalability.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Cloudera

JUNE 22, 2023

It permits transparent access to distributed data irrespective of its location, ensuring interoperability with legacy systems, breaking down legacy silos by enabling financial services to selectively migrate non-sensitive data to cost-effective cloud infrastructure. The top-line benefits of a hybrid data platform include: Cost efficiency.

Data Cleanse 65

Data Cleanse Data Governance Unstructured Data Cloud Storage 65

Lyft Engineering

JANUARY 30, 2023

These model-based decisions include price optimization for rides, incentives allocation for drivers, fraud detection, ETA prediction, and innumerable others that impact how riders move and drivers earn. Making real-time inferences with machine learning (ML) models at scale is complex. Constraints imposed by our legacy system.

Machine Learning 92

Machine Learning Systems Python Coding 92

Zalando Engineering

DECEMBER 18, 2023

This blog post will deep dive into how to use modules and a common issue for Testcontainers with Golang. Services often use external dependencies like datastore or queues. What we use it for? Starting a container with the image of the dependency is a convenient way to verify that the application works as expected. Docker, Podman.)

Java 48

Java Coding IT 48

LinkedIn Engineering

APRIL 11, 2023

In this blog post, we’ll discuss some of the challenges we faced with JSON and the process we used to evaluate new solutions and ultimately move forward with Google Protocol Buffers (Protobuf) as a replacement. Microservices arranges our platforms as a collection of loosely coupled services, communicating through lightweight protocols.

Programming Language 91

Programming Language Java Scala Programming 91

LinkedIn Engineering

JANUARY 16, 2024

In a previous blog post, we talked about how we built our anti-abuse platform using CASAL. In a previous blog post, we talked about how we built our anti-abuse platform using CASAL. In this blog post, we'll go deeper into how we manage account restrictions. We recognize that not all restrictions are created equal.

Kafka 84

Kafka Relational Database Java Architecture 84

Data Engineering Weekly

MAY 21, 2023

A couple of thing stands out for me in the blog. The Event store is moving from S3/ parquet storage to DeltaLake storage—a sign of LakeHouse format adoption across the board. The Event store is moving from S3/ parquet storage to DeltaLake storage—a sign of LakeHouse format adoption across the board.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Data Engineering Weekly

JANUARY 29, 2023

The data observability solutions require more attention in the remediation workflow to ensure they don’t end up as a disjointed workflow like Data Catalogs. Upsolver SQLake lets you process fast-moving data by simply writing a SQL query. Sign up free to test out the tool today. Please try it out and let us know what you build!!!

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52