Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

DoorDash Engineering

NOVEMBER 7, 2023

When users fail to complete a purchase after adding items to their carts, we send push notifications such as the one shown in Figure 1 to remind them that their orders are still pending. Figure 1: An example of a real-time push notification to prompt purchase completion.

Designing 95

Designing Coding Process Management 95

Snowflake

MARCH 2, 2023

That proves to be a difficult task for data engineering teams that have to manage separate infrastructure for batch data and streaming data. To address this challenge, we are happy to announce the public preview of Snowpipe Streaming as the latest addition to our Snowflake ingestion offerings. How does Snowpipe Streaming work?

Kafka 117

Kafka Data Ingestion Data Pipeline Cloud Storage 117

Knowledge Hut

JANUARY 16, 2024

” ― Denis Waitley What are SAFe Agile Events (or) Ceremonies? The key link between these levels is the SAFe-specific events which help with smooth value delivery facilitation. The events help with alignment across teams, ARTs, etc thus helping in managing risk by providing a level-based cadence and synchronization.

Portfolio 97

Portfolio Business Analyst Programming Certification 97

Netflix Tech

AUGUST 1, 2022

A Data Movement and Processing Platform @ Netflix By Bo Lei , Guilherme Pires , James Shao , Kasturi Chatterjee , Sujay Jain , Vlad Sydorenko Background Realtime processing technologies (A.K.A stream processing) is one of the key factors that enable Netflix to maintain its leading position in the competition of entertaining our users.

Process 108

Process Transportation Kafka Entertainment 108

DoorDash Engineering

OCTOBER 17, 2023

Figure 1: If we have two groups that are expected to have a distribution of 50/50, we expect the SRM check would pass if that 50/50 split is indeed observed. Cautionary tales of faux gains and real losses Example 1: The $10 Million Mirage Imagine that your target is to improve weekly revenue per user.

Education 87

Education Kafka Algorithm Data Warehouse 87

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters. Kafka is the pub-sub layer of the Iguazu pipeline, which provides real-time event delivery at DoorDash. Terraform : Infrastructure-as-code (IaC) platform.

Kafka 90

Kafka Programming Language Metadata Architecture 90

Lyft Engineering

OCTOBER 3, 2023

It is now the norm for tech companies to have a well-developed data platform. We will then introduce our in-house product, Verity, and showcase how it serves as a central platform for ensuring data quality in our Hive Data Warehouse. In future posts we will discuss how Verity addresses data quality elsewhere in our data platform.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. It also allowed us to optimize for handling time-series data and event data at scale. For new platform engineers, it presented a steeper learning curve.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Data Engineering Weekly

MAY 16, 2023

I won’t bore you with the importance of data quality in the blog. Theories of the Data Pipeline 1. Like the staging environment, Fronting Kafka receives all the events without validation. The event routers typically follow a few characteristics Event Routers can broadcast the same events from one-to-many destinations.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Pinterest Engineering

MAY 2, 2023

At Pinterest, most of our streaming jobs are built on top of Apache Flink , because Flink is a mature streaming framework with a lot of adoption in the industry. We set up a separate dataset for each event type indexed by our system, because we want to have the flexibility to scale these datasets independently.

Lambda Architecture 65

Lambda Architecture Datasets Software Engineer Software Engineering 65

Netflix Tech

MAY 4, 2023

Migrating Critical Traffic At Scale with No Downtime — Part 1 Shyam Gala , Javier Fernandez-Ivern , Anup Rokkam Pratap , Devang Shah Hundreds of millions of customers tune into Netflix every day, expecting an uninterrupted and immersive streaming experience.

Utilities 135

Utilities Systems Architecture Coding 135

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®.

Kafka 94

Kafka Cloud Data Pipeline PostgreSQL 94

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 55

Data Ingestion Kafka Java Data Pipeline 55

Data Engineering Weekly

MARCH 24, 2023

The real-world scenario is much more complex than this, but for the scope of this blog, let’s keep the ride-sharing business process into three simple steps. Data engineering starts to add value to the business by capturing events at each step of the business process. Event Sourcing Change Data Capture [CDC] Outbox pattern 1.

Engineering 67

Engineering Data Transportation Database 67

Knowledge Hut

MARCH 19, 2024

I will explore the top 10 AWS applications and their use cases in this blog. It is a scalable and feature-complete cloud platform that helps companies of all sizes to take advantage of cloud computing. Now let us see some top AWS applications list and their use cases: 1. What is AWS? Amazon EC2 (Elastic Compute Cloud).

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Pinterest Engineering

SEPTEMBER 29, 2023

Sanchay Javeria | Software Engineer, Ads Data Infrastructure To support metrics reporting for ads from external advertisers and real-time ad budget calculations at Pinterest, we run streaming pipelines using Apache Flink. To narrow down the scope of the problem, we found whether it was caused by a platform bug or an application logic issue.

Utilities 93

Utilities Coding Kafka Engineering 93

DoorDash Engineering

APRIL 4, 2023

This is the third blog post in a series of articles showcasing our 2022 summer intern projects. The Revenue Platform (RP) team is attempting to mitigate these issues by providing mechanisms for recording financial transactions in a compliant and auditable way that is amenable to accounting and reporting.

Project 74

Project Banking Kafka Database 74

Cloudera

JANUARY 17, 2024

Cloudera DataFlow for the Public Cloud (CDF-PC) is a complete self-service streaming data capture and movement platform based on Apache NiFi. In this blog we will dive into how CDF-PC’s support for NiFi reporting tasks can be used to monitor key metrics in Prometheus and Grafana. Supported values are “true” and “false.”

Bytes 101

Bytes Architecture Designing Building 101

Cloudera

JULY 18, 2022

In part 1 of this blog we discussed how Cloudera DataFlow for the Public Cloud (CDF-PC), the universal data distribution service powered by Apache NiFi, can make it easy to acquire data from wherever it originates and move it efficiently to make it available to other applications in a streaming fashion. Data decays!

Process 85

Process Kafka Scala SQL 85

Picnic Engineering

MARCH 28, 2023

Introduction We continue our story on the Analytics Platform setup in Picnic. Therefore the Analytics Platform team was looking into how to help and drive this project to success as well. That’s why the Analytics Platform team has developed a new data pipeline: The FCA pipeline. FCA is Picnic’s own automated fulfillment center.

Kafka 52

Kafka Transportation Data Warehouse Database 52

Knowledge Hut

MARCH 19, 2024

In this blog, Here we are to help you choose the right career. Before you select a course, consider the following: 1. Career Options After 12th Science The scientific stream provides a vibrant professional environment rich in specializations and chances for exploration. How Do I Choose the Right Course After 12th Science?

Medical 98

Medical Hospitality Education Healthcare 98

Netflix Tech

AUGUST 30, 2021

By Benson Ma , Alok Ahuja Introduction At Netflix, hundreds of different device types, from streaming sticks to smart TVs, are tested every day through automation to ensure that new software releases continue to deliver the quality of the Netflix experience that our customers enjoy. In this blog post, we will focus on the latter feature set.

Management 85

Management Kafka Transportation Cloud 85

Cloudera

JUNE 5, 2023

This blog is to congratulate our winner and review the top submissions. Cloudera VP of Engineering and NiFi founder Joe Witt, joined by principal committers Mark Payne and Matt Gillman, addressed the global community via a virtual event dubbed “ Meet the Committers.” On the verge of the release of NiFi 2.0,

Google Cloud 73

Google Cloud Cloud Storage Data Lake Data Pipeline 73

Confluent

FEBRUARY 27, 2019

This is important as each platform has different persistence models and APIs, tying you to that vendor’s ecosystem. What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? The key to event-first systems design is understanding that a series of events captures behavior.

Kafka 109

Kafka AWS Architecture Cloud 109

Rockset

MARCH 27, 2023

This is part two in Rockset’s Making Sense of Real-Time Analytics on Streaming Data series. In part 1 , we covered the technology landscape for real-time analytics on streaming data. In this post, we’ll explore the differences between real-time analytics databases and stream processing frameworks. So do you need just one?

Database 52

Database Process Scala SQL 52

Rockset

MAY 3, 2023

Rockset is a database used for real-time search and analytics on streaming data. In scenarios involving analytics on massive data streams, we’re often asked the maximum throughput and lowest data latency Rockset can achieve and how it stacks up to other databases. lower latency than Elasticsearch for streaming data ingestion.

Data Ingestion 59

Data Ingestion Kafka Database Architecture 59

Knowledge Hut

JANUARY 18, 2024

Using these high-tech tools, I can explore economic trends, figure out patterns, and predict future events. Get to know more about measures of dispersion through our blogs. Apache Spark Apache Spark is a batch-processing framework with the capability of stream processing and making it a hybrid framework.

Data Process 52

Data Process Process Hadoop Scala 52

LinkedIn Engineering

DECEMBER 12, 2023

Lastly, we discuss the analytics platform at LinkedIn, LEIA, which enables PEDAL to scale across multiple analytics products. Less stringent privacy baselines have been proposed in the literature, such as event-level privacy which would consider the privacy of each new view in our setting.

Algorithm 77

Algorithm Metadata SQL Datasets 77

LinkedIn Engineering

DECEMBER 20, 2023

A key part of this work is our content abuse defense systems, which function behind the scenes to help detect and remove policy-violating content from the platform, while helping surface professional, relevant, and meaningful content that adds value to the member experience.

Machine Learning 90

Machine Learning Datasets Algorithm Architecture 90

Pinterest Engineering

JULY 11, 2023

Divye , Teja , Chen , Sam , Lu , Heng , Kanchi , Rainie , Dinesh , Ashish , Nishant , Pooja | Stream Processing Platform Team At Pinterest, stream data processing powers a wide range of real-time use cases. In addition, incidents and unexpected influx of events mean that any job might want to burst at any point of time.

AWS 70

AWS Utilities Transportation Engineering 70

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108

Data Engineering Weekly

DECEMBER 24, 2023

My take on the rising trends 🟡 Intelligent Data Platform: Yes, I fully agree. No 1 rule of the product experience is “Don’t make the user think”; for me, “prompting” makes me think a lot. The blog explores how Google uses ML to improve the efficiency of ML workloads!

Data Engineering 62

Data Engineering Data Engineer Engineering Deep Learning 62

LinkedIn Engineering

MAY 9, 2023

In the past, we addressed latency, throughput and cost issues by migrating off Oracle onto Espresso , an open-source document platform, and adding more nodes. In this blog post we’ll discuss our decision to leverage Couchbase, the challenges that arose, and how we addressed each challenge in our final solution.

Database 133

Database Project Datasets Designing 133

Cloudera

JULY 21, 2022

Rill’s services and platform ensure the performance, reliability, and security required to meet the most demanding SLAs. . Figure 1: Rill and Cloudera Architecture. Druid’s native support for ingesting data from Apache Kafka allows it to stream data from Cloudera DataFlow to Rill’s fully managed Druid service.

BI 82

BI Digital Media Data Warehouse Kafka 82

Confluent

JUNE 26, 2019

In these projects, microservice architectures use Kafka as an event streaming platform. These are joined together with events, creating a unidirectional dependency graph that decouples each bounded context from those that arise downstream, to create rich event streaming business applications. Microservices.

Kafka 109

Kafka Designing Architecture ETL Tools 109

Rockset

MARCH 26, 2023

Folks have definitely tried, and while Apache Kafka® has become the standard for event-driven architectures, it still struggles to replace your everyday PostgreSQL database instance in the modern application stack. At Confluent I talked often about the fanciful sounding “Stream and Table Duality”.

Cloud 52

Cloud PostgreSQL Kafka Database 52