Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Real-time Ingestion Events from our real-time analytics pipeline were configured to be sent into our internal Flink application, streamed to Kafka, and written into Druid.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Pinterest Engineering

NOVEMBER 7, 2023

A central component of data ingestion infrastructure at Pinterest is our PubSub stack, and the Logging Platform team currently runs deployments of Apache Kafka and MemQ. years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 100

Kafka Java Software Engineer Software Engineering 100

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 56

Data Ingestion Kafka Java Data Pipeline 56

Data Engineering Weekly

MAY 16, 2023

I won’t bore you with the importance of data quality in the blog. Data Testing vs. Data Observability Data testing and data observability are two important aspects of data quality. Data testing ensures that data meets specific requirements. The Fronting Kafka pattern follows a two-cluster approach.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

JANUARY 31, 2024

Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. Introduction Kafka as an event streaming component can be applied to a wide variety of use cases. and its use is no longer recommended.

Management 69

Management Kafka Big Data Cloud 69

Cloudera

MAY 1, 2023

By breaking down monolithic apps into microservices architectures, for example, or making modularized data products, organizations do their best to enable more rapid iterative cycles of design, build, test, and deployment of innovative solutions. release of Cloudera’s SQL Stream Builder (available on CDP Public Cloud 7.2.16

SQL 78

SQL Project Kafka Accessible 78

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Data Engineering Weekly

SEPTEMBER 24, 2023

The blog talks about the limitations of rule engines and how LLM can enrich additional context to make the rule engine more effective. The blog specifically focuses on the following areas of LLM implementations. The blog is an excellent read about the “Scale First, Learn Last” problem in the data processing world.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Cloudera

JUNE 28, 2022

In a previous blog of this series, Turning Streams Into Data Products , we talked about the increased need for reducing the latency between data generation/ingestion and producing analytical results and insights from this data. Building real-time streaming analytics data pipelines requires the ability to process data in the stream.

Process 85

Process Kafka SQL Machine Learning 85

Netflix Tech

AUGUST 30, 2021

By Benson Ma , Alok Ahuja Introduction At Netflix, hundreds of different device types, from streaming sticks to smart TVs, are tested every day through automation to ensure that new software releases continue to deliver the quality of the Netflix experience that our customers enjoy.

Management 85

Management Kafka Transportation Cloud 85

Cloudera

SEPTEMBER 28, 2020

They have dev, test, and production clusters running critical workloads and want to upgrade their clusters to CDP Private Cloud Base. Modernize their architecture to ingest data in real-time using the new streaming features available in CDP Private Cloud Base in order to make the data available to their users quickly. Background: .

Cloud 132

Cloud Kafka Professional Services Metadata 132

Lyft Engineering

JUNE 28, 2023

However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others. While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort.

Machine Learning 116

Machine Learning Building Metadata Kafka 116

Knowledge Hut

JANUARY 18, 2024

Get to know more about measures of dispersion through our blogs. Apache Spark Apache Spark is a batch-processing framework with the capability of stream processing and making it a hybrid framework. It also supports a pseudo-distributed local mode that can be used for development or testing. What Are Big Data Frameworks?

Data Process 52

Data Process Process Hadoop Scala 52

Data Engineering Weekly

JANUARY 21, 2024

A thorough quickstart guide, created in partnership with Snowflake, is available, complete with a sample dataset so you can test-drive the tool. link] Grab: Kafka on Kubernetes: Reloaded for fault tolerance. Grab Engineering writes about enhancing its Kafka on the Kubernetes platform for improved fault tolerance.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Cloudera

FEBRUARY 22, 2023

Cloudera’s SQL Stream Builder (SSB) is a versatile platform for data analytics using SQL. As apart of Cloudera Streaming Analytics it enables users to easily write, run, and manage real-time SQL queries on streams with a smooth user experience, while it attempts to expose the full power of Apache Flink. Our TOTZ UDF did the job!

SQL 84

SQL Raw Data Programming Language Kafka 84

Cloudera

JANUARY 19, 2022

We are excited to announce the release of Cloudera Streaming Analytics (CSA) 1.6 Cloudera SQL Stream Builder was initially released in CSA 1.3. But we want to address one question that keeps coming up – does CSA CE make sense as my main development environment for stream processing jobs? for CDP Private Cloud Base.

Java 89

Java Kafka SQL Cloud 89

Data Engineering Weekly

FEBRUARY 12, 2023

Sign up free to test out the tool today. To begin, we partner with Pathway.com to launch a three-part series about unlocking stream processing , - where Pathway talks about applying linear regression & classification in real-time. I believe the current data testing platforms can’t support the complex nature of data testing.

Data Engineering 52

Data Engineering Data Engineer Engineering Hadoop 52

DoorDash Engineering

OCTOBER 17, 2023

SRM represents one of the most egregious data quality issues in A/B tests because it fundamentally compromises the basic assumption of random assignment. Statistical approaches for identifying imbalance The most common approach for identifying SRM is to use a chi-square test that can quickly detect when something is wrong.

Education 87

Education Kafka Algorithm Data Warehouse 87

Snowflake

AUGUST 16, 2023

Read our Summit recap blog for highlights across industries or watch Summit sessions now on-demand. Developers can now start building and testing Snowflake Native Apps in their accounts in AWS. Data comes in a continuous manner, and often a separate architecture is required to handle streaming data. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Cloudera

JULY 21, 2022

Cloudera users can securely connect Rill to a source of event stream data, such as Cloudera DataFlow , model data into Rill’s cloud-based Druid service, and share live operational dashboards within minutes via Rill’s interactive metrics dashboard or any connected BI solution. Native streaming ingestion support from Kafka and Kinesis.

BI 86

BI Digital Media Data Warehouse Kafka 86

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Rockset

MAY 3, 2023

Rockset is a database used for real-time search and analytics on streaming data. In scenarios involving analytics on massive data streams, we’re often asked the maximum throughput and lowest data latency Rockset can achieve and how it stacks up to other databases. lower latency than Elasticsearch for streaming data ingestion.

Data Ingestion 59

Data Ingestion Kafka Database Architecture 59

dbt Developer Hub

NOVEMBER 3, 2022

Here are a few things we’ll address: Why you may want to use an architecture like this How to structure your event messages How to use dbt macros to make it easy to ingest new event streams Event Streams at Merit ​ At Merit, we’re building the leading verified identity platform. One key focus of our platform is data quality.

Kafka 52

Kafka ETL Tools BI Database 52

Cloudera

MARCH 14, 2023

In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities. In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development.

Data Pipeline 84

Data Pipeline Designing Kafka Metadata 84

LinkedIn Engineering

JANUARY 19, 2024

Apache Kafka - A distributed event streaming platform. The requests and processing metadata are streamed over Kafka. Apache Samza - A distributed stream processing framework. The request endpoint, entity processors are exposed over Rest.li.

Recruitment 88

Recruitment Data Process Process Kafka 88

Confluent

JUNE 12, 2019

Confluent Cloud, a fully managed event cloud-native streaming service that extends the value of Apache Kafka ® , is simple, resilient, secure, and performant, allowing you to focus on what is important—building contextual event-driven applications, not infrastructure. KSQL and Kafka Connect example. and Helm/Tiller 2.8.2+

Cloud 93

Cloud Kafka Healthcare Software Engineer 93

Data Engineering Podcast

APRIL 19, 2020

In this episode Eventador Founder and CEO Kenny Gorman describes how the platform is architected, the challenges inherent to managing reliable streams of data, the simplicity offered by a SQL interface, and the interesting projects that his customers have built on top of it. How does it fit into an application architecture?

Building 100

Building PostgreSQL MongoDB SQL 100

Cloudera

OCTOBER 4, 2022

This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes.

Kafka 79

Kafka Data Cloud Big Data 79

dbt Developer Hub

AUGUST 2, 2023

Jetblue has outgrown lambda views due to performance constraints (a view can only be so performant) and we are at another milestone in dbt’s journey to support streaming. For the sake of simplicity, when I refer to materialized views in this blog, I mean dynamic tables in Snowflake. So, what does that mean?

Datasets 52

Datasets Kafka SQL Cloud 52

Confluent

APRIL 10, 2019

Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites. First and foremost, Confluent Control Center can manage multi-datacenter Apache Kafka ® deployments, whether on prem or in the cloud.

Kafka 86

Kafka Metadata Java Cloud 86

Rockset

JUNE 8, 2023

Streaming data adoption continues to accelerate - over 80% of Fortune 100 companies already use Apache Kafka - driven by organizations creating value by putting data to use in real time. Much of this streaming data will land in real-time analytics databases as event streams. No overprovisioning required.

Kafka 52

Kafka Algorithm Database Architecture 52

Data Engineering Weekly

NOVEMBER 13, 2022

Sign up free to test out the tool today. The blog highlights the need for a longer training window time for the neural ranking model compared to the decision tree model and the need for high-quality backward compatible historical events. This blog is an excellent overview of incorporating a data quality check with Airflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

LinkedIn Engineering

JANUARY 16, 2024

In a previous blog post, we talked about how we built our anti-abuse platform using CASAL. In this blog post, we'll go deeper into how we manage account restrictions. Espresso’s tight integration with LinkedIn’s Brooklin –a near real-time data streaming framework–enabled seamless data streaming through Kafka messages.

Kafka 84

Kafka Relational Database Java Architecture 84

Data Engineering Weekly

JULY 9, 2023

link] Capital One: Democratizing machine learning It is an exciting blog post + video interview from Capital One focusing on the people and technology aspect of democratizing the machine learning practice across the org. The blog narrates the optimization and cluster architectural changes by ByteDance on top of ClickHouse.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Data Engineering Weekly

FEBRUARY 5, 2023

Sign up free to test out the tool today. The blog is an excellent overview of problems with floating points and integer data types. Streaming plus batch unified in a single platform. Applying data engineering for system performance is one of my favorites, so the blog enjoyed reading this article.

Data Engineering 57

Data Engineering Data Engineer Engineering Food 57

Data Engineering Podcast

DECEMBER 9, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. Who uses Spark? What are the tools offered to Spark users?

Scala 100

Scala MySQL Kafka Hadoop 100