Zalando Engineering

NOVEMBER 8, 2023

Postgres Logical Replication at Zalando Builders at Zalando have access to a low-code solution that allows them to declare event streams that source from Postgres databases. At the time of writing, there are hundreds of these Postgres-sourced event streams out in the wild at Zalando.

PostgreSQL 139

PostgreSQL Java Database Bytes 139

Rock the JVM

FEBRUARY 18, 2024

WebSockets are used in applications such as Instant Messaging, Gaming, Simultaneous editing, and stock tickers to mention but a few. In this article, we’ll be creating a chat application using Http4s’ implementation of WebSockets. Let’s start by creating an HTML page that will be served to access the application.

Scala 94

Scala Coding Programming Accessible 94

Data Engineering Podcast

OCTOBER 22, 2023

Summary The primary application of data has moved beyond analytics. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. As more people start using AI for projects, two things are clear: It’s a rapidly advancing field, but it’s tough to navigate. With Materialize, you can!

BI 162

BI SQL Machine Learning Programming Language 162

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Data Engineering Podcast

SEPTEMBER 24, 2023

In this episode Louis Brandy discusses the applications for vector search capabilities both in and outside of AI, as well as the challenges of maintaining real-time indexes of vector data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

BI 147

BI SQL Python Machine Learning 147

Data Engineering Podcast

MAY 21, 2023

Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. Stream processing technologies have been around for around a decade.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

OCTOBER 8, 2023

Learn more about Datafold by visiting dataengineeringpodcast.com/datafold As more people start using AI for projects, two things are clear: It’s a rapidly advancing field, but it’s tough to navigate. How can you get the best results for your use case? With Materialize, you can! Closing Announcements Thank you for listening!

Insurance 162

Insurance BI SQL Machine Learning 162

Waitingforcode

JULY 6, 2023

That's often a dilemma, whether we should put multiple sinks working on the same data source in the same or in different Apache Spark Structured Streaming applications? Both solutions may be valid depending on your use case but let's focus here on the former one including multiple sinks together.

Data 130

Data 130

Engineering at Meta

MARCH 18, 2024

Systems and application logs play a key role in operations, observability, and debugging workflows at Meta. Logarithm is a hosted, serverless, multitenant service, used only internally at Meta, that consumes and indexes these logs and provides an interactive query interface to retrieve and view logs. in PyTorch).

Engineering 89

Engineering Metadata Architecture Designing 89

Lyft Engineering

JUNE 28, 2023

However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others. While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort.

Machine Learning 115

Machine Learning Building Metadata Software Engineer 115

Rock the JVM

DECEMBER 17, 2023

Introduction UDP stands for User Datagram Protocol, it’s a transport layer communication protocol built on top of IP that is used to transmit data across a network. In the last section, we’ll create a live audio streaming server with the knowledge we’ve gained. val scala3Version = "3.3.1" lazy val root = project. in ( file ( "." )).

Scala 90

Scala Bytes Java Coding 90

Netflix Tech

MARCH 5, 2024

More than half of our critical streaming video services are now running on JDK 21 with Generational ZGC, so it’s a good time to talk about our experience and the benefits we’ve seen. If you’re interested in how we use Java at Netflix, Paul Bakker’s talk How Netflix Really Uses Java , is a great place to start.

Java 91

Java Bytes Utilities Metadata 91

Snowflake

DECEMBER 5, 2023

To ensure data remains protected from unintended use, Snowflake Cortex (now in private preview) gives users access to industry-leading LLMs (e.g., Llama 2) to equip them to quickly analyze data and build differentiated AI applications.

AWS 93

AWS Amazon Web Services Government Cloud Computing 93

Data Engineering Podcast

JULY 23, 2023

To bring streaming data in reach of application engineers Matteo Pelati helped to create Dozer. In this episode he explains how investing in high performance and operationally simplified streaming with a familiar API can yield significant benefits for software and data teams together. Can you describe how Dozer is implemented?

Building 130

Building SQL Machine Learning Data Ingestion 130

Data Engineering Weekly

MARCH 31, 2024

The challenge multiplies when designing a singular AI/ML system amid rapid application and model advancements, from XGboost to deep learning recommendation and large language models. The blog narrates how Uber utilizes the existing infrastructure to support various AI/ML use cases.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Precisely

DECEMBER 28, 2023

The concept of streaming data was born of necessity. Business success is based on how we use continuously changing data. That’s where streaming data pipelines come into play. This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Knowledge Hut

APRIL 26, 2024

Machine learning uses algorithms that comb through data sets and continuously improve the machine learning model. A d ataset is used to draw better insights and get a clear picture of a particular problem statement. Datasets play a crucial role and are at the heart of all Machine Learning models.

Datasets 81

Datasets Machine Learning Deep Learning Finance 81

Towards Data Science

FEBRUARY 18, 2024

How to Stream and Apply Real-Time Prediction Models on High-Throughput Time-Series Data Photo by JJ Ying on Unsplash Most of the stream processing libraries are not python friendly while the majority of machine learning and data mining libraries are python based. An event is generated by a producer (e.g. online dashboard).

Kafka 79

Kafka Python Process Google Cloud 79

Rockset

MARCH 18, 2024

Flink is one of the most popular stream processing technologies, ranked as a top five Apache project and backed by a diverse committer community including Alibaba and Apple. Rockset customers using Flink often share how challenging it is to self-manage Flink for streaming transformations. What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Lyft Engineering

JUNE 6, 2023

Discover how Lyft identified and fixed performance issues in our streaming pipelines. Background Every streaming pipeline is unique. Since most of our pipelines are based on Apache Beam and with pipeline code in Python, we use Pyflame and async profiler to generate a heat map of CPU utilization.

Utilities 123

Utilities Coding Python Systems 123

LinkedIn Engineering

JUNE 26, 2023

We have various user experiences that let application teams provision their own resources and infrastructure, generally by filling out forms or using command-line tools. Just as important, a growing number of use cases are driven by developers building applications.

Data Pipeline 112

Data Pipeline Kafka MySQL SQL 112

Striim

DECEMBER 8, 2023

Note: To follow best practices guide, you must have the Persisted Streams add-on in Striim Cloud or Striim Platform. The more efficient approach is to have a single read client that pipes out to a stream, which is then ingested by multiple writers. Stream: Time-ordered log of events transferring data between processing components.

Kafka 64

Kafka Database Data Warehouse Data 64

Striim

APRIL 1, 2024

28% of enterprise applications are integrated, a recent study by Microsoft and MuleSoft found , highlighting a significant challenge for businesses aiming to effectively utilize their vast application ecosystems. What is Striim Cloud for Application Integration?

Hospitality 52

Hospitality Transportation Google Cloud Cloud 52

Striim

APRIL 3, 2024

deploy an average of 105 applications , with new applications continuously being adopted. This explosion in cloud application use has led to significant challenges in data integration and the delivery of insightful data to stakeholders. What is Striim Cloud for Application Integration? Enterprises in the U.S.

Google Cloud 52

Google Cloud Transportation Data Integration Aggregated Data 52

Rockset

MARCH 1, 2023

But, these two functions directly compete for the available compute resources, creating a fundamental limitation that makes it difficult to build efficient, reliable real-time applications at scale. When data ingestion has a flash flood moment, your queries will slow down or time out making your application flaky. This changes today.

Data Ingestion 144

Data Ingestion Database Architecture Cloud Storage 144

Rock the JVM

APRIL 24, 2024

Introduction In modern application development, efficient database interactions are crucial for building scalable and maintainable systems. One such powerful tool among others ( Doobie , Slick , Quill etc) is the Skunk Scala library , which provides a functional and typesafe interface for PostgreSQL databases access in Scala applications.

Scala 52

Scala Database PostgreSQL SQL 52

Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. This is crucial for use cases like market signaling and forecasting which benefit from, and depend upon, the most up-to-date information. An example of how we use Druid rollup at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Rockset

JANUARY 30, 2024

It also unlocks ways to make it easier and cheaper to build applications on Rockset. Microbatching : An option to microbatch ingestion based on the latency requirements of the use case. In this blog, we delve into each of these features and how they are giving users more cost controls for their search and AI applications.

Data Ingestion 64

Data Ingestion Utilities Architecture SQL 64

Workfall

JUNE 27, 2023

Reading Time: 7 minutes The world of creating real-time applications can get complex when we have to consider latency, fault tolerance, scalability, and possible data losses. Traditionally, web sockets were the go-to option when it came to real-time applications, but think of a situation whereby there’s server downtime.

Kafka 75

Kafka Architecture AWS Transportation 75

LinkedIn Engineering

DECEMBER 12, 2023

This then posed a challenging question: how to protect the privacy of the viewers of posts while still providing useful post analytics to the post author in real-time? Hence, we want to safeguard the privacy of members when they view posts while also providing useful analytics of viewers on their own posts.

Algorithm 77

Algorithm Metadata Datasets SQL 77

Netflix Tech

JUNE 5, 2023

In this article we explain our journey towards productizing a better viewing experience for our members by utilizing features and capabilities in consumer streaming devices. For content with a native frame rate of 24 fps, the source device needs to get 72 frames from the streaming application in a period of 3 seconds.

Algorithm 122

Algorithm Entertainment Manufacturing Data Science 122

Knowledge Hut

MARCH 19, 2024

It offers more than 170 AWS services to the developers so they can use them from anywhere when required. AWS Applications provide many services, from storage to serverless computing, and can be tailored to meet diverse business requirements. I will explore the top 10 AWS applications and their use cases in this blog.

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Snowflake

MARCH 26, 2024

And we heard from customers about how they’re using the Telecom Data Cloud to build a simplified data foundation that fast-tracks progress toward key goals such as operational efficiency, resilience and sustainability — and supports data strategies that meet the challenges of the future today. The bottom line?

Telecommunication 70

Telecommunication Data Consolidation Manufacturing Cloud 70

Netflix Tech

APRIL 27, 2023

Ruchir Jha , Brian Harrington , Yingwu Zhao TL;DR Streaming alert evaluation scales much better than the traditional approach of polling time-series databases. It allows us to overcome high dimensionality/cardinality limitations of the time-series database. It opens doors to support more exciting use-cases. OK, Results?

Database 115

Database Architecture Consulting Systems 115

Towards Data Science

APRIL 3, 2023

Building a Mental Model for Engineers and Anyone in Between Stream Processing can be handled gently and with care, or wildly, and almost out of control! How you transition from a batch mindset to a streaming mindset although can also be tricky, so let’s start small and build. You be the judge of what future you’d rather embrace.

Process 86

Process Data Lake Systems Bytes 86

Data Engineering Weekly

MARCH 24, 2024

Companies are more open to adopting Gen AI for their internal use cases but have reservations about rolling it out to their clients. link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

LinkedIn Engineering

MARCH 23, 2023

This, in turn, helps engineers to develop and deploy data processing applications quickly and easily, powering various business requirements, and enhancing member experience on LinkedIn. In the past, we often used lambda architecture for processing jobs, meaning that our developers used two different systems for batch and stream processing.

Process 97

Process Lambda Architecture Kafka Datasets 97