Zalando Engineering

JULY 12, 2017

While developing Zalando’s real-time business process monitoring solution, we encountered the need to generate complex events upon the detection of specific patterns of input events. In this blog post we describe the generation of such events using Apache Flink, and share our experiences and lessons learned in the process.

Process 40

Process Kafka AWS Architecture 40

ProjectPro

JULY 23, 2021

AWS (Amazon Web Services) is the world’s leading and widely used cloud platform, with over 200 fully featured services available from data centers worldwide. Real-time Data Processing Application 7. Sentiment Analysis on Real-time Twitter Data 23. AWS Athena Big Data Project for Querying COVID-19 Data 25.

AWS 52

AWS Project Amazon Web Services Cloud Computing 52

Lyft Engineering

JUNE 6, 2023

Since most of our pipelines are based on Apache Beam and with pipeline code in Python, we use Pyflame and async profiler to generate a heat map of CPU utilization. Flink Dashboard Lately, the Flink console has added visual tools that show various performance data points in order to identify performance bottlenecks.

Utilities 123

Utilities Coding Python Systems 123

Knowledge Hut

MAY 3, 2024

A new breed of ‘Fast Data’ architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. Dean Wampler (Renowned author of many big data technology-related books) Dean Wampler makes an important point in one of his webinars.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Towards Data Science

FEBRUARY 18, 2024

How to Stream and Apply Real-Time Prediction Models on High-Throughput Time-Series Data Photo by JJ Ying on Unsplash Most of the stream processing libraries are not python friendly while the majority of machine learning and data mining libraries are python based. This design enables the re-reading of old messages.

Kafka 80

Kafka Python Process Google Cloud 80

Pinterest Engineering

JULY 11, 2023

Divye , Teja , Chen , Sam , Lu , Heng , Kanchi , Rainie , Dinesh , Ashish , Nishant , Pooja | Stream Processing Platform Team At Pinterest, stream data processing powers a wide range of real-time use cases. Our Flink clusters are multitenant and run jobs that concurrently process more than 20M msgs/sec across 12 clusters.

AWS 72

AWS Utilities Transportation Engineering 72

LinkedIn Engineering

MARCH 23, 2023

Co-Authors: Yuhong Cheng , Shangjin Zhang , Xinyu Liu, and Yi Pan Efficient data processing is crucial in reducing learning curves, simplifying maintenance efforts, and decreasing operational complexity. In this blog post, we will share our progress, challenges, and lessons learned from implementing Apache Beam.

Process 97

Process Lambda Architecture Kafka Datasets 97

Towards Data Science

APRIL 30, 2024

Balancing correctness, latency, and cost in unbounded data processing Image created by the author. Intro Google Dataflow is a fully managed data processing service that provides serverless unified stream and batch data processing. Apache Beam lets users define processing logic based on the Dataflow model.

Google Cloud 63

Google Cloud Process Cloud Lambda Architecture 63

Pinterest Engineering

MAY 2, 2023

At Pinterest, most of our streaming jobs are built on top of Apache Flink , because Flink is a mature streaming framework with a lot of adoption in the industry. However, when we process a new event for a user, we do not want to read the existing N events, update them, and then write them all back to the respective dataset.

Lambda Architecture 66

Lambda Architecture Datasets Software Engineer Software Engineering 66

AltexSoft

JULY 29, 2022

Depending on how you measure it, the answer will be 11 million newspaper pages or… just one Hadoop cluster and one tech specialist who can move 4 terabytes of textual data to a new location in 24 hours. Developed in 2006 by Doug Cutting and Mike Cafarella to run the web crawler Apache Nutch, it has become a standard for Big Data analytics.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

Netflix Tech

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Process 84

Process Data Pipeline Datasets Aggregated Data 84

DoorDash Engineering

APRIL 4, 2023

You can read about each project below. There are various challenges with respect to data mutability, long intervals between changes, and data fixes. The primary purpose of Archiver is to pick up events containing financial transaction data sent by different upstream teams and store this data in a data lake.

Project 74

Project Banking Kafka Database 74

Zalando Engineering

MARCH 30, 2016

Saiki is Zalando’s next generation data integration and distribution platform in a world of microservices. Saiki ingests data generated by operational systems and makes it available to analytical systems. Zalando’s operational systems continuously produce events and publish those to Saiki’s unified log (Apache Kafka).

Kafka 40

Kafka Data Lake Data Warehouse Business Intelligence 40

Rockset

APRIL 5, 2022

Recently, Imply published revised Apache Druid results on the SSB with denormalized numbers. Leveraging more Single Instruction/Multiple Data (SIMD) instructions as part of the vectorized execution engine to take advantage of higher throughput offered by modern processors.

Datasets 52

Datasets Metadata Kafka Database 52

ProjectPro

JANUARY 31, 2023

If you're looking to break into the exciting field of big data or advance your big data career, being well-prepared for big data interview questions is essential. Get ready to expand your knowledge and take your big data career to the next level! “Data analytics is the future, and the future is NOW!

Big Data 52

Big Data Hadoop AWS Relational Database 52

Cloudera

AUGUST 26, 2020

On August 18, we completed our Enterprise Data Cloud vision of bringing a truly hybrid cloud experience with the general availability of Cloudera Data Platform Private Cloud (CDP Private Cloud). Informatica and Cloudera deliver a proven set of solutions for rapidly curating data into trusted information.

Machine Learning 80

Machine Learning BI Big Data Data Warehouse 80

Rock the JVM

FEBRUARY 5, 2023

Enter Giannis: Flink SQL is a powerful high level API for running queries on streaming (and batch) datasets. Batch SQL Queries operate on static data, i.e. on data stored on disk, already available and the results are considered complete. Streaming (and Batch) SQL 1.1 Streaming (and Batch) SQL 1.1

SQL 61

SQL Kafka Metadata Database 61