Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. This is crucial for use cases like market signaling and forecasting which benefit from, and depend upon, the most up-to-date information. An example of how we use Druid rollup at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Christophe Blefari

MARCH 17, 2023

For people that are not use to it it means last in, first out. Airflow alternatives meetup Click on the image to go the the LinkedIn event. We are organising next week with the Paris Apache Airflow Meetup group an online event to discuss about Airflow alternatives. We will host another event soon after with others.

Data 130

Data SQL Deep Learning Kafka 130

Snowflake

MARCH 2, 2023

But before data can be transformed and served or shared, it must be ingested from source systems. As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. This solution is both scalable and reliable, as we have been able to effortlessly ingest upwards of 1GB/s throughput.”

Kafka 117

Kafka Data Ingestion Data Pipeline Cloud Storage 117

Jesse Anderson

DECEMBER 12, 2022

Doug Cutting took those papers and created Apache Hadoop in 2005. They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. We lacked a scalable pub/sub system.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

FEBRUARY 21, 2023

SQL Stream Builder (SSB) is a versatile platform for data analytics using SQL as a part of Cloudera Streaming Analytics, built on top of Apache Flink. As an essential part of ETL, as data is being consolidated, we will notice that data from different sources are structured in different formats. This might be OK for some cases.

SQL 107

SQL Kafka Raw Data Data 107

Cloudera

JULY 18, 2022

In part 1 of this blog we discussed how Cloudera DataFlow for the Public Cloud (CDF-PC), the universal data distribution service powered by Apache NiFi, can make it easy to acquire data from wherever it originates and move it efficiently to make it available to other applications in a streaming fashion. Use case recap.

Process 85

Process Kafka Scala SQL 85

Data Engineering Weekly

AUGUST 6, 2023

😃🎊 Click here to Register for DEWCon - 2023 → 🤺Hot Take on The Great Orchestration Battle for Data Land ⚔️ Last week a sequence of blog posts revealed the underlying ambition of the data orchestration companies. But 4 years later, in 2023 — where has the data mesh gotten us?

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data.

Kafka 52

Kafka Banking Medical Healthcare 52

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

Knowledge Hut

JANUARY 18, 2024

For example, big data is used to pick out trends in economics, and those trends and patterns are used to predict what will happen in the future. Using these high-tech tools, I can explore economic trends, figure out patterns, and predict future events. Get to know more about measures of dispersion through our blogs.

Data Process 52

Data Process Process Hadoop Scala 52

Cloudera

JANUARY 31, 2024

Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. Introduction Kafka as an event streaming component can be applied to a wide variety of use cases. The MM1 behavior is crucial for some use-cases.

Management 68

Management Kafka Big Data Cloud 68

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. Some of the common supported data reprocessing use cases are listed below.

Management 94

Management Kafka Metadata Media 94

Data Engineering Weekly

FEBRUARY 4, 2024

[link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. It also reminds me there is no modern alternative to Secor.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Cloudera

JUNE 7, 2021

It enabled users to easily write, run and manage real-time SQL queries on streams from Apache Kafka with an exceptionally smooth user experience. . Improved Kafka and Schema Registry integration. Improved Kafka and Schema Registry integration. We have further streamlined the integration with Kafka and Schema Registry.

Kafka 100

Kafka SQL Accessible Accessibility 100

Monte Carlo

NOVEMBER 8, 2023

Kafka and Vector Database support According to Databricks’ State of Data and AI report , the number of companies using SaaS LLM APIs has grown more than 1300% since November 2022 with a nearly 411% increase in the number of AI models put into production during that same period.

Kafka 64

Kafka Database High Quality Data Datasets 64

Data Engineering Weekly

JULY 30, 2023

😄🎢🚀 High Scalability: Lessons Learned Running Presto At Meta Scale Presto, potentially ranking as one of the most influential open-source initiatives of the past ten years, stands shoulder to shoulder with the likes of Apache Kafka. But 4 years later, in 2023 — where has the data mesh gotten us?

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®.

Kafka 94

Kafka Cloud Data Pipeline PostgreSQL 94

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108

Confluent

JUNE 26, 2019

I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®. In these projects, microservice architectures use Kafka as an event streaming platform. Apache Kafka – An event streaming platform for microservices. This works well for many use cases.

Kafka 109

Kafka Designing Architecture ETL Tools 109

Cloudera

JULY 21, 2022

Cloudera has partnered with Rill Data, an expert in metrics at any scale, as Cloudera’s preferred ISV partner to provide technical expertise and support services for Apache Druid customers. We want Cloudera customers that rely on Apache Druid to know that their clusters are secure and supported by the Cloudera partner ecosystem.

BI 82

BI Digital Media Data Warehouse Kafka 82

Cloudera

JUNE 28, 2022

In a previous blog of this series, Turning Streams Into Data Products , we talked about the increased need for reducing the latency between data generation/ingestion and producing analytical results and insights from this data. This blog will be published in two parts. The use case. This is what we call the first-mile problem.

Process 80

Process Kafka SQL Machine Learning 80

Data Engineering Weekly

DECEMBER 24, 2023

🔴AI Agents: This is not possible until the data tools figure out a way to integrate into their workflow to use AI Agent instead of relying on users to enter the prompts. RudderStack just launched Trino as a Reverse ETL source. The blog explores how Google uses ML to improve the efficiency of ML workloads!

Data Engineering 62

Data Engineering Data Engineer Engineering Deep Learning 62

Cloudera

NOVEMBER 16, 2021

With the general availability of Cloudera DataFlow for the Public Cloud (CDF-PC) , our customers can now self-serve deployments of Apache NiFi data flows on Kubernetes clusters in a cost effective way providing auto scaling, resource isolation and monitoring with KPI-based alerting. Event driven use cases.

Google Cloud 104

Google Cloud AWS Cloud Computing Kafka 104

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

Kafka 111

Kafka Transportation Metadata Consulting 111

LinkedIn Engineering

JANUARY 19, 2024

Figure 1: Talent pool report for recruiters - LinkedIn Talent Insights During mergers and acquisitions, the source company’s user licenses and data are transferred to the acquiring company. All the source data should transition over to the destination user. Figure 2 captures the dependency of all the major entities.

Recruitment 88

Recruitment Data Process Process Kafka 88

Picnic Engineering

SEPTEMBER 8, 2022

The data is loaded into Snowflake, Picnic’s single source of truth Data Warehouse (DWH). Almost all internal services emit events over RabbitMQ. Our pipeline captures these events and sends them to Confluent Cloud. We use the RabbitMQ Source connector for Apache Kafka Connect.

Kafka 52

Kafka Metadata AWS Java 52

Data Engineering Podcast

AUGUST 24, 2020

Summary In order to scale the use of data across an organization there are a number of challenges related to discovery, governance, and integration that need to be solved. With their open-source foundation, fixed pricing, and unlimited volume, they are enterprise ready, but accessible to everyone.

Metadata 100

Metadata Management Kafka Data Engineering 100

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Confluent

JUNE 11, 2019

For event streaming application developers, it is important to continuously update the streaming pipeline based on the need for changes in the individual applications in the pipeline. It is also important to understand some of the common streaming topologies that streaming developers use to build an event streaming pipeline.

Kafka 85

Kafka Cloud Java MongoDB 85

Scott Logic

NOVEMBER 14, 2023

Introduction In any organisation there is a need to distribute data from a source system to other systems. Rather, I am talking here about asynchronously distributing data to other services that maintain their own copy of the data in order to do their work or where we want to emit and distribute events for other services to process.

Technology 52

Technology Kafka AWS Data 52

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. There is no need to pre-create a schema to run real-time analytics on event streams from Kafka.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Snowflake

AUGUST 16, 2023

Read our Summit recap blog for highlights across industries or watch Summit sessions now on-demand. Custom Event Billing capabilities enable you to build your own pricing strategy. You can charge customers based on their usage and specify billing events based on your preferences, such as consumed rows, ingested rows and more.

Scala 52

Scala Transportation Kafka Data Lake 52

Cloudera

DECEMBER 9, 2022

Cloudera has been providing enterprise support for Apache NiFi since 2015, helping hundreds of organizations take control of their data movement pipelines on premises and in the public cloud. Developers need to onboard new data sources, chain multiple data transformation steps together, and explore data as it travels through the flow.

Designing 95

Designing Coding Google Cloud AWS 95

Picnic Engineering

MARCH 28, 2023

Pipeline overview Since the TS is an externally developed hardware and software system, Picnic cannot change it to expose analytics events directly. So how can we get our hands on events for analytical purposes if we cannot change the TS to expose them? To do this, we use Debezium , a popular change data capture tool.

Kafka 52

Kafka Transportation Data Warehouse Database 52

Rockset

FEBRUARY 24, 2023

Introduction Let’s get this out of the way at the beginning: understanding effective streaming data architectures is hard, and understanding how to make use of streaming data for analytics is really hard. Kafka or Kinesis ? Open source or fully managed? Stream processing or an OLAP database?

Kafka 52

Kafka AWS Amazon Web Services Programming Language 52

Rockset

JUNE 8, 2023

Streaming data adoption continues to accelerate - over 80% of Fortune 100 companies already use Apache Kafka - driven by organizations creating value by putting data to use in real time. Much of this streaming data will land in real-time analytics databases as event streams.

Kafka 52

Kafka Algorithm Database Architecture 52

Data Engineering Weekly

APRIL 2, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real-time with your own JavaScript or Python code. Now you can win $1,000 cash by contributing a Transformation to our open-source library. Go to rtasummit.com and register with DEW30 for 30% off.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52