Cloudera

JULY 21, 2022

Cloudera users can securely connect Rill to a source of event stream data, such as Cloudera DataFlow , model data into Rill’s cloud-based Druid service, and share live operational dashboards within minutes via Rill’s interactive metrics dashboard or any connected BI solution. Figure 1: Rill and Cloudera Architecture. Exactly once support.

BI 82

BI Digital Media Data Warehouse Kafka 82

Cloudera

JUNE 7, 2021

Cloudera has been named as a Strong Performer in the Forrester Wave for Streaming Analytics, Q2 2021. We are proud to have been named as one of “ The 14 providers that matter most ” in streaming analytics. Otherwise, they miss the huge opportunity windows for transforming their businesses. It’s too late.

Kafka 100

Kafka Data Ingestion Architecture Cloud 100

Cloudera

SEPTEMBER 28, 2020

Modernize their architecture to ingest data in real-time using the new streaming features available in CDP Private Cloud Base in order to make the data available to their users quickly. Support Kafka connectivity to HDFS, AWS S3 and Kafka Streams. Cluster management and replication support for Kafka clusters.

Cloud 130

Cloud Kafka Professional Services Metadata 130

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! Using fixed lookback windows to always reprocess data, assuming that most late-arriving events will occur within that window. Let’s dive in! How does late-arriving data impact us?

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Confluent

JULY 11, 2019

Reading, writing, and transforming data in Apache Kafka ® using KSQL is an effective way to rapidly deliver event streaming applications for clients (e.g., streaming insurance events ). To inspire and help developers embrace this fantastic event streaming technology, Stéphane Maarek and I authored a new KSQL course.

Kafka 83

Kafka Insurance SQL Architecture 83

DoorDash Engineering

OCTOBER 17, 2023

Experimentation isn’t just a cornerstone for innovation and sound decision-making; it’s often referred to as the gold standard for problem-solving, thanks in part to its roots in the scientific method. The term itself conjures a sense of rigor, validity, and trust. One of these challenges is sample ratio mismatch, or SRM.

Education 87

Education Kafka Algorithm Data Warehouse 87

Cloudera

OCTOBER 4, 2022

This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes.

Kafka 76

Kafka Data Cloud Big Data 76

dbt Developer Hub

AUGUST 2, 2023

Jetblue has outgrown lambda views due to performance constraints (a view can only be so performant) and we are at another milestone in dbt’s journey to support streaming. For the sake of simplicity, when I refer to materialized views in this blog, I mean dynamic tables in Snowflake. So, what does that mean?

Datasets 52

Datasets Kafka SQL Cloud 52

Cloudera

AUGUST 26, 2020

Live data-streaming offers businesses exciting new opportunities to transform the way they operate, leveraging real-time insights to drive better decision making and enhance operational efficiency. Hello Dinesh, thank you for joining us for Part II of our Q&A on streaming data. How does that happen in near real-time?

Banking 80

Banking Data Ingestion Kafka Data Lake 80

Lyft Engineering

OCTOBER 3, 2023

Streaming compute however, empowers more complex window queries on semantic correctness. Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Zalando Engineering

JULY 12, 2017

In this blog post we describe the generation of such events using Apache Flink, and share our experiences and lessons learned in the process. You can read more on why we have chosen Apache Flink over other stream processing frameworks here: Apache Showdown: Flink vs. Spark. It has a start and a completion event.

Process 40

Process Kafka AWS Architecture 40

Rockset

FEBRUARY 7, 2020

In this IoT example, we examine how to enable complex analytic queries on real-time Kafka streams from connected car sensors. Building Real-Time Analytics on Connected Car IoT Data For our example, we have a fleet of connected vehicles that send the sensor data they generate to a Kafka cluster.

Kafka 40

Kafka Transportation Data SQL 40

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

Zalando Engineering

NOVEMBER 29, 2017

Second in our series about the use of Apache Kafka’s Streams API by Zalando This is the second in a series about the use of Apache Kafka’s Streams API by Zalando, Europe’s leading online fashion platform. See Ranking Websites in Real-time with Apache Kafka’s Streams API for the first post in the series.

Kafka 40

Kafka AWS Amazon Web Services Utilities 40

Cloudera

APRIL 20, 2023

Cloudera SQL Stream builder gives non-technical users the power of a unified stream processing engine so they can integrate, aggregate, query, and analyze both streaming and batch data sources in a single SQL interface. SSB provides a convenient way to work with Kafka as we can do the whole setup using the UI.

SQL 78

SQL Kafka Data Pipeline Project 78

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®.

Kafka 93

Kafka Cloud Data Pipeline PostgreSQL 93

Rockset

MARCH 27, 2023

This is part two in Rockset’s Making Sense of Real-Time Analytics on Streaming Data series. In part 1 , we covered the technology landscape for real-time analytics on streaming data. In this post, we’ll explore the differences between real-time analytics databases and stream processing frameworks. With that, let’s dive in.

Database 52

Database Process Scala SQL 52

Cloudera

AUGUST 10, 2022

Cloudera has a strong track record of providing a comprehensive solution for stream processing. Cloudera Stream Processing (CSP), powered by Apache Flink and Apache Kafka, provides a complete stream management and stateful processing solution. Cloudera Stream Processing Community Edition.

Process 85

Process Kafka PostgreSQL MySQL 85

LinkedIn Engineering

MARCH 23, 2023

In the past, we often used lambda architecture for processing jobs, meaning that our developers used two different systems for batch and stream processing. In the past, we often used lambda architecture for processing jobs, meaning that our developers used two different systems for batch and stream processing.

Process 97

Process Lambda Architecture Kafka Datasets 97

Knowledge Hut

JANUARY 16, 2024

In this comprehensive blog, we delve into the foundational aspects and intricacies of the machine learning landscape. Embarking on a journey in the highly demanded field of Machine Learning (ML) opens doors to diverse career opportunities. Look at the stats that show a positive trend for machine learning projects and careers.

Machine Learning 98

Machine Learning Engineering Programming Language Algorithm 98

Confluent

APRIL 30, 2019

With the release of Apache Kafka ® 2.1.0, Kafka Streams introduced the processor topology optimization framework at the Kafka Streams DSL layer. This framework opens the door for various optimization techniques from the existing data stream management system (DSMS) and data stream processing literature.

Kafka 90

Kafka Coding Process Bytes 90

Confluent

APRIL 4, 2019

In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream. In the Apache Kafka ® world, this means that each of those microservice client applications subscribes to a common Kafka topic. IoT: a stream of sensor data in which each sensor reading is an event.

Kafka 108

Kafka Data Cleanse Retail Finance 108

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? When it comes to the emerging serverless world, It makes sense to validate how Apache Kafka ® fits in considering that it is mission critical in 90 percent of companies. FaaS for streaming processing. What is FaaS?

Kafka 109

Kafka AWS Architecture Cloud 109

Cloudera

MAY 4, 2021

Event-driven and streaming architectures enable complex processing on market events as they happen, making them a natural fit for financial market applications. Flink SQL is a data processing language that enables rapid prototyping and development of event-driven and streaming applications. Streaming VWAP.

SQL 76

SQL Business Analyst Data Java 76

Data Engineering Weekly

NOVEMBER 13, 2022

Streaming and batch unified in a single platform No Airflow - orchestration inferred from the data $99 / TB of data ingested | transformations free Start Your 30 Day Trial Wealthfront: Event Tracking System at Wealthfront A robust event-tracking system is critical for an efficient data management platform. Not Endless DAGs!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Cloudera

DECEMBER 14, 2020

One trend that we’ve seen this year, is that enterprises are leveraging streaming data as a way to traverse through unplanned disruptions, as a way to make the best business decisions for their stakeholders. . Gain comprehensive and newer streaming capabilities with CDP. Why upgrade to CDP now?

Process 68

Process Kafka Government Big Data 68

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. Introduction. KSQL primer.

Kafka 94

Kafka Management Bytes SQL 94

Cloudera

JANUARY 11, 2021

Why use NiFi when one can use Kafka as an entry point to the cluster. Here are ways you can determine when to use NiFi and when to use Kafka. . Kafka is designed for stream-oriented use cases primarily for smaller files, and ingesting large files is not a good idea. MiNiFi comes in two versions: C++ and Java.

Kafka 59

Kafka Data Cleanse Java Datasets 59

Confluent

JUNE 18, 2019

It also requires both systems to always be available, so no maintenance windows are possible. Distributed transactions are very hard to implement successfully, which is why we’ll introduce a log-inspired system such as Apache Kafka ®. Building an indexing pipeline at scale with Kafka Connect.

Architecture 95

Architecture Building Kafka Database-centric 95

Cloudera

OCTOBER 19, 2022

In this post I will demonstrate how Kafka Connect is integrated in the Cloudera Data Platform (CDP), allowing users to manage and monitor their connectors in Streams Messaging Manager while also touching on security features such as role-based access control and sensitive information handling. Kafka Connect.

Kafka 70

Kafka MySQL Bytes Data 70

Rockset

FEBRUARY 24, 2022

Which databases are optimized for ingesting streaming events and analyzing them in real time? We’ll start by evaluating three options for running real-time analytics on AWS Kinesis event streams. I’ll focus on the use of events to help understand, analyze and diagnose problems using various OLAP databases and AWS Kinesis data streams.

AWS 52

AWS Amazon Web Services Kafka SQL 52

ProjectPro

JANUARY 25, 2022

Here’s What You Need to Know About PySpark This blog will take you through the basics of PySpark, the PySpark architecture, and a few popular PySpark libraries , among other things. Did you know that, according to Linkedin, over 24,000 Big Data jobs in the US list Apache Spark as a required skill? Why use PySpark?

Big Data 52

Big Data Data Process Process Kafka 52

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. According to reports by DICE Insights, the job of a Data Engineer is considered the top job in the technology industry in the third quarter of 2020.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Snowflake

MARCH 30, 2023

In this blog post we will use what we have learned in this Data Vault blog series to support the data preparation requirements for ML on Snowflake, using Data Vault patterns for modeling and automation. Based on Tecton blog So is this similar to data engineering pipelines into a data lake/warehouse?

Engineering 85

Engineering Raw Data Data Science Scala 85

Confluent

JULY 16, 2019

How cool would it be to build your own burglar alarm system that can alert you before the actual event takes place simply by using a few network-connected cameras and analyzing the camera images with Apache Kafka ® , Kafka Streams, and TensorFlow? I will show how to implement this use case in this blog post.

Machine Learning 100

Machine Learning Kafka Java Datasets 100

Big Data Tools

FEBRUARY 9, 2022

We all know Apache NiFi, a stream processing tool with its own processing engine. Kafka: Add range and scan query over kv-store in IQv2 — The name of this KIP speaks for itself. Due to the public holidays in Russia and my own vacation time, I didn’t get a chance to write an Annotated for December. Apache Hop is different in many ways.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

FEBRUARY 9, 2022

We all know Apache NiFi, a stream processing tool with its own processing engine. Kafka: Add range and scan query over kv-store in IQv2 — The name of this KIP speaks for itself. Due to the public holidays in Russia and my own vacation time, I didn’t get a chance to write an Annotated for December. Apache Hop is different in many ways.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52