Confluent

JULY 11, 2019

Reading, writing, and transforming data in Apache Kafka ® using KSQL is an effective way to rapidly deliver event streaming applications for clients (e.g., For a KSQL newbie the practical exercises show you how to process data in Apache Kafka using an interactive SQL interface. streaming insurance events ).

Kafka 83

Kafka Insurance SQL Architecture 83

Cloudera

MARCH 14, 2023

We just announced the general availability of Cloudera DataFlow Designer , bringing self-service data flow development to all CDP Public Cloud customers. In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities.

Data Pipeline 80

Data Pipeline Designing Kafka Metadata 80

Rockset

AUGUST 21, 2019

Rockset continuously ingests data streams from Kafka, without the need for a fixed schema, and serves fast SQL queries on that data. We created the Kafka Connect Plugin for Rockset to export data from Kafka and send it to a collection of documents in Rockset. This blog covers how we implemented the plugin.

Kafka 40

Kafka IT Data Storage Relational Database 40

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. Data scientists love Python, period.

Machine Learning 106

Machine Learning Python Kafka Java 106

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Snowflake

AUGUST 16, 2023

Read our Summit recap blog for highlights across industries or watch Summit sessions now on-demand. Applications Snowflake Native App Framework now available in AWS – public preview Snowflake Native Apps are an entirely new way to put data to work. If you missed out, not to worry! Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Data Engineering Weekly

APRIL 2, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real-time with your own JavaScript or Python code. Meta: Presto - A Decade of SQL Analytics at Meta Presto and Kafka are the two systems that greatly impacted data infrastructure in the last decade.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

ProjectPro

JANUARY 24, 2023

With over 20 pre-built connectors and 40 pre-built transformers, AWS Glue is an extract, transform, and load (ETL) service that is fully managed and allows users to easily process and import their data for analytics. You can leverage AWS Glue to discover, transform, and prepare your data for analytics.

AWS 52

AWS Data Lake ETL Tools Scala 52

Confluent

AUGUST 27, 2019

We are excited to announce the release of Confluent Cloud Schema Registry in general availability (GA), available in Confluent Cloud , our fully managed event streaming service based on Apache Kafka ®. Confluent Schema Registry provides a serving layer for your metadata and a RESTful interface for storing and retrieving Avro schemas.

Cloud 18

Cloud Kafka Electronics Metadata 18

Monte Carlo

JULY 7, 2022

At the time, he was one of a very few people using the term “data contract.” Data contracts have since became one of the most discussed topics in data engineering. For posterity, we have preserved Barr’s forward that examines what was then a very nascent trend, but we have also added an updated data contract FAQ as an addendum.

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

Rockset

FEBRUARY 24, 2023

Introduction Let’s get this out of the way at the beginning: understanding effective streaming data architectures is hard, and understanding how to make use of streaming data for analytics is really hard. Kafka or Kinesis ? What Is Streaming Data? We’re going to start with a basic question: what is streaming data?

Kafka 52

Kafka AWS Amazon Web Services Programming Language 52

Cloudera

MARCH 11, 2022

With the launch of CDP Public Cloud 7.2.14, Cloudera Streams Messaging for Data Hub deployments has gotten some powerful new features! In this release , the Streams Messaging templates in Data Hub will come with Apache Kafka 2.8 Kafka & Cruise Control Updates. Kafka Updates: Deployments with Kafka 2.5

Cloud 100

Cloud Kafka Utilities Database 100

Confluent

JULY 17, 2019

Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. The official MongoDB Connector for Apache Kafka is developed and supported by MongoDB engineers. Getting started.

MongoDB 21

MongoDB Kafka Medical Database 21

Picnic Engineering

MARCH 28, 2023

Last time , we looked at the internal services data pipeline. That’s why the Analytics Platform team has developed a new data pipeline: The FCA pipeline. Enter change data capture (CDC)! Our approach is to capture these changes and stream them via our Apache Kafka based analytics platform to our Snowflake data warehouse.

Kafka 52

Kafka Transportation Data Warehouse Database 52

Confluent

MAY 21, 2019

The profile service will publish the changes in profiles, including address changes to an Apache Kafka ® topic, and the quote service will subscribe to the updates from the profile changes topic, calculate a new quote if needed and publish the new quota to a Kafka topic so other services can subscribe to the updated quote event.

Kafka 110

Kafka Insurance Architecture Database 110

LinkedIn Engineering

MAY 9, 2023

In this blog post we’ll discuss our decision to leverage Couchbase, the challenges that arose, and how we addressed each challenge in our final solution. Member profiles are stored in Espresso in Avro binary format. Data flow for a profile view or update A datum reader is used to deserialize an Avro binary content.

Database 133

Database Project Datasets Designing 133

Netflix Tech

AUGUST 1, 2022

Data Mesh?—?A A Data Movement and Processing Platform @ Netflix By Bo Lei , Guilherme Pires , James Shao , Kasturi Chatterjee , Sujay Jain , Vlad Sydorenko Background Realtime processing technologies (A.K.A After evaluating the options , the team has decided to create Data Mesh as our next generation data pipeline solution.

Process 108

Process Transportation Kafka Entertainment 108

Confluent

FEBRUARY 22, 2019

The requirements for fast and reliable data pipelines are growing quickly at Deliveroo as the business continues to grow and innovate. We have delivered an event streaming platform which gives strong guarantees on data quality, using Apache Kafka ® and Protocol Buffers.

Kafka 80

Kafka Programming Language Metadata Data 80

Cloudera

AUGUST 10, 2022

Cloudera Stream Processing (CSP), powered by Apache Flink and Apache Kafka, provides a complete stream management and stateful processing solution. In CSP, Kafka serves as the storage streaming substrate, and Flink as the core in-stream processing engine that supports SQL and REST interfaces. Apache Kafka and SMM.

Process 85

Process Kafka PostgreSQL MySQL 85

Confluent

JUNE 26, 2019

I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®. In these projects, microservice architectures use Kafka as an event streaming platform. Apache Kafka – An event streaming platform for microservices. So why is Kafka such a good fit? Apache Kafka and domain-driven microservices.

Kafka 109

Kafka Designing Architecture ETL Tools 109

Scribd Technology

MAY 18, 2021

Streaming data from Apache Kafka into Delta Lake is an integral part of Scribd’s data platform, but has been challenging to manage and scale. We use Spark Structured Streaming jobs to read data from Kafka topics and write that data into Delta Lake tables. To serve this need, we created kafka-delta-ingest.

Kafka 52

Kafka Data Warehouse Bytes Metadata 52

Data Engineering Weekly

NOVEMBER 13, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. The schematization of data plays a vital role in a data platform.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Rockset

MARCH 26, 2023

Breaking Bad… Data Silos We haven’t quite figured out how to avoid using relational databases. Folks have definitely tried, and while Apache Kafka® has become the standard for event-driven architectures, it still struggles to replace your everyday PostgreSQL database instance in the modern application stack.

Cloud 52

Cloud PostgreSQL Kafka Database 52

Confluent

JUNE 18, 2019

It involves many moving parts, from data preparation to building indexing and query pipelines. Luckily, this task looks a lot like the way we tackle problems that arise when connecting data. Distributed transactions are very hard to implement successfully, which is why we’ll introduce a log-inspired system such as Apache Kafka ®.

Architecture 95

Architecture Building Kafka Database-centric 95

Confluent

APRIL 17, 2019

In particular, the management and monitoring capabilities that we added to Confluent Control Center have evolved it into an indispensable tool for anyone working with Apache Kafka ®. Part 2: Managing Kafka Configurations at Scale with Confluent Control Center. Edit schemas as your data evolves.

Management 74

Management Kafka Java Building 74

Confluent

JULY 16, 2019

How cool would it be to build your own burglar alarm system that can alert you before the actual event takes place simply by using a few network-connected cameras and analyzing the camera images with Apache Kafka ® , Kafka Streams, and TensorFlow? I will show how to implement this use case in this blog post.

Machine Learning 100

Machine Learning Kafka Java Datasets 100

Confluent

MAY 22, 2019

In light of this, we’ll share an emerging machine-to-machine (M2M) architecture pattern in which MQTT, Apache Kafka ® , and Scylla all work together to provide an end-to-end IoT solution. The explosive number of devices generating, tracking and sharing data across a variety of networks is overwhelming to most data management solutions.

Kafka 96

Kafka Google Cloud NoSQL Entertainment 96

dbt Developer Hub

NOVEMBER 3, 2022

One key focus of our platform is data quality. In this blog post we’ll dive into how we tackled one source of quality issues: directly relying on upstream database schemas. We use Snowflake as our data warehouse where we build dashboards both for internal use and for customers. Each of these microservices has its own database.

Kafka 52

Kafka ETL Tools BI Database 52

Confluent

MARCH 12, 2019

On the heels of part one in this blog series, Spring for Apache Kafka – Part 1: Error Handling, Message Conversion and Transaction Support , here in part two we’ll focus on another project that enhances the developer experience when building streaming applications on Kafka: Spring Cloud Stream. What is Spring Cloud Stream?

Kafka 111

Kafka Cloud Programming Coding 111

Rockset

JANUARY 17, 2020

In this blog we will set up a real-time SQL API on Kafka using AWS Lambda and Rockset. If you already have data you want to build an API on skip to Step 1. The Kafka set up instructions for the remainder of this tutorial are for Confluent Kafka. Start up a local version of Kafka.

Kafka 40

Kafka SQL AWS Accessible 40

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. For a system like Elasticsearch , engineers need to have in-depth knowledge of the underlying architecture in order to efficiently ingest streaming data.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Confluent

APRIL 17, 2019

In particular, the management and monitoring capabilities that we added to Confluent Control Center have evolved it into an indispensable tool for anyone working with Apache Kafka ®. Part 2: Managing Kafka Configurations at Scale with Confluent Control Center. Edit schemas as your data evolves.

Kafka 40

Kafka Management Java Building 40

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value.

Engineering 81

Engineering Raw Data Data Science Scala 81

Confluent

JULY 3, 2019

From a data perspective, the World Cup represents an interesting source of information. The idea in this blog post is to mix information coming from two distinct channels: the RSS feeds of sport-related newspapers and Twitter feeds of the FIFA Women’s World Cup. Data sources. Ingesting Twitter data. connector.state].

Data Analysis 21

Data Analysis Kafka Datasets Java 21

Confluent

MAY 9, 2019

Storing events in a stream and connecting streams via stream processors provide a generic, data-centric, distributed application runtime that you can use to build ETL, event streaming applications, applications for recording metrics and anything else that has a real-time data requirement. Avro or Protobuf ). Control plane.

Kafka 92

Kafka Pipeline-centric Architecture Database-centric 92

Confluent

OCTOBER 24, 2019

I’ve written an event sourcing bank simulation in Clojure (a lisp build for Java virtual machines or JVMs) called open-bank-mark , which you are welcome to read about in my previous blog post explaining the story behind this open source example. All messages use a String for the key and Avro for the value. The bank application.

Kafka 18

Kafka Java Banking Bytes 18

Confluent

AUGUST 14, 2019

These secrets may include passwords, such as the values for ssl.key.password , ssl.keystore.password , and ssl.truststore.password configuration parameters (as shown below), or any other sensitive data in the configuration files or log files. Before Confluent Platform 5.3, path/to/secrets.txt ).

Kafka 12

Kafka Metadata Accessible Accessibility 12