Data Engineering Weekly

NOVEMBER 26, 2023

Databricks is already doing a lot of these move-up stack products. However, I do believe there is tons of innovation left in the deep engineering of LakeHouse, such as In a way, you can think of DuckDB -like systems as a feature of LakeHouse rather than a separate product.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Data Engineering Podcast

JULY 6, 2020

As with many things in the space of data, local development and pre-production testing and validation are complicated due to the potential scale and variability of a production system. What are some useful strategies for collecting and analyzing traces of data flows?

Systems 100

Systems Kafka SQL Government 100

Data Engineering Podcast

MARCH 2, 2020

In this episode Michael Drogalis, product manager for ksqlDB at Confluent, explains how the system is implemented, how you can use it for building your own stream processing applications, and how it fits into the lifecycle of your data infrastructure.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Confluent

APRIL 10, 2019

Operators can manage those clusters, view topic data and schemas and run KSQL queries against data streams. Disclaimer: This is just for testing—do not take this Docker setup into production!). First and foremost, Confluent Control Center can manage multi-datacenter Apache Kafka ® deployments, whether on prem or in the cloud.

Kafka 86

Kafka Metadata Java Cloud 86

Striim

JANUARY 4, 2023

These companies collect data from production applications and B2B SaaS tools (e.g., Although it might not affect production, it might break something in other cases. Assess how data behaves on the fly During production, you can use data contracts as a data validation tool to see how data needs to behave in real time.

PostgreSQL 52

PostgreSQL Data Warehouse Data Lake Data 52

Rockset

APRIL 26, 2023

An insurance company sets thresholds for various types of support tickets, identifying issues with services or products before they affect revenue. An eCommerce marketplace recommends products based on users’ browsing history, what’s in stock, and what similar users have purchased.

Architecture 52

Architecture Transportation Data Lake Insurance 52

Confluent

MAY 1, 2019

That was the reason we created KSQL which has now become one of the most popular tools in the Kafka ecosystem. The underlying approach of continuously updated views on streaming data matched our vision of how developers build summarizations and joins on event streams with KSQL.

Kafka 21

Kafka Datasets Database Technology 21

Confluent

JULY 10, 2019

KSQL user-defined functions. KSQL provides a nice collection of built-in SQL functions for use in functional transformation logic when doing stream processing, whether the need is scalar functions for working with data a row at a time or aggregate functions used for grouping multiple rows into one summary record of output.

Kafka 87

Kafka Java Bytes SQL 87

Confluent

OCTOBER 9, 2019

Based on the votes of Summit attendees from within the Kafka Summit mobile app, here are the top-rated talks: Building Stream Processing Applications with Apache Kafka Using KSQL by Robin Moffatt of Confluent. KSQL Performance Tuning for Fun and Profit by Nick Dearden of Confluent. Why Stop the World When You Can Change It?

Kafka 17

Kafka Software Engineer Software Engineering Data Pipeline 17

Confluent

SEPTEMBER 26, 2019

Among these are Confluent Schema Registry , which ensures the right message structure, and KSQL for continuous stream processing on data streams, such as filtering, transformations, and aggregations using simple SQL commands. Plus, there will be swag waiting for you… Shruti Bhat is SVP of products at Rockset.

Kafka 21

Kafka BI SQL Datasets 21

Confluent

MAY 15, 2019

But these legacy warehouses were slow to ship from and couldn’t keep up with more modern inventory management strategies, including same-day dropshipping or bringing the product physically to their customers’ locations using industrial vending. So we chose to use the Confluent Platform and specifically KSQL to curate our streams of data.

Kafka 91

Kafka ETL Tools Cloud Data Integration 91

Confluent

AUGUST 29, 2019

Stream processing engines like KSQL furthermore give you the ability to manipulate all of this fluently. In a production scenario, a microservice would likely be responsible for generating these “recommendations,” tracking recommendations over time as data in Neo4j and letting the data publishing to Kafka take care of the rest.

Kafka 55

Kafka Process Algorithm Cloud 55

Confluent

MAY 8, 2019

In anything but the smallest deployment of Apache Kafka ® , there are often going to be multiple clusters of Kafka Connect and KSQL. Kafka Connect is used for building event streaming data pipelines between upstream and downstream systems with Kafka, and KSQL is used for building stream processing applications declared in a SQL-like language.

Kafka 87

Kafka Management Hadoop Database 87

Confluent

JUNE 26, 2019

This blog post discusses why Apache Kafka became the de facto standard and backbone for microservice architectures—not just replacing other traditional middleware but also building the microservices themselves using DDD and Kafka-native APIs like Kafka Streams, KSQL, and Kafka Connect. Microservices.

Kafka 109

Kafka Designing Architecture ETL Tools 109

Confluent

MARCH 13, 2019

An important note here is that I’m using the FileStream connector for demonstration purposes, but it is not recommended for use in production. This can be seen from the metrics: It can also be seen from inspecting the topic itself: ksql> LIST TOPICS; Kafka Topic | Registered | Partitions | Partition Replicas | Consumers | ConsumerGroups.

Kafka 111

Kafka Bytes Metadata NoSQL 111

Confluent

SEPTEMBER 5, 2019

As an application developer, you’re responsible for creating your topic instead of relying on auto-topic creation, which should be false in production environments. If this tutorial was helpful and you’re on the hunt for more on stream processing using Kafka Streams, KSQL, and Kafka, don’t forget to check out Kafka Tutorials.

Java 20

Java Kafka Food Cloud 20

Confluent

MAY 22, 2019

Various options like KSQL and Single Message Transforms (SMT) are available at this stage. Start the KSQL server. We will be interacting with the data in Kafka via KSQL. Once the KSQL server starts, open a KSQL shell and list all the topics available. Download the Confluent Platform.

Kafka 97

Kafka Google Cloud NoSQL Entertainment 97

Confluent

APRIL 25, 2019

Confluent Cloud provides us with a fully managed Kafka cluster for both development and production, so we can iterate the way we capture and use events with a select partner or two at each step before rolling new applications out to all interested parties along the food value chain.”. This is business critical for those customers’ operations.

Food 79

Food Cloud Kafka Transportation 79

Knowledge Hut

JANUARY 16, 2024

Machine Learning Skills Organizations are showing massive interest in using machine learning in their products, which would in turn bring plenty of opportunities for machine learning enthusiasts. Best measures to avoid bottlenecks and designing the final product such that it is user-friendly.

Machine Learning 98

Machine Learning Engineering Programming Language Algorithm 98

Confluent

JUNE 7, 2019

Our data science team uses KSQL to experiment with raw or lifted streams to ultimately deploy new machine learning models ( using custom user-defined functions ) without writing a single line of Java code. In production, each new block produces a state record update of 150 KB on average.

Accessible 92

Accessible Accessibility Kafka Scala 92

Rockset

AUGUST 5, 2022

Today, through live shopping videos, sellers offer products in more than 100 categories, from Pokemon and baseball cards to sneakers, antique coins and much more. It gathers signals in real-time from our audience: the videos they are watching, the comments and social interactions they are leaving, and the products they are buying.

Kafka 52

Kafka Machine Learning SQL Data Pipeline 52

Confluent

APRIL 17, 2019

With the Developer License , all of the Confluent Platform features are free of charge for an indefinite duration in a non-production environment with a single broker. Part 3: Managing Multi-Cluster Kafka Connect and KSQL with Confluent Control Center. Part 2: Managing Kafka Configurations at Scale with Confluent Control Center.

Management 75

Management Kafka Java Building 75

Confluent

JULY 31, 2019

delivers enhancements that will help you: Automate Apache Kafka ® operations for production environments. Open source, production-ready Ansible playbooks provide a simpler, more automated way of deploying Confluent Platform in non-containerized environments, fully supported by Confluent. Deploy to production in minutes.

Kafka 19

Kafka Cloud Accessible Accessibility 19

Confluent

MARCH 4, 2019

This case study is based off real, concrete war stories, and will detail how Confluent Cloud was a key enabler for their digital strategy, unfolding new ways to capitalize on their product and differentiate their business, by leveraging the power of Apache Kafka and the cloud at the same time. Journey from mainframe to cloud.

Cloud 88

Cloud Banking Kafka NoSQL 88

Confluent

APRIL 17, 2019

With the Developer License , all of the Confluent Platform features are free of charge for an indefinite duration in a non-production environment with a single broker. Part 3: Managing Multi-Cluster Kafka Connect and KSQL with Confluent Control Center. Part 2: Managing Kafka Configurations at Scale with Confluent Control Center.

Kafka 40

Kafka Management Java Building 40

Confluent

APRIL 2, 2019

Several components of Confluent Platform have always been available for free, such as KSQL, Schema Registry and REST Proxy, which are licensed under our Confluent Community License. KSQL adds new query expressions and gets GUI enhancements. KSQL has grown some features that we think you’ll find useful. librdkafka is now 1.0,

Kafka 21

Kafka Java Cloud Metadata 21

Confluent

SEPTEMBER 25, 2019

That is, the core processes a business executes—from how it produces a product, to how it interacts with customers, to how it delivers services—are increasingly specified, monitored, and executed in software. virtually all began life in the era of software as a human productivity aid.

Database-centric 56

Database-centric Kafka Pipeline-centric Retail 56

Rockset

JULY 25, 2019

Product managers Back in the day, product managers used to do a lot manual work, talking to customers, asking them how they use the product, what features in the product slow them down, etc. For example, SQL, HiveQL, KSQL etc. They have varying expressive powers: SQL supports full joins whereas KSQL does not.

Software Engineer 40

Software Engineer Software Engineering Engineering PostgreSQL 40

Confluent

FEBRUARY 13, 2019

Consumers in this context are anything that requests data; they could be stream processors, Java or.NET applications or KSQL server nodes. Various aspects of the function include working with prominent customers, working with product to drive innovation into the Kafka ecosystem and thought-lead about the next frontier of innovation.

Architecture 90

Architecture Programming Kafka Database-centric 90

Confluent

SEPTEMBER 20, 2019

Key characteristics of a fully managed service that you can trust for production and mission-critical applications. Some products may automatically create Kafka clusters in a dedicated compute instance and provide a way to connect to it, but eventually, users might need to scale the cluster, patch it, upgrade it, create backups, etc.

Kafka 21

Kafka Management Cloud AWS 21

Confluent

FEBRUARY 27, 2019

Stream processors allow us to work natively with these streams in a “correct” fashion by supporting a myriad of patterns with either bespoke logic, such as Kafka Streams, or a higher order grammar: KSQL. These are composed as an event stream that works quite well…which leads us to the question: When and where should FaaS be used?

Kafka 109

Kafka AWS Architecture Cloud 109

Confluent

OCTOBER 10, 2019

Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets. Digital services and subscriptions are provided to customers instead of just selling them products. and Connect and KSQL clusters.

Kafka 20

Kafka Google Cloud Architecture Machine Learning 20

Confluent

AUGUST 14, 2019

Now you can deploy end-to-end Secret Protection in your production event pipeline, including the brokers, Connect, KSQL, Confluent Schema Registry, Confluent Control Center, Confluent REST Proxy, etc. These hosts may include Kafka brokers, Connect workers, Schema Registry instances, KSQL servers, Confluent Control Center, etc.—any

Kafka 12

Kafka Metadata Accessible Accessibility 12

Confluent

AUGUST 19, 2019

Because our partners know how to interface with their products best, we actually prefer it when partners offer connectors to our platform and will always prioritize partner-sourced connectors. Stream processors: integrations that process data in Kafka via KSQL, Kafka Streams, or something else. Philip Rathle, VP of Products, Neo4j.

Programming 15

Programming Kafka Database-centric MongoDB 15