Rockset

AUGUST 16, 2022

Real quick disclaimer, I currently work at Rockset but previously worked at Confluent, a company known for building Kafka based platforms and cloud services. You can look at the source code, deploy it wherever you want and even fork the source code, create a new product and sell it! Let’s find out!

Kafka 52

Kafka AWS Cloud Java 52

Confluent

FEBRUARY 6, 2019

This structure worked well for production training and deployment of many models but left a lot to be desired in terms of overhead, flexibility, and ease of use, especially during early prototyping and experimentation [where Notebooks and Python shine]. Impedance mismatch between data scientists, data engineers and production engineers.

Machine Learning 107

Machine Learning Python Kafka Java 107

Data Engineering Weekly

MARCH 26, 2023

Come and hear talks from companies like StarTree, Confluent, LinkedIn, DoorDash, Imply, and Uber on how they are advancing the state-of-the-art in user-facing analytics delivered instantly. Lack of coordination between data producers and data consumers are perceived by all respondents to be this year’s top threat to the ecosystem.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Rockset

OCTOBER 21, 2022

Real-time customer 360 applications are essential in allowing departments within a company to have reliable and consistent data on how a customer has engaged with the product and services. We’ll see the customer’s 360 profile that’ll include their product recommendations. In my workshop, I used Mockaroo data and sent that to Kafka.

Kafka 52

Kafka Building SQL Database 52

Monte Carlo

JANUARY 25, 2023

Over the past year, data contracts have taken the data world by storm as a novel approach to ensuring data quality at scale in production services. In this article, Chad Sanderson , Head of Product, Data Platform , at Convoy and creator of Data Quality Camp , introduces a new application of data contracts: in your data warehouse.

Data Warehouse 52

Data Warehouse Data High Quality Data Metadata 52

Data Engineering Podcast

DECEMBER 24, 2023

Elad Eldor has experienced these challenges first-hand, leading to his work writing the book "Kafka: : Troubleshooting in Production" In this episode he highlights the sources of complexity that contribute to Kafka's operational difficulties, and some of the main ways to identify and mitigate potential sources of trouble.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Confluent

MAY 1, 2019

When we founded Confluent, we took the radical approach of viewing data—and the infrastructure that supported it—as a series of real-time streaming events rather than something kept in static, sedentary data repositories. In four short years, Confluent has grown from an idea into an industry pioneer.

Kafka 21

Kafka Datasets Database Technology 21

Confluent

SEPTEMBER 26, 2019

The significant difference today is that companies use Apache Kafka as an event streaming platform for building mission-critical infrastructures and core operations platforms. Apache Kafka as an event streaming platform for real-time analytics. However, Apache Kafka is more than just messaging. Connecting Kafka to Rockset.

Kafka 21

Kafka BI SQL Datasets 21

Cloudera

OCTOBER 17, 2023

Recently, Confluent hosted Current 2023 (formerly Kafka summit) in San Jose on Sept 26th and 27th. More of a Confluent conference now than a kafka conference. Expecting a product to be GA’d. The big announcement everyone was waiting for was the unveiling of Apache Flink in Confluent Cloud.

Database-centric 75

Database-centric Kafka Pipeline-centric Database 75

Monte Carlo

FEBRUARY 20, 2024

There’s the dichotomy between batch and streaming, between all-in-one platforms and niche connectors in your modern data stack, and the trade-off between ease-of-use and powerful customization options. Fivetran is the leader in the data ingestion space, known for its ease of use and extensive connector ecosystem.

Data Ingestion 52

Data Ingestion Google Cloud Kafka AWS 52

AltexSoft

OCTOBER 21, 2022

Apache Kafka is an open-source, distributed streaming platform for messaging, storing, processing, and integrating large data volumes in real time. In other words, Kafka can serve as a messaging system, commit log, data integration tool, and stream processing platform. What is Kafka? Kafka advantages. Scalability.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

Confluent

JUNE 26, 2019

In these projects, microservice architectures use Kafka as an event streaming platform. Apache Kafka – An event streaming platform for microservices. With these three pillars built into one distributed event streaming platform, you can decouple various microservices (i.e., Martin Fowler. This can be answered in two parts: 1.

Kafka 109

Kafka Designing Architecture ETL Tools 109

Monte Carlo

JANUARY 16, 2024

Solutions like Confluent with Databricks allow organizations to prep, join, enrich and query streaming data sets in Databricks SQL, which they claim has up to 12x better price-performance than a traditional data warehouse. And how hard will it be to interact outside of the zero ETL ecosystem? How will data governance be handled?

Data Lake 52

Data Lake Data Warehouse Unstructured Data AWS 52

Confluent

MARCH 4, 2019

This case study is based off real, concrete war stories, and will detail how Confluent Cloud was a key enabler for their digital strategy, unfolding new ways to capitalize on their product and differentiate their business, by leveraging the power of Apache Kafka and the cloud at the same time. Journey from mainframe to cloud.

Cloud 88

Cloud Banking Kafka NoSQL 88

Rockset

MARCH 26, 2023

To do this, Rockset has partnered with Confluent, the original creators of Kafka who provide the cloud-native data streaming platform Confluent Cloud. In the interest of full disclosure, it’s worth mentioning that I am a former Confluent employee, now working at Rockset.

Cloud 52

Cloud PostgreSQL Kafka Database 52

Confluent

JUNE 7, 2019

How we use Apache Kafka and the Confluent Platform. One can easily imagine that the Confluent Schema Registry is extremely helpful for typeability and systematic evolution of our models. The Confluent Platform is an amazing toolbox, which every architect and data engineer should know of and utilize.

Accessible 92

Accessible Accessibility Kafka Scala 92

Confluent

JUNE 18, 2019

Greater flexibility, maintainability, and changeability by decoupling production from data consumption and allowing all systems to run independently. Kafka Connect has the great benefit of simplifying your deployment requirements, as it is bundled with Apache Kafka and its ecosystem. Schema evolution.

Architecture 96

Architecture Building Kafka Database-centric 96

Confluent

MAY 22, 2019

Apache Kafka is an open source, distributed message queuing and event streaming platform capable of handling a high volume and velocity of events. Since its creation at LinkedIn in 2011, Kafka has quickly evolved from a message queuing system to a full-fledged event streaming platform. Demo of Scylla and Confluent integration.

Kafka 97

Kafka Google Cloud NoSQL Entertainment 97

Confluent

APRIL 25, 2019

By capturing Internet of Things (IoT) event data from farm to fork with Apache Kafka ® and Confluent Cloud, BAADER is increasing its value as part of this chain, creating new business opportunities and enabling its partners to optimize their operations. “By “With Confluent Cloud, we get more than just a Kafka service.

Food 79

Food Cloud Kafka Transportation 79

Confluent

MAY 30, 2019

As a continuation from the previous blog series, this blog post explains how Spring Cloud Data Flow helps you gain developer productivity and manage Apache-Kafka-based event streaming application development. We will cover the following in this post: Overview of the Spring Cloud Data Flow ecosystem. Spring Cloud Data Flow ecosystem.

Kafka 94

Kafka Cloud Data Pipeline PostgreSQL 94

Confluent

FEBRUARY 27, 2019

This is important as each platform has different persistence models and APIs, tying you to that vendor’s ecosystem. Peeking Behind the Curtains of Serverless Platforms, 2018. Do you want to stick to the cloud vendor platform or adopt a vendor-agnostic approach? Do they complement or compete? Next steps.

Kafka 109

Kafka AWS Architecture Cloud 109

Confluent

APRIL 24, 2019

Finally after a few incidents in production, and with velocity slowing down due to the deployment pipeline frequently being clogged up by flaky integration tests, you start to think about what you want from your test suite. A remote Kafka cluster (requiring only HTTP access to Confluent REST Proxy). The Test Machine. Show me the code.

Systems 94

Systems Kafka Transportation Coding 94

Pinterest Engineering

JULY 11, 2023

Divye , Teja , Chen , Sam , Lu , Heng , Kanchi , Rainie , Dinesh , Ashish , Nishant , Pooja | Stream Processing Platform Team At Pinterest, stream data processing powers a wide range of real-time use cases. Pinterest runs 130+ Flink jobs across eight multitenant production YARN clusters. The quick takeaways? These jobs vary in size.

AWS 71

AWS Utilities Transportation Engineering 71

Confluent

FEBRUARY 13, 2019

The Reactive Manifesto has influenced many frameworks and platforms, including Akka (an actor-based library circa 1973 ), Vert.x We think of streams and events much like database tables and rows; they are the basic building blocks of a data platform. Reactive Systems are highly responsive, giving users effective interactive feedback.

Architecture 90

Architecture Programming Kafka Database-centric 90

Confluent

JULY 31, 2019

Delivers the new Confluent Operator for cloud-native automation on Kubernetes, a redesigned Confluent Control Center user interface to simplify how you manage event streams, and a preview of Role-Based Access Control for enterprise-grade security. To solve these challenges, Confluent Platform 5.3

Kafka 19

Kafka Cloud Accessible Accessibility 19

Confluent

SEPTEMBER 25, 2019

That is, the core processes a business executes—from how it produces a product, to how it interacts with customers, to how it delivers services—are increasingly specified, monitored, and executed in software. virtually all began life in the era of software as a human productivity aid. Emergence of event streams.

Database-centric 56

Database-centric Kafka Pipeline-centric Retail 56

Confluent

SEPTEMBER 20, 2019

Luckily for on-premises scenarios, a myriad of deployment options are available, such as the Confluent Platform which can be deployed on bare metal, virtual machines, containers, etc. Cloud Memorystore, Amazon ElastiCache, and Azure Cache), applying this concept to a distributed streaming platform is fairly new.

Kafka 21

Kafka Management Cloud AWS 21

Confluent

OCTOBER 10, 2019

Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets. Confluent MQTT Proxy , which ingests data from IoT devices without needing a MQTT broker. Example: Severstal.

Kafka 20

Kafka Google Cloud Architecture Machine Learning 20

Confluent

AUGUST 19, 2019

This post specifically outlines the process by which we verify partner integrations, and is a means of letting the world know about our partner’s contributions to our connector ecosystem. Gold verifications indicate the tightest integration with Confluent Platform, whereas Standard verifications are functional and practical.

Programming 15

Programming Kafka Database-centric MongoDB 15

Confluent

MAY 9, 2019

For this reason, I’m not only going to focus on the payment processing pipeline but also work through the other functionality required to architect a production-worthy solution. With event-first design, the data becomes the API which, like any production system, needs to support change and evolution (i.e., Avro or Protobuf ).

Kafka 93

Kafka Pipeline-centric Architecture Database-centric 93

Confluent

JUNE 11, 2019

In Spring Cloud Stream terms, a named destination is a specific destination name in the messaging middleware or event streaming platform. This is especially useful for Apache Kafka users, because in most of the cases, the event streaming platform is Apache Kafka itself. Common event streaming topologies in Spring Cloud Data Flow.

Kafka 85

Kafka Cloud Java MongoDB 85

Confluent

OCTOBER 24, 2019

As a next step, specifically for this article I’ve added SSL and combined some topics together, using the subject name strategy option of Confluent Schema Registry , making it more production like, adding security, and making it possible to put multiple kinds of commands on one topic. If so, it should send the same response back.

Kafka 18

Kafka Java Banking Bytes 18

ProjectPro

JUNE 29, 2021

Confluent Python Kafka has the best performance among the three. It is offered by Confluent as a wrapper around 'librdkafka' (C library implementation of Kafka). What is the Confluent Replicator? The Confluent Replicator allows easy and reliable replication of topics from a source cluster to a destination cluster.

Kafka 40

Kafka Bytes Big Data Java 40

Data Engineering Weekly

APRIL 9, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. Chinese citizens are among those who feel the most positively about AI products and services.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Monte Carlo

AUGUST 19, 2023

A data platform, often referred to as a ‘modern data stack,’ is the central processing hub for an organization’s data ecosystem. The data platform manages the collection, normalization, transformation, and application of data for a given data product—from business insights and dashboards to ML and AI engineering.

Building 52

Building BI Data Lake Data Governance 52