Data Engineering Weekly

MARCH 24, 2024

OpenAI has more production deployments, but Google is catching on to it. As we predicted in the key trends of 2023 about Apache Flink as a clear winner in the stream processing frameworks, we see Confluent offering Flink as a service. Data Mesh is certainly a hotly debated topic in the data industry.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Precisely

MAY 13, 2024

A true enterprise-grade integration solution calls for source and target connectors that can accommodate: VSAM files COBOL copybooks open standards like JSON modern platforms like Amazon Web Services ( AWS ), Confluent , Databricks , or Snowflake Questions to ask each vendor: Which enterprise data sources and targets do you support?

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Confluent

APRIL 18, 2019

And Confluent Cloud and Apache Kafka ® are at the center of this transformation. So working with Confluent Cloud and the Kafka experts was an obvious choice,” says Stefan Frehse, Software Architect, Digitalization, for food processing machinery company BAADER. Retail is another industry being disrupted by IoT and Confluent Cloud.

Food 83

Food Cloud Retail Kafka 83

Confluent

APRIL 23, 2019

This freedom of choice ultimately allows you to build an event streaming platform with the language best suited to your business needs. At Confluent, we have an engineering team dedicated to the development of these Kafka clients. They make these clients more robust so that you can confidently deploy them in production.

Programming Language 101

Programming Language Kafka Programming Scala 101

Data Engineering Podcast

MARCH 2, 2020

In this episode Michael Drogalis, product manager for ksqlDB at Confluent, explains how the system is implemented, how you can use it for building your own stream processing applications, and how it fits into the lifecycle of your data infrastructure. Set up a demo and mention you’re a listener for a special offer!

Kafka 100

Kafka Process PostgreSQL MySQL 100

Data Engineering Podcast

JULY 10, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. RudderStack helps you build a customer data platform on your warehouse or data lake. In fact, while only 3.5%

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

DECEMBER 10, 2017

When you are processing that data in multiple systems it can be difficult to ensure that they all have an accurate representation of that schema, which is why Confluent has built a schema registry that plugs into Kafka. Summary To process your data you need to know what shape it has, which is why schemas are important.

Kafka 100

Kafka Data Pipeline Data Engineering Data Engineer 100

Data Engineering Weekly

NOVEMBER 26, 2023

The companies backing Apache Hudi and Iceberg write articles about comparative ACID support in both the platforms here and here. Databricks is already doing a lot of these move-up stack products. Unless the Fed reduces the interest rate to zero, the unified platform is the trend for foreseeing.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Rockset

AUGUST 16, 2022

Real quick disclaimer, I currently work at Rockset but previously worked at Confluent, a company known for building Kafka based platforms and cloud services. You can look at the source code, deploy it wherever you want and even fork the source code, create a new product and sell it! Let’s find out!

Kafka 52

Kafka AWS Cloud Java 52

Confluent

MAY 14, 2019

We’re grateful that the Apache Kafka ® and Confluent communities are composed of hundreds of thousands of people all over the world who are constantly doing all of these things. Thus, we are very excited to launch the Confluent Community Catalyst Program. What does it take to be a Confluent Community Catalyst? You know things.

Programming 83

Programming Kafka Education Coding 83

Confluent

APRIL 10, 2019

Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites. The key to the solution is Confluent Replicator, which enables frictionless data replication between sites. Try it out yourself.

Kafka 86

Kafka Metadata Java Cloud 86

Confluent

MARCH 24, 2020

In July 2017, Confluent launched a private preview of what would later be known as Confluent Cloud. This platform as a service product has grown rapidly; less than three years […].

Cloud 68

Cloud Building Architecture Kafka 68

Rockset

APRIL 26, 2023

An insurance company sets thresholds for various types of support tickets, identifying issues with services or products before they affect revenue. This architecture has a few key components: Streaming data : Streaming data is generated by website activity and transported to Rockset via Confluent Cloud.

Architecture 52

Architecture Transportation Data Lake Insurance 52

Rockset

OCTOBER 21, 2022

Real-time customer 360 applications are essential in allowing departments within a company to have reliable and consistent data on how a customer has engaged with the product and services. We’ll see the customer’s 360 profile that’ll include their product recommendations. In my workshop, I used Mockaroo data and sent that to Kafka.

Kafka 52

Kafka Building SQL Database 52

Data Engineering Podcast

AUGUST 10, 2020

The team at Iteratively are building a platform to manage the end to end flow of collaboration around what events are needed, how to structure the attributes, and how they are captured. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Data Collection 100

Data Collection Data Engineering Data Engineer Education 100

Confluent

FEBRUARY 6, 2019

This structure worked well for production training and deployment of many models but left a lot to be desired in terms of overhead, flexibility, and ease of use, especially during early prototyping and experimentation [where Notebooks and Python shine]. Impedance mismatch between data scientists, data engineers and production engineers.

Machine Learning 107

Machine Learning Python Kafka Java 107

Monte Carlo

MARCH 10, 2022

In particular, it’s exciting to see product-led growth companies and those in the developer tool space continue to push the envelope of what’s possible.” Data reliability is no longer a choice but a necessity, and this recognition only underscores how data observability is becoming an indispensable component of the modern data platform.”

Technology 52

Technology Data 52

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. In this example, we’ll assume that you already have a Kafka cluster and topic setup.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Knowledge Hut

JULY 3, 2023

Data-driven Innovations: Due to a continuous stream of data that can be used for product development and innovation. Assessing real-time data can help to recommend customer preferences, identify trends and patterns, and allow organizations to build new products and services that serve customers better.

Data Ingestion 52

Data Ingestion Pipeline-centric Google Cloud Media 52

Data Engineering Podcast

MAY 6, 2019

FoundationDB is a distributed key-value store that provides the primitives that you need to build a custom database platform. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Systems 100

Systems MongoDB NoSQL Database 100

Rockset

JANUARY 3, 2023

Lenley Hansarling, the Chief Product Officer at Aerospike, predicts that real-time streaming data will continue to pick up in 2023 and be used for high-value initiatives. Danica Fine, a Senior Developer Advocate at Confluent, echoes this sentiment: “This year, the concept of data as a product will become more mainstream.

Data Warehouse 52

Data Warehouse Education Database Data 52

Data Engineering Weekly

APRIL 2, 2023

Come and hear talks from companies like StarTree, Confluent, LinkedIn, DoorDash, Imply, and Uber on how they are advancing the state-of-the-art in user-facing analytics delivered instantly. Google announces DataPerf, the first community, and platform to build leaderboards for data benchmarks. I echoed a similar statement here.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Rockset

AUGUST 21, 2019

Implementing a working plugin What is Kafka Connect and Confluent Hub? The core functionality of a Kafka Connect plugin is just that - the Kafka Connect platform takes care of the rest, including calling the method for every event in the topics the user lists and serializing or deserializing the data in the Kafka stream.

Kafka 40

Kafka IT Data Storage Relational Database 40

Snowflake

AUGUST 16, 2023

At Snowflake Summit, we announced a wave of product innovations: Snowpark ML Modeling API, Snowflake Native App Framework, Dynamic Tables and more. The Snowflake Native App Framework remains available in private preview on Google Cloud Platform and Azure. If you missed out, not to worry! Learn more here. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Confluent

MARCH 26, 2019

Moreover, these distributed components can be implemented in different programming languages, deployed on different platforms and managed by different teams. Numerous connectors are available from Confluent Hub , and there is an API for building your own if you can’t find what you need.

Kafka 111

Kafka Transportation Metadata Consulting 111

Lyft Engineering

NOVEMBER 29, 2023

For new platform engineers, it presented a steeper learning curve. We involved our stakeholder users in these tests and took queries running in our Druid production system, dynamically transpiled them to ClickHouse syntax, and fired both queries against Druid and ClickHouse respectively.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Data Engineering Weekly

MARCH 26, 2023

Come and hear talks from companies like StarTree, Confluent, LinkedIn, DoorDash, Imply, and Uber on how they are advancing the state-of-the-art in user-facing analytics delivered instantly. 71% of respondents rated data team productivity and agility positively, while data ownership ranked as a top concern for most.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Monte Carlo

AUGUST 17, 2021

With this round, we will fuel the growth of the Data Observability category, further develop our product offerings to better serve our customers, support more use cases, and expand to new markets. Once trust is lost, it’s hard to gain back.

Insurance 40

Insurance Data Pipeline Kafka Data Engineering 40

Big Data Tools

JUNE 8, 2022

On top of that, it’s a part of the Hadoop platform, which created additional work that we otherwise would not have had to do. DataHub is a completely independent product by LinkedIn, and the folks there definitely know what metadata is and how important it is. However, you can find a diff with the 4.0.0 version on GitHub.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

JUNE 8, 2022

On top of that, it’s a part of the Hadoop platform, which created additional work that we otherwise would not have had to do. DataHub is a completely independent product by LinkedIn, and the folks there definitely know what metadata is and how important it is. However, you can find a diff with the 4.0.0 version on GitHub.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Confluent

APRIL 3, 2019

Let’s consider a simple example of products with their associated SKUs: ksql> SELECT SKU, PRODUCT FROM PRODUCTS; H1235 | Toaster H1425 | Kettle F0192 | Banana F1723 | Apple x1234 | Cat. The leading character of the SKU encodes the department of the product. we’ve also stepped up the maturity of the product.

Food 95

Food Kafka Bytes Data Cleanse 95

Monte Carlo

SEPTEMBER 12, 2022

Taking all the data that flowed in from the dozens—if not hundreds—of SaaS products a company uses and piping it into a warehouse, where it could be managed and queried, took a lot of time and energy. First, open-source is becoming more prevalent throughout the modern tech stack, as it’s continually layered within other software products. “If

BI 52

BI Data Governance ETL Tools Data Warehouse 52

Monte Carlo

JANUARY 25, 2023

Over the past year, data contracts have taken the data world by storm as a novel approach to ensuring data quality at scale in production services. In this article, Chad Sanderson , Head of Product, Data Platform , at Convoy and creator of Data Quality Camp , introduces a new application of data contracts: in your data warehouse.

Data Warehouse 52

Data Warehouse Data High Quality Data Metadata 52

Rockset

JUNE 2, 2020

If you’re short on time, fear not- here are our top 10 MongoDB sessions to watch out for: 10 Join the Data Movement: MongoDB and Apache Kafka One of the go-to picks for companies that need a streaming platform is Apache Kafka. If you feel that your platform needs a facelift, this talk is for you. How can hedged reads change that?

MongoDB 52

MongoDB Data Lake Kafka NoSQL 52

Rockset

AUGUST 28, 2023

Here we track every click to every product in our webshop, and then ingest this data into Rockset via Confluent Cloud. However, for sales analytics purposes, we want to keep a copy of specific events, where the event was a product order. Products p ON e.EventDetails.ProductID = p. Let’s look at an e-commerce example.

AWS 52

AWS SQL Datasets Metadata 52

Big Data Tools

JULY 13, 2022

In a nutshell, InLong is a SaaS-based streaming platform that scales. The platform’s new features include MLflow 2.0, serverless model endpoints, model monitoring, and many other features aimed at MLOps and production-ready data science models and experiments. The State of Data Engineering 2022 – I like this kind of content.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

JULY 13, 2022

In a nutshell, InLong is a SaaS-based streaming platform that scales. The platform’s new features include MLflow 2.0, serverless model endpoints, model monitoring, and many other features aimed at MLOps and production-ready data science models and experiments. The State of Data Engineering 2022 – I like this kind of content.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52