Data Engineering Podcast

JULY 1, 2019

Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Kafka 100

Kafka Finance Media Architecture 100

Precisely

DECEMBER 28, 2023

Enterprise technology is having a watershed moment; no longer do we access information once a week, or even once a day. Streaming data pipelines, by extension, offer an architecture capable of handling large volumes of data, accommodating millions of events in near real time. But insights derived from day-old data don’t cut it.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

Rockset

JULY 28, 2022

CDC enables true real-time analytics on your application data, assuming the platform you send the data to can consume the events in real time. Options For Change Data Capture on MongoDB Apache Kafka The native CDC architecture for capturing change events in MongoDB uses Apache Kafka.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Lyft Engineering

NOVEMBER 29, 2023

It also allowed us to optimize for handling time-series data and event data at scale. Real-time Ingestion Events from our real-time analytics pipeline were configured to be sent into our internal Flink application, streamed to Kafka, and written into Druid. ioConfig: Kafka server info, topic names, etc. (ex.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Cloudera

FEBRUARY 21, 2023

This transformation can be performed on incoming records of a Kafka topic before SSB sees the data. If the Kafka topic has CSV data that we want to add keys and types to it. If the schema you want does not match the incoming Kafka topic. data transformations can be defined using the Kafka Table Wizard.

SQL 108

SQL Kafka Raw Data Data 108

Rockset

APRIL 26, 2022

As Kafka Summit is in full swing in London this week and the topic of event streaming is all over my Linkedin feed, I saw a post asking " Is streaming dead? That streaming is rocking and with Kafka Summit this week, I thought it a good time to emphasize the importance of streaming data in today’s modern real-time data stack.

Kafka 52

Kafka Data Warehouse Database Data 52

Netflix Tech

JUNE 1, 2023

New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing. It also included metadata about ads, such as ad placement and impression-tracking events. We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. In an identity/access management application, it’s the relationships between roles and their privileges that matters most. The approach we’ll use works with any Kafka run though. 8, and so on.

Kafka 55

Kafka Process Algorithm Cloud 55

Cloudera

JUNE 7, 2021

It enabled users to easily write, run and manage real-time SQL queries on streams from Apache Kafka with an exceptionally smooth user experience. . Improved Kafka and Schema Registry integration. Flink SQL catalogs are now supported directly on the streambuilder platform allowing easy access to data stored in other systems.

Kafka 101

Kafka SQL Accessible Accessibility 101

Christophe Blefari

MARCH 17, 2023

Airflow alternatives meetup Click on the image to go the the LinkedIn event. We are organising next week with the Paris Apache Airflow Meetup group an online event to discuss about Airflow alternatives. We will host another event soon after with others. You can either register on LinkedIn either join the meetup event.

Data 130

Data SQL Deep Learning Kafka 130

Netflix Tech

AUGUST 30, 2021

In the Device Management Platform, this is achieved by having device updates be event-sourced through the control plane to the cloud so that NTS will always have the most up-to-date information about the devices available for testing. In particular, the Kafka integration is the most relevant for this blog post.

Management 85

Management Kafka Transportation Cloud 85

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Can you describe your experiences with Kafka? What are the operational challenges that you have had to overcome while working with Kafka?

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

SEPTEMBER 28, 2020

Summary Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine.

Kafka 100

Kafka BI Big Data Data Engineering 100

Cloudera

JULY 18, 2022

This information will be efficiently fed to downstream systems through Kafka, so that appropriate actions, like blocking the card or calling the user, can be initiated immediately. The scored transactions are written to the Kafka topic that will feed the real-time analytics process that runs on Apache Flink. Apache Flink.

Process 86

Process Kafka Scala SQL 86

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Figure 1: sales group SELECT access.

Cloud 97

Cloud Kafka Metadata SQL 97

Cloudera

MARCH 14, 2023

When a draft is ready to be deployed in production, it is published to the Catalog, and can be productionalized with serverless DataFlow Functions for event-driven, micro-bursty use cases or auto-scaling DataFlow Deployments for low latency, high throughput use cases. You want to see how to get data from Kafka and write it to Iceberg?

Data Pipeline 81

Data Pipeline Designing Kafka Metadata 81

Data Engineering Weekly

SEPTEMBER 24, 2023

Request Access to Data Validate with Exploration [link] Ramp: How Ramp Accelerated Machine Learning Development to Simplify Finance Ramp writes about its machine learning infrastructure and the choice of Metaflow for running the ML workload. Those challenges are a thing of the past with RudderStack’s Kafka source integration.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Cloudera

JANUARY 31, 2024

Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. Introduction Kafka as an event streaming component can be applied to a wide variety of use cases. and its use is no longer recommended.

Management 69

Management Kafka Big Data Cloud 69

Knowledge Hut

JUNE 26, 2023

Top Data Engineering Projects with Source Code Data engineers make unprocessed data accessible and functional for other data professionals. Source Code: Stock and Twitter Data Extraction Using Python, Kafka, and Spark 2. If you are struggling with Data Engineering projects for beginners, then Data Engineer Bootcamp is for you.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Cloudera

MAY 1, 2023

Cloudera is therefore uniquely able to decouple the development of business/event logic from other aspects of application development, to further empower domain experts and accelerate development of real time data apps. Resources that the user has access to can be found under “External Resources”.

SQL 76

SQL Project Kafka Accessible 76

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

Data Engineering Weekly

AUGUST 6, 2023

link] Sponsored: Great Data Debate–The State of Data Mesh Since 2019, the data mesh has woven itself into every blog post, event presentation, and webinar. Access the Report AWS: A side-by-side comparison of Apache Spark and Apache Flink for common streaming use cases Is Flink a better choice for streaming or Spark streaming?

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Podcast

JULY 27, 2021

Datastax has recently introduced a new managed offering for Pulsar workloads in the form of Astra Streaming that lowers those barriers and make stremaing workloads accessible to a wider audience. How do you handle migrating existing projects, particularly if they are using Kafka currently?

Building 100

Building Management Kafka Data Warehouse 100

Data Engineering Podcast

MAY 11, 2020

His most recent endeavor at StreamNative is focused on combining the capabilities of Pulsar with the cloud native movement to make it easier to build and scale real time messaging systems with built in event processing capabilities. How have projects such as Kafka and Pulsar impacted the broader software and data landscape?

Lambda Architecture 100

Lambda Architecture Cloud Kafka Hadoop 100

Cloudera

FEBRUARY 17, 2021

Spin up clusters of NiFi, Kafka, or Flink very quickly onto your public cloud environments on AWS or Azure. Extending the Kafka ecosystem to CDP – Cloudera (and Hortonworks) has always prided itself on its investment and innovations around Apache Kafka. Our keynote , delivered by our VP of Engineering, Joe Witt, and Sr.

Kafka 60

Kafka Food Manufacturing Healthcare 60

Precisely

SEPTEMBER 12, 2023

Precisely data integrity solutions fuel your Confluent and Apache Kafka streaming data pipelines with trusted data that has maximum accuracy, consistency, and context and we’re ready to share more with you at the upcoming Current 2023. Let’s cover some additional information to know before attending.

Kafka 52

Kafka Data Integration Data Pipeline Datasets 52

Knowledge Hut

MAY 3, 2024

Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Spark Streaming Kafka Streams 1 Data received from live input data streams is Divided into Micro-batched for processing. 6 Spark streaming is a standalone framework.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Confluent

JUNE 12, 2019

Confluent Cloud, a fully managed event cloud-native streaming service that extends the value of Apache Kafka ® , is simple, resilient, secure, and performant, allowing you to focus on what is important—building contextual event-driven applications, not infrastructure. KSQL and Kafka Connect example. and Helm/Tiller 2.8.2+

Cloud 92

Cloud Kafka Healthcare Software Engineer 92

Precisely

FEBRUARY 26, 2024

Observability helps IT professionals to link causes and effects within a highly complex and interactive chain of events. Getting Access to the Right Information To drive better insights and agility, IT teams need access to the logs, metrics, and traces that originate on their mainframe & IBM i systems.

IT 75

IT Banking Programming Language Systems 75

Rockset

SEPTEMBER 26, 2023

AI-powered applications almost always need access to real-time data to deliver accurate results in a responsive user experience that the market has come to expect. As smart as ChatGPT appears to be, it can’t summarize current events accurately if it was last trained a year ago and not told what’s happening now.

Metadata 64

Metadata Kafka Cloud Database 64

Snowflake

AUGUST 16, 2023

Custom Event Billing capabilities enable you to build your own pricing strategy. You can charge customers based on their usage and specify billing events based on your preferences, such as consumed rows, ingested rows and more. The new Kafka connector, built with Snowpipe Streaming , now supports schema detection and evolution.

Scala 52

Scala Transportation Kafka Data Lake 52

Cloudera

DECEMBER 9, 2022

This observation further emphasizes the need for universal developer accessibility , which makes sure that developer tooling is easy to use for newcomers while giving power users the advanced options they need. A critical aspect of universal developer accessibility is to provide dataflow development as a self-service offering to developers.

Designing 96

Designing Coding Google Cloud AWS 96

Data Engineering Podcast

SEPTEMBER 25, 2022

With the increasing expecation for information to be instantly accessible, it drives the need for reliable change data capture. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

MARCH 8, 2021

explains how using Pilosa as the core he built the Molecula platform to eliminate the need to copy data between systems in able to make it accessible for analytical and machine learning purposes. After tasking some of his top engineers to consider the problem in a new light they created the Pilosa engine. In this episode H.O.

IT 100

IT Data Warehouse MongoDB Kafka 100

Zalando Engineering

NOVEMBER 29, 2017

See Ranking Websites in Real-time with Apache Kafka’s Streams API for the first post in the series. Running Kafka Streams applications in AWS At Zalando, Europe’s leading online fashion platform, we use Apache Kafka for a wide variety of use cases. Our team at Zalando was an early adopter of the Kafka Streams API.

Kafka 40

Kafka AWS Amazon Web Services Utilities 40