ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data. What is Apache Kafka Used For?

Kafka 52

Kafka Banking Medical Healthcare 52

Cloudera

SEPTEMBER 13, 2023

Soon after, I became a huge fan of Apache Kafka. Coupled with amazing technology was an amazing team that only grew and improved with the merger with Cloudera. It’s the Apache Software Foundation’s biggest event of the year. The post A Talented Team, Innovative Technology, and The Opportunity to Grow.

Technology 81

Technology Hadoop Kafka Project 81

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Confluent

MAY 23, 2019

Let us cut to the chase: Kafka Summit London session videos are available! This year’s London event was almost double the size of last year, so I’m especially happy to see these numbers on the page: 2 days of Apache Kafka ®. World-class Kafka keynotes. And if you weren’t there? Well, dig in and start learning! 3 keynotes.

Kafka 109

Kafka Cloud Database Engineering 109

Knowledge Hut

NOVEMBER 19, 2023

With advances in technology, wearable devices do provide some trace of your health. Furthermore, Data Scientists analyze this data, build reports and train ML models on top of it. This data can be used to build reports (batch or real-time) and train Machine Learning models. How do they collect, process, and analyze data for you?

Data Science 52

Data Science Technology Machine Learning Big Data 52

Data Engineering Podcast

JULY 27, 2021

The technology to make that possible has been around for a number of years, but the barriers to adoption have still been high due to the level of technical understanding and operational capacity that have been required to run at scale. How do you handle migrating existing projects, particularly if they are using Kafka currently?

Building 100

Building Management Kafka Data Warehouse 100

Rockset

AUGUST 16, 2022

Streams for Everyone If you have come this far it means you have already considered or are considering using event streaming in your data architecture for the wide variety of benefits it can offer. In either case, both Amazon Kinesis and Apache Kafka can help but which one is the right fit for you and your goals.

Kafka 52

Kafka AWS Cloud Java 52

Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JULY 1, 2019

Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Kafka 100

Kafka Finance Media Architecture 100

Confluent

MAY 13, 2019

In the last year, we’ve experienced enormous growth on Confluent Cloud, our fully managed Apache Kafka ® service. As Confluent Cloud has grown, we’ve noticed two gaps that very clearly remain to be filled in managed Apache Kafka services. Five seconds to Kafka (or, never make another cluster again!).

Kafka 100

Kafka Cloud Building Management 100

Data Engineering Podcast

MAY 15, 2022

Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 100

Data Lake Building BI Architecture 100

Confluent

JUNE 27, 2019

Confluent’s clients for Apache Kafka ® recently passed a major milestone—the release of version 1.0. Magnus Edenhill first started developing librdkafka about seven years ago, later joining Confluent in the very early days to help foster the community of Kafka users outside the Java ecosystem. Build()) {. }. In the v0.11.6

Kafka 106

Kafka Designing Java Coding 106

Confluent

FEBRUARY 28, 2019

Only a little more than one month after the first release, we are happy to announce another milestone for our Kafka integration. Today, you can grab the Kafka Connect Neo4j Sink from Confluent Hub. . Neo4j extension – Kafka sink refresher. Then it is really up to you what you want to with the event data.

Kafka 86

Kafka Java Programming Language Data Pipeline 86

Data Engineering Podcast

DECEMBER 30, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Process 100

Process Building Hadoop Java 100

Data Engineering Podcast

OCTOBER 28, 2019

In an effort to create a better abstraction for building data applications Nick Schrock created Dagster. In this episode he explains his motivation for creating a product for data management, how the programming model simplifies the work of building testable and maintainable pipelines, and his vision for the future of data programming.

Building 100

Building Data Pipeline Programming Language Metadata 100

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Pipeline Data Engineering

FEBRUARY 18, 2022

furthermore - and this might be the most frequent setup - some people come without any clue about what they really wan’t to build. some come with a data stack problem that they are currently facing at work, and they need professional guidance and clarity about how to go about solving it. And that’s totally fine.

Building 52

Building Portfolio Kafka Machine Learning 52

Confluent

MAY 2, 2019

When managing Apache Kafka ® clusters at scale, tasks that are simple on small clusters turn into significant burdens. Relatedly, KIP-226 enabled dynamic broker reconfiguration since Apache Kafka 1.1. See the documentation (or, if you please, the Apache Kafka wiki ) for a complete list of which parameters this applies to.

Kafka 87

Kafka Management Food Consulting 87

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Confluent

MAY 2, 2019

When managing Apache Kafka ® clusters at scale, tasks that are simple on small clusters turn into significant burdens. Relatedly, KIP-226 enabled dynamic broker reconfiguration since Apache Kafka 1.1. See the documentation (or, if you please, the Apache Kafka wiki ) for a complete list of which parameters this applies to.

Kafka 57

Kafka Management Food Consulting 57

Data Engineering Podcast

SEPTEMBER 28, 2020

Summary Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine.

Kafka 100

Kafka BI Big Data Data Engineering 100

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Those challenges are a thing of the past with RudderStack’s Kafka source integration.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases. 8, and so on. Here we go!

Kafka 55

Kafka Process Algorithm Cloud 55

Data Engineering Podcast

MAY 11, 2020

His most recent endeavor at StreamNative is focused on combining the capabilities of Pulsar with the cloud native movement to make it easier to build and scale real time messaging systems with built in event processing capabilities. How have projects such as Kafka and Pulsar impacted the broader software and data landscape?

Lambda Architecture 100

Lambda Architecture Cloud Kafka Hadoop 100

Data Engineering Podcast

AUGUST 3, 2021

By adding support for small, incremental inserts into large table structures, and building support for arbitrary update and delete operations the Hudi project brings the best of both worlds together. Go to dataengineeringpodcast.com/97things today to get your copy! RudderStack’s smart customer data pipeline is warehouse-first.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Knowledge Hut

MAY 2, 2024

Since then, Apache Spark has seen a very high adoption rate from top-notch technology companies like Google, Facebook, Apple, Netflix etc. Spark streaming also has in-built connectors for Apache Kafka which comes very handy while developing Streaming applications. Apache Spark was developed by a team at UC Berkeley in 2009.

Scala 52

Scala Hospitality Healthcare Retail 52

Knowledge Hut

JUNE 26, 2023

Before working on these initiatives, you should be conversant with topics and technologies. Top 4 Data Engineering Project Ideas: Beginner & Final Year Students Becoming an expert data engineer necessitates familiarity with the best practices and cutting-edge technologies in your field.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Confluent

MAY 26, 2020

Using a powerful, event-driven application can help you unlock insights contained in the event streams of your business. Before we get into the technology, let’s go over some questions you […].

Building 116

Building Technology Kafka Process 116

Data Engineering Podcast

MARCH 8, 2021

This introduces challenges such as keeping information synchronized, managing schema evolution, building transformations to match the expectations of the destination systems. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more.

IT 100

IT Data Warehouse MongoDB Kafka 100

Data Engineering Podcast

JULY 6, 2020

In this episode CTO Andrew Stevenson discusses the challenges that arise from building decoupled systems, the benefits of using SQL as the common interface for your data, and the metrics that need to be tracked to keep the overall system healthy. What was your reason for building your own SQL engine and what is unique about it?

Systems 100

Systems Kafka SQL Government 100

Precisely

SEPTEMBER 12, 2023

Precisely data integrity solutions fuel your Confluent and Apache Kafka streaming data pipelines with trusted data that has maximum accuracy, consistency, and context and we’re ready to share more with you at the upcoming Current 2023. Let’s cover some additional information to know before attending.

Kafka 52

Kafka Data Integration Data Pipeline Datasets 52

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform. Need to automate data pipelines with less red tape?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

MAY 10, 2021

Summary Building an API for real-time data is a challenging project. This is a great conversation about the challenges of building a maintainable business from a technical and product perspective. Can you start by describing what you are building at Tinybird and the story behind it? How is the Tinybird platform architected?

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. When is CDC the wrong choice?

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

JANUARY 5, 2020

Debezium is an open source platform for reliable change data capture that you can use to build supplemental systems for everything from maintaining audit trails to real-time updates of your data warehouse. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh.

Database 100

Database Kafka PostgreSQL MySQL 100

Data Engineering Podcast

DECEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

MAY 8, 2022

Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Closing Announcements Thank you for listening!

Database 100

Database Data Lake BI Business Intelligence 100