Cloudera

MARCH 2, 2023

In this blog post, we are going to share with you how Cloudera Stream Processing ( CSP ) is integrated with Apache Iceberg and how you can use the SQL Stream Builder ( SSB ) interface in CSP to create stateful stream processing jobs using SQL. To provide the CM host we can copy the FQDN of the node where Cloudera Manager is running.

Process 113

Process SQL Kafka Database 113

Pinterest Engineering

NOVEMBER 7, 2023

A central component of data ingestion infrastructure at Pinterest is our PubSub stack, and the Logging Platform team currently runs deployments of Apache Kafka and MemQ. years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 99

Kafka Java Software Engineer Software Engineering 99

Cloudera

MAY 1, 2023

The newly introduced Environments feature allows you to export only the generic, reusable parts of code and resources, while managing environment-specific configuration separately. Businesses everywhere have engaged in modernization projects with the goal of making their data and application infrastructure more nimble and dynamic.

SQL 77

SQL Project Kafka Accessible 77

LinkedIn Engineering

OCTOBER 19, 2023

In this case study, LinkedIn's Bingfeng Xia, Engineering Manager, and Xinyu Liu, Senior Staff Engineer, shed light on how the Apache Beam programming model's unified, portable, and user-friendly data processing framework has enabled a multitude of sophisticated use cases and revolutionized streaming processing at LinkedIn.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Rockset

MARCH 18, 2024

Rockset customers using Flink often share how challenging it is to self-manage Flink for streaming transformations. That’s why we’re thrilled that Confluent Cloud is making it easier to use Flink, providing efficient and performant stream processing while saving engineers from complex infrastructure management. What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Cloudera

MARCH 11, 2022

In this release , the Streams Messaging templates in Data Hub will come with Apache Kafka 2.8 Kafka & Cruise Control Updates. Kafka Updates: Deployments with Kafka 2.5 clusters can now be upgraded to Kafka 2.8, benefitting from all the improvements and features from Kafka 2.6, 27 and 2.8.

Cloud 104

Cloud Kafka Utilities Database 104

Christophe Blefari

FEBRUARY 24, 2023

This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. But if you want to continue using the underlying tools here an overlook of Flink architecture or a few techniques you should know as a Kafka streams developer. Is it Kafka?

Kafka 130

Kafka Data Lake Data Storage Data 130

Cloudera

OCTOBER 25, 2021

From this release, Streams Messaging templates will support scaling with automatic rebalancing allowing you to grow or shrink your Apache Kafka cluster based on demand. Kafka Scaling. It should be noted that at the time of writing this, clusters that upgrade to 7.2.12 Clusters newly provisioned with 7.2.12

Cloud 88

Cloud Kafka Metadata Management 88

Cloudera

OCTOBER 4, 2022

This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes.

Kafka 78

Kafka Data Cloud Big Data 78

Cloudera

NOVEMBER 27, 2023

This blog is a summary of what we observed there for anyone who was unable to attend or just wants to stay on top of what’s happening in streaming. The conference had a laser focus on the open source technology and the developers who bring it to their organizations. Flink is, relatively speaking, a newer technology. billion events/s.

Kafka 81

Kafka SQL ETL Tools Data Lake 81

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka? How Apache Kafka streams relate to Franz Kafka’s books. What Kafka is used for.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

Netflix Tech

MARCH 10, 2023

By Meenakshi Jindal Overview At Netflix, we built the asset management platform (AMP) as a centralized service to organize, store and discover the digital media assets created during the movie production. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

Data Engineering Weekly

SEPTEMBER 11, 2023

Niels Claeys: Head-to-head comparison of 3 dbt SQL engines The blog post compares benchmarks for the popular dbt SQL engines Trino, DuckDB, and Spark. The blog narrates the key concepts of the Kimball model and a modern outlook on the concepts. See how it works today. era with automation, collaboration, and AI.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Knowledge Hut

JANUARY 18, 2024

“Big data Analytics” is a phrase that was coined to refer to amounts of datasets that are so large traditional data processing software simply can’t manage them. Get to know more about measures of dispersion through our blogs. Diving into Big Data Analytics has really been an eye-opener for me. What Are Big Data Frameworks?

Data Process 52

Data Process Process Hadoop Scala 52

DoorDash Engineering

MARCH 27, 2024

At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. We will delve here into how we set up multi-tenancy with a messaging queue system based on Kafka. In Kafka, a test tenant processing production event can cause data inconsistencies, including outages and other incidents.

Kafka 103

Kafka Architecture Algorithm Systems 103

Cloudera

JULY 15, 2021

This blog post provides an overview of best practice for the design and deployment of clusters incorporating hardware and operating system configuration, along with guidance for networking and security as well as integration with existing enterprise infrastructure. Traditional data clusters for workloads not ready for cloud.

Architecture 99

Architecture Cloud Kafka Hadoop 99

Jesse Anderson

JANUARY 9, 2023

I’ve always been vocal about ksqlDB’s and Kafka Stream’s limitations. The Future of ksqlDB and Kafka Streams With this announcement, the future of primarily ksqlDB and, to a lesser extent, Kafka Streams comes into view. Since Kafka Streams is part of the Apache project, I don’t see it going away as quickly.

Kafka 147

Kafka Coding Technology SQL 147

Cloudera

SEPTEMBER 28, 2020

They have dev, test, and production clusters running critical workloads and want to upgrade their clusters to CDP Private Cloud Base. Support Kafka connectivity to HDFS, AWS S3 and Kafka Streams. Cluster management and replication support for Kafka clusters. New Features CDH to CDP.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Knowledge Hut

APRIL 25, 2024

In this blog post, we will discuss such technologies. Big data storage technologies is a compute-and-storage architecture that collects and manages large data sets while also allowing real-time data analytics. Let's explore the technologies for managing and analyzing big data. It is especially true in the world of big data.

Big Data 98

Big Data Technology NoSQL Hadoop 98

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Arjun Mohnot , Jenchang Ho , Anthony Quigley , Xing Lin , Anil Alluri , Michael Kuchenbecker LinkedIn operates one of the world’s largest Apache Hadoop big data clusters. Historically, deploying code changes to Hadoop big data clusters has been complex.

Big Data 92

Big Data Hadoop Metadata Data 92

Cloudera

FEBRUARY 10, 2022

Without a cloud-native service to run NiFi flows on Microsoft Azure, organizations resorted to building and operating NiFi clusters on either virtual machines or their own container based infrastructure. A common use case on Azure is SIEM Optimization (SIEM=security information and event management) for analyzing application log data.

Cloud 114

Cloud Kafka AWS Data Ingestion 114

Data Engineering Weekly

MARCH 31, 2024

The blog further emphasizes its increased investment in Data Mesh and clean data. The blog is an excellent overview of all the improvements made to PySpark in 2023. The blog narrates how Uber utilizes the existing infrastructure to support various AI/ML use cases.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Christophe Blefari

DECEMBER 3, 2022

Joe Reis launched his Substack — Joe is the co-author of the great The Fundamentals of Data Engineering and his blog already have 2 articles I deeply recommend: No extra credit for complexity & Groundhog Days. I'm terribly late with this newsletter. What if we decide to schedule Airflow DAGs in Gitlab?

Kafka 130

Kafka BI Finance Data 130

Cloudera

AUGUST 10, 2021

x customers to upgrade in-place to CDP PvC Base without having to create a new cluster. Cloudera Manager enhancements for greater efficiency and hardened platform security. We expand on this feature later in this blog. With the release of CDP Private Cloud (PvC) Base 7.1.7, please review the Release Summary.

Cloud 96

Cloud Kafka Metadata SQL 96

Cloudera

JANUARY 21, 2022

This blog will summarise the security architecture of a CDP Private Cloud Base cluster. The release of CDP Private Cloud Base has seen a number of significant enhancements to the security architecture including: Apache Ranger for security policy management. Updated Ranger Key Management service. No security configured.

Architecture 109

Architecture Transportation Certification Government 109

Cloudera

AUGUST 16, 2021

We are excited to announce the general availability of Cloudera DataFlow for the Public Cloud (CDF-PC) – a brand new experience on the Cloudera Data Platform (CDP) to address some of the key operational and monitoring challenges of standard Apache NiFi clusters that are overloaded with high-performant flows.

Cloud 108

Cloud AWS Kafka Utilities 108

Cloudera

OCTOBER 19, 2022

In this post I will demonstrate how Kafka Connect is integrated in the Cloudera Data Platform (CDP), allowing users to manage and monitor their connectors in Streams Messaging Manager while also touching on security features such as role-based access control and sensitive information handling. Kafka Connect.

Kafka 72

Kafka MySQL Bytes Data 72

Cloudera

JULY 21, 2022

We want Cloudera customers that rely on Apache Druid to know that their clusters are secure and supported by the Cloudera partner ecosystem. Rill relieves that burden with a managed service offering or Druid monitoring for existing clusters. The Druid native Kafka indexing service features: Pull-based ingestion.

BI 84

BI Digital Media Data Warehouse Kafka 84

Netflix Tech

NOVEMBER 3, 2023

In our last blog post, we introduced “Data Mesh” — A Data Movement and Processing Platform. The pipeline is composed of individual “Processors” that are connected by Kafka topics. By keeping the logic of individual Processors simple, it allowed them to be reusable so we could centrally manage and operate them at scale.

SQL 106

SQL Pipeline-centric Kafka Data 106

Pinterest Engineering

NOVEMBER 28, 2023

Average GC pauses in some high volume clusters can be as high as 10 seconds per minute. When there is a full GC, it leads to full halt to the data processing pipeline and causes both back-pressure for upstream kafka clusters and cascading failure for downstream TSDB. Generally Goku-Ingestor has a throughput of 2.5

Kafka 81

Kafka Bytes Architecture Utilities 81

Cloudera

MARCH 15, 2023

In this post I am going to set up a demo environment with a Spring Boot microservice and a streaming cluster using Cloudera Public Cloud. This external consumer can be an asynchronous process that scans the “outbox” table or the database logs for new entries, and sends the message to an event bus, such as Apache Kafka.

PostgreSQL 88

PostgreSQL Kafka Database Data 88

Data Engineering Weekly

AUGUST 6, 2023

The first one that caught my eye is Astronomer published a blog post Introducing Cosmos 1.0: The blog narrates how to schedule dbt jobs in Airflow by parsing dbt's manifest.json file, and auto construct Airflow tasks. See how it works today. Editor’s Note: DewCon.ai Registration is Now Open Great news! See you there!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

LinkedIn Engineering

MARCH 23, 2023

In this blog post, we will share our progress, challenges, and lessons learned from implementing Apache Beam. Co-Authors: Yuhong Cheng , Shangjin Zhang , Xinyu Liu, and Yi Pan Efficient data processing is crucial in reducing learning curves, simplifying maintenance efforts, and decreasing operational complexity.

Process 97

Process Lambda Architecture Kafka Datasets 97

Cloudera

SEPTEMBER 13, 2021

There are two big gaps in the Apache Kafka project when we think of operating a cluster. The first is monitoring the cluster efficiently and the second is managing failures and changes in the cluster. Cruise Control is integrated with Kafka through metrics reporting. About Cruise Control. Architecture.

Kafka 71

Kafka Bytes Utilities Management 71

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. And don’t forget to thank them for their continued support of this show! Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Netflix Tech

JANUARY 13, 2022

This blog will explore these two systems and how they perform auto-diagnosis and remediation across our Big Data Platform and Real-time infrastructure. These steps launch jobs on clusters running Apache Spark and Presto via Genie. And we can’t discount the productivity impact it causes on data platform users.

Kafka 95

Kafka Big Data Data Machine Learning 95

Netflix Tech

MARCH 10, 2023

By Burak Bacioglu , Meenakshi Jindal Asset Management at Netflix At Netflix, all of our digital media assets (images, videos, text, etc.) We built an asset management platform (AMP), codenamed Amsterdam , in order to easily organize and manage the metadata, schema, relations and permissions of these assets.

Management 103

Management Metadata Digital Media Kafka 103