Confluent

APRIL 10, 2019

Previously in 3 Ways to Prepare for Disaster Recovery in Multi-Datacenter Apache Kafka Deployments , we provided resources for multi-datacenter designs, centralized schema management, prevention of cyclic repetition of messages, and automatic consumer offset translation to automatically resume applications.

Kafka 86

Kafka Metadata Java Cloud 86

Data Engineering Weekly

MAY 16, 2023

I won’t bore you with the importance of data quality in the blog. Since Kafka is almost synonymous with real-time data processing, we often call this a “Fronting Kafka” pattern. The Fronting Kafka pattern follows a two-cluster approach. Should We Build a New Tool? Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Christophe Blefari

JANUARY 20, 2024

The idea behind is to solve data problem by building software. Main technologies around stream are bus messages like Kafka and processing framework like Flink or Spark on top of the bus. Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill). This is not.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

MARCH 14, 2023

In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities. In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development.

Data Pipeline 81

Data Pipeline Designing Kafka Metadata 81

Cloudera

SEPTEMBER 28, 2020

Cloudera delivers an enterprise data cloud that enables companies to build end-to-end data pipelines for hybrid cloud, spanning edge devices to public or private cloud, with integrated security and governance underpinning it to protect customers data. Support Kafka connectivity to HDFS, AWS S3 and Kafka Streams. Kafka, SRM, SMM.

Cloud 131

Cloud Kafka Professional Services Metadata 131

AltexSoft

FEBRUARY 11, 2023

It serves as a foundation for the entire data management strategy and consists of multiple components including data pipelines; , on-premises and cloud storage facilities – data lakes , data warehouses , data hubs ;, data streaming and Big Data analytics solutions ( Hadoop , Spark , Kafka , etc.); Feel free to enjoy it.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Cloudera

JULY 15, 2021

This blog post provides an overview of best practice for the design and deployment of clusters incorporating hardware and operating system configuration, along with guidance for networking and security as well as integration with existing enterprise infrastructure. Introduction and Rationale. Role allocation. Networking .

Architecture 97

Architecture Cloud Kafka Hadoop 97

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases. 8, and so on.

Kafka 55

Kafka Process Algorithm Cloud 55

Cloudera

AUGUST 10, 2021

With Apache Ozone on the Cloudera Data Platform (CDP) , they can implement a scale-out model and build out their next generation storage architecture without sacrificing security, governance and lineage. Then you can import Kafka lineage using the Atlas Kafka import tool provided with CDP. ozone sh bucket list /data.

Hadoop 104

Hadoop Kafka Datasets Government 104

Cloudera

JANUARY 19, 2022

We have been working hard to build our cloud-native data services on Cloudera Data Platform (CDP), which include CDP Data Warehouse, CDP Operational Database, CDP Machine Learning, CDP Data Engineering and CDP Data Flow. Download the reports to see the detailed scores . 2021 Gartner Magic Quadrant for Cloud DBMS .

Database 87

Database Cloud Data Warehouse Data Lake 87

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Extending Atlas’ metadata model. The example 1_typedef-server.json describes the server typedef used in this blog. .

Data Governance 93

Data Governance Government Metadata Datasets 93

Zalando Engineering

NOVEMBER 29, 2017

See Ranking Websites in Real-time with Apache Kafka’s Streams API for the first post in the series. Running Kafka Streams applications in AWS At Zalando, Europe’s leading online fashion platform, we use Apache Kafka for a wide variety of use cases. Our team at Zalando was an early adopter of the Kafka Streams API.

Kafka 40

Kafka AWS Amazon Web Services Utilities 40

Cloudera

FEBRUARY 9, 2021

This ‘need for speed’ drives a rethink on building a more modern data warehouse solution, one that balances speed with platform cost management, performance, and reliability. In the real-time layer, Kafka automatically cleans the historical events with configurable retention. The Cost-Effective Data Warehouse Architecture.

Data Warehouse 69

Data Warehouse Cloud Kafka Cloud Storage 69

Lyft Engineering

OCTOBER 3, 2023

Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods. These flow through Kafka , our event streaming platform, before being processed by Flink , our streaming compute framework.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Rockset

JULY 8, 2020

Building a scalable, distributed, and real-time ETA prediction system is a tough task, but what if we could simplify its design? For this example, we will use Kafka. The service then pushes the geohash along with the coordinates to a Kafka topic. Rockset has a built-in connector to Apache Kafka.

Kafka 40

Kafka Designing Systems Food 40

LinkedIn Engineering

DECEMBER 19, 2023

Figure 3: Generalized rolling upgrade deployment flow Namenode deployment overview The namenode is the central component of HDFS and is responsible for storing the metadata information about files and directories in the HDFS cluster. This metadata includes the namespace, file permissions, and the mapping of data blocks to datanodes.

Big Data 92

Big Data Hadoop Metadata Data 92

ProjectPro

JANUARY 24, 2023

It is a popular ETL tool well-suited for big data environments and extensively used by data engineers today to build and maintain data pipelines with minimal effort. Its integration with other popular AWS services like Redshift, S3, and Amazon Athena makes it a valuable tool for data engineers to build end-to-end data engineering projects.

AWS 52

AWS Data Lake Scala ETL Tools 52

Cloudera

DECEMBER 9, 2022

Now, we shift focus on the needs of developers and addressing the challenges they face when building dataflows in the cloud. We’ve observed organizations using more and more data sources and destinations , as well as expecting a more diverse range of developers to build data movement flows. Enabling self-service for developers.

Designing 96

Designing Coding Google Cloud AWS 96

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Rockset

MARCH 18, 2024

While it's well-known that Flink excels at filtering, joining and enriching streaming data from Apache Kafka® or Confluent Cloud , what is less known is that it is increasingly becoming ingrained in the end-to-end stack for AI-powered applications. What are the challenges building RAG pipelines? What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Cloudera

OCTOBER 28, 2020

These processors will be dedicated to specific services that you would want to target, e.g. HDFS, YARN, Apache Kafka, Hive, Impala, etc. These variables that are extracted will be attached as metadata to the outgoing flowfile, so only extract what is needed. The post Log Reduction Techniques with CFM appeared first on Cloudera Blog.

Kafka 64

Kafka SQL Professional Services Consulting 64

Rockset

JUNE 7, 2021

The rule of thumb is that if you are looking to build a real-time data processing system then the push approach should be used. The CDC system then periodically polls the source file system to check for any new files using the file metadata it stored earlier as a reference. Any new files are then captured and their metadata stored too.

IT 40

IT Kafka Database MongoDB 40

Rockset

DECEMBER 22, 2020

There are ways to build relationships in Elasticsearch documents, most common are: nested objects, parent-child joins, and application side joins. Building with Elasticsearch In Elasticsearch I have three indexes, one for each of the data types: product, purchase, and review. To do this we will need to build three separate queries.

SQL 40

SQL Data MongoDB Aggregated Data 40

Confluent

APRIL 30, 2019

With the release of Apache Kafka ® 2.1.0, Kafka Streams introduced the processor topology optimization framework at the Kafka Streams DSL layer. In what follows, we provide some context around how a processor topology was generated inside Kafka Streams before 2.1, Kafka Streams topology generation 101.

Kafka 90

Kafka Coding Process Bytes 90

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

Pinterest Engineering

JANUARY 10, 2023

A single and shared Kafka topic will be used for all logs going through this pipeline. Architecture High level Figure 1 — architecture of the logging pipeline Schema Client side service integration will provide the metadata, and developers just need to provide the name of the log and actual log payload. Nothing else is required.

Systems 54

Systems Building Software Engineer Software Engineering 54

LinkedIn Engineering

JANUARY 19, 2024

Some of the key components include: Rest.li - An open source REST framework for building robust, scalable RESTful architectures using type-safe bindings and asynchronous, non-blocking IO. Apache Kafka - A distributed event streaming platform. The requests and processing metadata are streamed over Kafka.

Recruitment 88

Recruitment Data Process Process Kafka 88

Netflix Tech

MARCH 25, 2019

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can push or pull.

Building 87

Building Metadata Transportation Data Ingestion 87

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. Now in part 2, we’ll discuss the challenges we faced developing, building, and deploying the KSQL portion of our application and how we used Gradle to address them.

Kafka 95

Kafka Management Bytes SQL 95

Confluent

MAY 21, 2019

When you build microservices architectures, one of the concerns you need to address is that of communication between the microservices. Either way, these promises, whether an API or a schema, is what allows us to connect microservices to each other and use them to build larger applications. What will the profile service publish?

Kafka 110

Kafka Insurance Architecture Database 110

LinkedIn Engineering

MARCH 23, 2023

In this blog post, we will share our progress, challenges, and lessons learned from implementing Apache Beam. Background Apache Beam is an open sourced, unified model for building both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs , users can build a program that defines the pipeline.

Process 97

Process Lambda Architecture Kafka Datasets 97

Data Engineering Weekly

JANUARY 15, 2023

2023 predictions from the panel are; Unified metadata becomes kingmaker. The blog narrates the causal machine-learning approach in designing promotional artwork for their TV shows. RudderStack builds your CDP on top of your data warehouse, giving you a more secure and cost-effective solution.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

ProjectPro

OCTOBER 31, 2016

” While we have answered the first question for our students through a series of blog posts on commonly asked Hadoop Interview Questions and tips for preparing a Hadoop resume. The data nodes store metadata and information about which files can be mapped to which particular block location.

Hadoop 52

Hadoop Big Data Utilities NoSQL 52

Confluent

FEBRUARY 22, 2019

We have delivered an event streaming platform which gives strong guarantees on data quality, using Apache Kafka ® and Protocol Buffers. Because it builds on top of Apache Kafka we decided to call it Franz. Avro was an intriguing option, particularly because of Confluent’s support for this on Kafka.

Kafka 80

Kafka Programming Language Metadata Data 80

Picnic Engineering

SEPTEMBER 8, 2022

We use the RabbitMQ Source connector for Apache Kafka Connect. One may wonder why don’t we replace RabbitMQ with Apache Kafka everywhere? In order to answer the first question, we should take a closer look at the difference between RabbitMQ and Apache Kafka in terms of services parallelism.

Kafka 52

Kafka Metadata AWS Java 52