Cloudera

JANUARY 20, 2021

This is the first in a six-part blog series that outlines the data journey from edge to AI and the business value data produces along the journey. Fig 1: The Enterprise Data Lifecycle. Security & Governance – an integrated set of security, management and governance technologies across the entire data lifecycle.

Manufacturing 103

Manufacturing Data Warehouse Kafka Retail 103

Netflix Tech

OCTOBER 15, 2019

Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), caching (Memcached etc.),

Transportation 77

Transportation MySQL Kafka Data 77

Rockset

AUGUST 30, 2021

Apache Kafka has made acquiring real-time data more mainstream, but only a small sliver are turning batch analytics, run nightly, into real-time analytical dashboards with alerts and automatic anomaly detection. The majority are still draining streaming data into a data lake or a warehouse and are doing batch analytics.

SQL 52

SQL Kafka MongoDB MySQL 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? Building streaming data pipelines for large data is enticing due to the velocity of the data. What is a Big Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Stream processing engines like KSQL furthermore give you the ability to manipulate all of this fluently. The approach we’ll use works with any Kafka run though. 8, and so on.

Kafka 55

Kafka Process Algorithm Cloud 55

Pinterest Engineering

AUGUST 18, 2022

In this blog post, we will first learn the various approaches considered for data migration with their trade offs. We will then do a deep dive on how the data migration was conducted from HBase to TiDB for one of the first use cases having 4 TB table size serving 14k read qps and 400 write qps with zero downtime.

Data Ingestion 52

Data Ingestion Hadoop Database Kafka 52

Rockset

APRIL 29, 2019

In this blog post I compare options for real-time analytics on DynamoDB - Elasticsearch , Athena, and Spark - in terms of ease of setup, maintenance, query capability, latency. We can use AWS Glue to perform the ETL process and create a complete copy of the DynamoDB table in S3.

NoSQL 52

NoSQL PostgreSQL AWS SQL 52

Snowflake

MARCH 2, 2023

That proves to be a difficult task for data engineering teams that have to manage separate infrastructure for batch data and streaming data. To address this challenge, we are happy to announce the public preview of Snowpipe Streaming as the latest addition to our Snowflake ingestion offerings. How does Snowpipe Streaming work?

Kafka 117

Kafka Data Ingestion Data Pipeline Cloud Storage 117

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 55

Data Ingestion Kafka Java Data Pipeline 55

LinkedIn Engineering

JANUARY 19, 2024

As of November 2023, roughly 150K+ recruiters switched jobs in the previous 12 months as shown in Figure 1. Figure 1: Talent pool report for recruiters - LinkedIn Talent Insights During mergers and acquisitions, the source company’s user licenses and data are transferred to the acquiring company.

Recruitment 88

Recruitment Data Process Process Kafka 88

Cloudera

OCTOBER 17, 2023

Recently, Confluent hosted Current 2023 (formerly Kafka summit) in San Jose on Sept 26th and 27th. With few conferences curating content specific to streaming developers, Current has historically been an important event for anyone trying to keep a pulse on what’s happening in the streaming space. Flink is here to stay.

Database-centric 74

Database-centric Kafka Pipeline-centric Database 74

Rockset

MARCH 27, 2023

This is part two in Rockset’s Making Sense of Real-Time Analytics on Streaming Data series. In part 1 , we covered the technology landscape for real-time analytics on streaming data. In this post, we’ll explore the differences between real-time analytics databases and stream processing frameworks.

Database 52

Database Process Scala SQL 52

Cloudera

MARCH 15, 2023

In this post, I will demonstrate how to use the Cloudera Data Platform (CDP) and its streaming solutions to set up reliable data exchange in modern applications between high-scale microservices, and ensure that the internal state will stay consistent even under the highest load. Introduction Many modern application designs are event-driven.

PostgreSQL 87

PostgreSQL Kafka Database Data 87

Confluent

JULY 10, 2019

As discussed in part 2, I created a GitHub repository with Docker Compose functionality for starting a Kafka and Confluent Platform environment, as well as the code samples mentioned below. gradlew composeUp. KSQL user-defined functions. KSQL user-defined functions.

Kafka 87

Kafka Java Bytes SQL 87

Confluent

FEBRUARY 27, 2019

FaaS functions only solve the compute part, but where is data stored and managed, and how is it accessed? What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? The key to event-first systems design is understanding that a series of events captures behavior.

Kafka 109

Kafka AWS Architecture Cloud 109

Picnic Engineering

MARCH 28, 2023

Pipeline overview Since the TS is an externally developed hardware and software system, Picnic cannot change it to expose analytics events directly. So how can we get our hands on events for analytical purposes if we cannot change the TS to expose them? It detects any changes made to a database and exposes them as events.

Kafka 52

Kafka Transportation Data Warehouse Database 52

Rockset

MARCH 26, 2023

Folks have definitely tried, and while Apache Kafka® has become the standard for event-driven architectures, it still struggles to replace your everyday PostgreSQL database instance in the modern application stack. Since then, databases have been a major part of my professional life and modern, everyday life for most folks.

Cloud 52

Cloud PostgreSQL Kafka Database 52

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108

LinkedIn Engineering

JANUARY 16, 2024

In a previous blog post, we talked about how we built our anti-abuse platform using CASAL. In this blog post, we'll go deeper into how we manage account restrictions. Figure 2: Relational database schema We adopted a pragmatic and scalable approach by distributing member restrictions across different Oracle tables.

Kafka 84

Kafka Relational Database Java Architecture 84

DoorDash Engineering

APRIL 4, 2023

This is the third blog post in a series of articles showcasing our 2022 summer intern projects. The primary purpose of Archiver is to pick up events containing financial transaction data sent by different upstream teams and store this data in a data lake. The event data is stored in AWS S3 in an encoded format in JSON files.

Project 74

Project Banking Kafka Database 74

Cloudera

MAY 4, 2021

Event-driven and streaming architectures enable complex processing on market events as they happen, making them a natural fit for financial market applications. Flink SQL is a data processing language that enables rapid prototyping and development of event-driven and streaming applications. Streaming VWAP.

SQL 76

SQL Business Analyst Data Java 76

DoorDash Engineering

NOVEMBER 21, 2022

While building out DashMart’s internal inventory management system to help DashMart associates manage inventory, the DashMart engineering team came to realize that since the inventory tables were so core and foundational to different operational use cases in a DashMart, some actions or code must be triggered every time the inventory level changes.

Data Process 57

Data Process Process Kafka Database 57

Confluent

MAY 15, 2019

These systems acted as both sources and targets for Oracle WMS Cloud and were represented by a host of different APIs, including relational tables, file-based interactions, SOAP and REST. This left us with only two data integration requirements to solve: relational tables and REST APIs. Streaming data into Oracle WMS Cloud.

Kafka 91

Kafka ETL Tools Cloud Data Integration 91

ProjectPro

JANUARY 25, 2022

Table of Contents Here’s What You Need to Know About PySpark What is PySpark? Here’s What You Need to Know About PySpark This blog will take you through the basics of PySpark, the PySpark architecture, and a few popular PySpark libraries , among other things. What if you could use both these technologies together?

Big Data 52

Big Data Data Process Process Kafka 52

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. In this blog post we’ll dive into how we tackled one source of quality issues: directly relying on upstream database schemas. What are Events?

Kafka 52

Kafka ETL Tools BI Database 52

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices.

Kafka 95

Kafka Management Bytes SQL 95

Cloudera

NOVEMBER 2, 2020

Ingest 100s of TB of network event data per day . real-time customer event data alongside CRM data; network sensor data alongside marketing campaign management data). The factors driving this trend are part technical, part business, and part cultural. Billion records are updated daily in aggregate tables.

Data Warehouse 90

Data Warehouse Kafka Lambda Architecture Telecommunication 90

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Event-driven architecture. Event-driven, reactive architecture.

Architecture 90

Architecture Programming Kafka Database-centric 90

Scott Logic

APRIL 22, 2024

In my previous blog post I looked at various technologies for sending data asynchronously between services including RabbitMQ, Kafka, AWS EventBridge. I’ve coloured the data entities according to their types and we see there’s a few different patterns like events and state which we’ll discuss in a moment.

AWS 52

AWS Metadata Systems Kafka 52

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. Four pillars of event streaming. Pillar 1 – Business function: Payment processing pipeline. Pillar 4 – Operational plane: Event logging, DLQs and automation. Parting thoughts and preparing for the future. Putting it all together.

Kafka 93

Kafka Pipeline-centric Architecture Database-centric 93

Confluent

JUNE 18, 2019

It involves many moving parts, from data preparation to building indexing and query pipelines. Distributed transactions are very hard to implement successfully, which is why we’ll introduce a log-inspired system such as Apache Kafka ®. Building an indexing pipeline at scale with Kafka Connect. Direct indexing.

Architecture 96

Architecture Building Kafka Database-centric 96

Confluent

MAY 22, 2019

In light of this, we’ll share an emerging machine-to-machine (M2M) architecture pattern in which MQTT, Apache Kafka ® , and Scylla all work together to provide an end-to-end IoT solution. MQTT Proxy + Apache Kafka (no MQTT broker). On the other hand, Apache Kafka may deal with high-velocity data ingestion but not M2M.

Kafka 97

Kafka Google Cloud NoSQL Entertainment 97

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. For a system like Elasticsearch , engineers need to have in-depth knowledge of the underlying architecture in order to efficiently ingest streaming data.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

ProjectPro

NOVEMBER 22, 2021

You can think of it as a database table. Example showing the use of StructType and StructField classes in PySpark- import pyspark from pyspark.sql import SparkSession from pyspark.sql.types import StructType,StructField, StringType, IntegerType spark = SparkSession.builder.master("local[1]").appName('ProjectPro').getOrCreate()

Hadoop 52

Hadoop Python Datasets Metadata 52

Confluent

APRIL 17, 2019

As we announced in Introducing Confluent Platform 5.2 , the latest release introduces many new features that enable you to build contextual event-driven applications. In this three-part blog series, we’ll walk you through some of those new capabilities in Control Center: Part 1: Managing and Evolving Schemas with Confluent Control Center.

Management 75

Management Kafka Java Building 75

Data Engineering Weekly

DECEMBER 4, 2022

My take on this The prediction is spot on with the cost optimization, but #1 (cost optimization) & #2 (specialization) conflict. Streaming plus batch unified in a single platform. I recently shared the thought and am excited to see Meta’s blog on static analysis of SQL queries. The author gives seven predictions.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

ProjectPro

JUNE 29, 2021

Your search for Apache Kafka interview questions ends right here! Let us now dive directly into the Apache Kafka interview questions and answers and help you get started with your Big Data interview preparation! How to study for Kafka interview? What is Kafka used for? What are main APIs of Kafka?

Kafka 40

Kafka Bytes Big Data Java 40