LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Snowflake

SEPTEMBER 12, 2023

Data Cloud Industry Day 2023 is a free virtual event on September 28, 2023, dedicated to what’s possible for you and your industry in the world of data. The event will feature industry-specific sessions hosted by Snowflake’s industry leaders along with exciting customer guest speakers. Attendance at this event is totally FREE.

Cloud 86

Cloud Manufacturing Retail Healthcare 86

Cloudyard

FEBRUARY 27, 2024

This blog post showcases a novel approach combining Snowflake’s Event Logging and Dynamic Tables to automate the creation and population of dynamic tables based on Copy operations. Event-Driven Logging The journey begins with an innovative approach to log and monitor data copy operations.

Data Warehouse 52

Data Warehouse Utilities Architecture Data Management 52

Confluent

NOVEMBER 30, 2022

What is stream processing, or complex event processing (CEP), and how does it work? Learn about real-time data and event stream analytics in this tutorial.

Process 125

Process Data IT 125

Knowledge Hut

APRIL 25, 2023

In this article, we will look at how a scrum master facilitates events such as daily scrum meetings, sprint planning, sprint review, and sprint retrospective meetings. They lead the team through the Scrum process, making sure that everyone is involved and that all events are carried out successfully.

Utilities 77

Utilities Project Certification Process 77

Snowflake

OCTOBER 9, 2023

Easily collect and store digital events directly to create a complete composable customer data platform (CDP) Marketers are increasingly leveraging the Snowflake Data Cloud as the foundation for all of their customer data analytics and activation.

Cloud 89

Cloud Building SQL Data 89

Lyft Engineering

MARCH 27, 2023

However, at Lyft we have come to realize that load testing in production is a powerful tool to prepare systems for unexpected bursty traffic and peak events. Lyft must operate seamlessly even when demand spikes It’s imperative for Lyft to be highly available during peak events like Halloween and New Year’s Eve.

Coding 131

Coding Database Systems Engineering 131

Cloudyard

JULY 16, 2023

Read Time: 2 Minute, 13 Second During the last post we discussed about the Event tables and how these are used for Error logging in your process. The UDTF was responsible of parsing JSON data in a tabular format and identify any erroneous records and log into an Event table. Event tables.

Metadata 52

Metadata Python Utilities Coding 52

Cloudyard

JULY 20, 2023

Read Time: 1 Minute, 38 Second During the previous post we discussed the usage of EVENT table with Python and Javascript procedure. In continuation of the same we will explore how we can leverage the EVENT tables in SQL scripting. We will use the EVENT tables to log the error and other details through SQL scripting procedure.

SQL 52

SQL Python Systems Designing 52

Confluent

FEBRUARY 8, 2024

Luckily, taking an event-driven approach can make the process more manageable. Struggling to get your GenAI app off the ground? You’re not alone—GenAI apps can be challenging to build. Learn how.

Building 52

Building Process Management 52

Confluent

SEPTEMBER 9, 2021

I’ve always found event sourcing to be fascinating. We spend so much of our lives as developers saving data in database tables—doing this in a completely different way seems almost […].

Database 91

Database Data Process 91

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. So our solution was to start using an intentional contract: Events. What are Events? Events are facts about what happened within your service.

Kafka 52

Kafka ETL Tools BI Database 52

Workfall

SEPTEMBER 26, 2023

Reading Time: 9 minutes In this blog, we will cover: What are Server-Sent Events? Why Stream Data Using Server-Sent Events (SSE)? Hands-On Conclusion What are Server-Sent Events? These events can carry information like notifications, updates, or alerts. What is FastAPI?

Python 85

Python Bytes Coding Data 85

Confluent

SEPTEMBER 21, 2022

A deep dive into how microservices work, why it’s the backbone of real-time applications, and how to build event-driven microservices applications with Python and Kafka.

Kafka 98

Kafka Python Building 98

Start Data Engineering

APRIL 4, 2020

One of the most common issues when ingesting and processing user generated events is, how to deal with late arriving events. Some of the general issues that data engineers usually have are “What should be considered the event time?” Yet this topic is not extensively discussed.

Data Engineering 130

Data Engineering Data Engineer Engineering Process 130

KDnuggets

JUNE 26, 2023

Have you ever wondered how fortune tellers, astrologers, or our well-known Baba Vanga used to predict future events? Or have you ever questioned whether AI and ML have the capabilities to predict future events as Baba Vanga did? For suppose if AI and ML have the capabilities, then up to how extent can it predict?

IT 82

IT 82

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. By exploring the benefits of using Data Mesh for your event streaming architecture, this article will help you decide if it’s the right solution for your organization.

Architecture 52

Architecture Generalist Government Datasets 52

Workfall

JUNE 27, 2023

In this blog, we’ll explore how to harness the power of Kafka to streamline event streaming within a microservices architecture and unlock its full potential for building scalable and responsive systems. In the event of server failure, Kafka can autonomously restart itself, ensuring uninterrupted data flow. Let’s get started!

Kafka 75

Kafka Architecture AWS Transportation 75

Confluent

JULY 8, 2020

In the article Should You Put Several Event Types in the Same Kafka Topic?, Martin Kleppmann discusses when to combine several event types in the same topic and introduces new […].

Kafka 138

Kafka 138

Data Engineering Podcast

AUGUST 10, 2020

Summary Event based data is a rich source of information for analytics, unless none of the event structures are consistent. The team at Iteratively are building a platform to manage the end to end flow of collaboration around what events are needed, how to structure the attributes, and how they are captured.

Data Collection 100

Data Collection Data Engineering Data Engineer Education 100

Confluent

NOVEMBER 25, 2020

The event provided an opportunity for participants to learn, build, and showcase […]. As part of this year’s Kafka Summit, the Confluent Community team hosted a community hackathon named Kafkathon 2020.

Kafka 104

Kafka Building 104

Cloudyard

JULY 11, 2023

Read Time: 4 Minute, 32 Second In this post, we will explore the utilization of EVENT tables to capture erroneous data. To achieve this, a Python process is employed to process the input feed file, identifying and excluding the flawed records, while simultaneously logging them in an event table. However, Import the below two module.

Python 40

Python Utilities Datasets Data Pipeline 40

Confluent

SEPTEMBER 11, 2023

See how you can leverage a data streaming platform to build event-driven microservices and do stream processing for faster, better customer onboarding.

Cloud 67

Cloud Building Process Data 67

Confluent

FEBRUARY 2, 2021

Since I first started using Apache Kafka® eight years ago, I went from being a student who had just heard about event streaming to contributing to the transformational, company-wide event […].

Kafka 96

Kafka Data Schemas Cloud Data 96

Uber Engineering

SEPTEMBER 23, 2021

This article focuses on how we … The post Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot appeared first on Uber Engineering Blog. With this new ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more.

Kafka 145

Kafka Process Systems Engineering 145

Confluent

OCTOBER 19, 2022

Designing Events and Event Streams: Introduction and Best Practices.

Designing 52

Designing Kafka 52

Waitingforcode

JULY 25, 2023

You certainly know it, the watermark (aka GC Watermark) is responsible for cleaning state store in Apache Spark Structured Streaming. But you may not know that it's not the single time-based condition. There is a different one involved in the stream-to-stream joins.

IT 130

IT 130

Confluent

JANUARY 22, 2021

We launched a transformation initiative three years ago that transitioned SEI Investments from a monolithic database-oriented architecture to a containerized services platform with an event-driven architecture based on Confluent Platform. […].

Architecture 98

Architecture Database Data Schemas Data Governance 98

KDnuggets

FEBRUARY 5, 2024

If you are new to generative AI or an expert who wants to learn more, O’Reilly offers a range of resources to kickstart your generative AI journey.

112

112

Confluent

SEPTEMBER 19, 2023

Discover the top 5 best practices for building event-driven architectures using Confluent and AWS Lambda. Learn how to optimize your architecture for scalability, reliability, and performance.

AWS 67

AWS Architecture Building 67

KDnuggets

NOVEMBER 30, 2021

Join AI experts from Microsoft and BCG at Put Responsible AI into Practice—a free Azure digital event on December 7. Learn best practice guidelines for building AI solutions responsibly.

Building 105

Building 105

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Confluent

MARCH 9, 2023

Event Streaming is one of many investments that the technology leaders can make. Here’s an assessment of how event streaming benefits your enterprise.

Technology 57

Technology 57

Confluent

AUGUST 25, 2022

G2 named Confluent the the event stream processing industry leader for top-rated performance, reliability, ease of use, integration APIs, data modeling features, and more.

Process 62

Process Data 62

Confluent

MAY 12, 2023

With Confluent Cloud, Loggi migrated to an event-driven architecture, powering real-time analytics, boosting productivity, and cutting costs.

Architecture 57

Architecture Designing Engineering Cloud 57

Confluent

MARCH 19, 2019

When should you use an event? I will attempt to articulate in layman’s terms what an event-driven architecture (EDA) is and contrast it with service-oriented architecture (SOA). In essence, this is an attempt to differentiate and/or associate APIs with events. Is not life simply a series of events? If it can, should it?

Architecture 110

Architecture Telecommunication Transportation Database 110