Lyft Engineering

MARCH 27, 2023

However, at Lyft we have come to realize that load testing in production is a powerful tool to prepare systems for unexpected bursty traffic and peak events. In the context of this article we mean any tool that creates traffic to stress test systems and see how they perform at the limits of their capacity.

Coding 131

Coding Database Systems Engineering 131

Confluent

MAY 13, 2021

Failures are inevitable in any system, and there are various options for mitigating them automatically. This is made possible by event-driven applications leveraging Apache Kafka® and built with fault tolerance […].

Kafka 95

Kafka Systems 95

Knowledge Hut

JANUARY 3, 2024

A system call is a function that a user program uses to ask the operating system for a particular service. User programmers can communicate with the operating system to request its services using the interface that is created by a system call. System calls serve as the interface between an operating system and a process.

Systems 52

Systems Programming Accessible Accessibility 52

Cloudyard

JULY 20, 2023

Read Time: 1 Minute, 38 Second During the previous post we discussed the usage of EVENT table with Python and Javascript procedure. In continuation of the same we will explore how we can leverage the EVENT tables in SQL scripting. We will use the EVENT tables to log the error and other details through SQL scripting procedure.

SQL 52

SQL Python Systems Designing 52

Pinterest Engineering

JANUARY 18, 2024

In particular, our machine learning powered ads ranking systems are trying to understand users’ engagement and conversion intent and promote the right ads to the right user at the right time. Specifically, such discrepancies unfold into the following scenarios: Bug-free scenario : Our ads ranking system is working bug-free.

Systems 78

Systems Machine Learning Data Mining Algorithm 78

Engineering at Meta

AUGUST 9, 2023

Explore is one of the largest recommendation systems on Instagram. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and flexible. locally popular media), which further contributes to system scalability.

Systems 85

Systems Media Algorithm Machine Learning 85

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. By exploring the benefits of using Data Mesh for your event streaming architecture, this article will help you decide if it’s the right solution for your organization.

Architecture 52

Architecture Generalist Government Datasets 52

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. This rule empowers microservices to change their database schemas however they like without worrying about breaking other systems. What are Events?

Kafka 52

Kafka ETL Tools BI Database 52

Workfall

JUNE 27, 2023

In this blog, we’ll explore how to harness the power of Kafka to streamline event streaming within a microservices architecture and unlock its full potential for building scalable and responsive systems. In the event of server failure, Kafka can autonomously restart itself, ensuring uninterrupted data flow. Let’s get started!

Kafka 75

Kafka Architecture AWS Transportation 75

Uber Engineering

SEPTEMBER 23, 2021

With this new ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more. This article focuses on how we … The post Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot appeared first on Uber Engineering Blog.

Kafka 145

Kafka Process Systems Engineering 145

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. Can you describe what you are building at Opaque Systems and the story behind it? Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Engineering at Meta

SEPTEMBER 7, 2023

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. We need a systemized source of truth that can simulate various performance factors across compute, storage, and network collectively. For instance, the AI Research SuperCluster for AI research.

Systems 101

Systems Architecture Utilities Designing 101

Confluent

FEBRUARY 24, 2022

A few years ago I helped build an event-driven system for gym bookings. The pitch was that we were building a better experience for both the gym members booking different […].

Systems 122

Systems Building Data 122

Data Engineering Podcast

FEBRUARY 20, 2022

He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers. Can you describe what Immunai is and the story behind it?

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Knowledge Hut

JANUARY 18, 2024

CISSP stands for Certified Information Systems Security Professional, and it is a certification in cyber security. This professional certification is developed and offered by (ISC)2, also known as International Information Systems Security Certification Consortium. are interconnected and can be a part of your job description.

Systems 52

Systems Certification Consulting Education 52

Data Engineering Podcast

APRIL 24, 2022

WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Machine Learning 100

Machine Learning Systems Data Lake Metadata 100

Data Engineering Podcast

JULY 6, 2020

Summary There are an increasing number of use cases for real time data, and the systems to power them are becoming more mature. For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home.

Systems 100

Systems Kafka SQL Government 100

Confluent

DECEMBER 2, 2022

With over 4,700 stores, learn how Walmart used Kafka to build an event-driven architecture for real-time inventory management, providing a seamless omnichannel experience.

Kafka 117

Kafka Systems Architecture Building 117

Rockset

DECEMBER 19, 2023

From his early days at Quora to leading projects at Facebook and his current venture at Fennel (a real-time feature store for ML), Nikhil has traversed the evolving landscape of machine learning engineering and machine learning infrastructure specifically in the context of recommendation systems.

Systems 52

Systems Machine Learning Deep Learning Media 52

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. What is an OS snapshot?

Systems 55

Systems Metadata Datasets Designing 55

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

U-Next

FEBRUARY 16, 2023

As my thoughts started wandering around our Banking systems and Cosmos Bank Cyber-attack 2018. Also, the recovery also gets affected as there is a lag of almost 24 months between fraud and detection. A robust fraud detection and monitoring system is required. The system should time and again monitor and report audit authorities.

Banking 74

Banking Systems Education Government 74

Knowledge Hut

DECEMBER 5, 2023

If you have a passion for information technology (IT) and dream of turning it into a fulfilling career, ponder the path of a systems engineer. Join us on a detailed exploration of who can pursue a career as a systems engineer and the steps to become one in the year 2024. Who is a System Engineer, and What Do They Do?

Systems 52

Systems Engineering Recruitment Electronics 52

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 85

Systems Metadata Media Kafka 85

Data Engineering Podcast

JULY 27, 2020

Summary A majority of the scalable data processing platforms that we rely on are built as distributed systems. Kyle Kingsbury created the Jepsen framework for testing the guarantees of distributed data processing systems and identifying when and why they break. This brings with it a vast number of subtle ways that errors can creep in.

Systems 100

Systems Building Scala Java 100

Confluent

MARCH 19, 2019

When should you use an event? I will attempt to articulate in layman’s terms what an event-driven architecture (EDA) is and contrast it with service-oriented architecture (SOA). In essence, this is an attempt to differentiate and/or associate APIs with events. Is not life simply a series of events? If it can, should it?

Architecture 110

Architecture Telecommunication Transportation Database 110

Data Engineering Podcast

JULY 20, 2020

Summary Wind energy is an important component of an ecologically friendly power system, but there are a number of variables that can affect the overall efficiency of the turbines. Michael Tegtmeier founded Turbit Systems to help operators of wind farms identify and correct problems that contribute to suboptimal power outputs.

Systems 100

Systems Manufacturing Machine Learning Algorithm 100

Data Engineering Podcast

FEBRUARY 1, 2021

As the systems that we build continue to grow in scale and complexity the need for reliable and manageable monitoring platforms increases proportionately. In this episode Rob Skillington, CTO of Chronosphere, shares his experiences building metrics systems that provide observability to companies that are operating at extreme scale.

Systems 100

Systems Cloud BI Data Warehouse 100

databricks

SEPTEMBER 12, 2023

As the lakehouse becomes increasingly mission-critical to data-forward organizations, so too grows the risk that unexpected events, outages, and security incidents may derail.

Systems 79

Systems Data 79

Cloudera

JUNE 2, 2022

In a recent customer workshop with a large retail data science media company, one of the attendees, an engineering leader, made the following observation: “Everytime I go to your competitor website, they only care about their system. How to onboard data into their system? I don’t care about their system.

Systems 99

Systems Data Lake Google Cloud Data Collection 99

Knowledge Hut

OCTOBER 11, 2023

System engineering is a multidisciplinary field that ensures that a system achieves its intended goals effectively and efficiently taking numerous technical, organizational, and operational elements into account. In this post I will walk you through the role of a system engineer. What Does a System Engineer Do?

Systems 52

Systems Engineering Cloud Computing Programming Language 52

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. Still, I’m puzzled by how long the system has been down. If it was an update to Oracle, or to the operating system, then why not roll back the update?

Banking 199

Banking Utilities Database Engineering 199

Data Engineering Weekly

MAY 27, 2023

[link] Expedia: How Expedia Reviews Engineering Is Using Event Streams as a Source Of Truth “Events as a source of truth” is a simple but powerful idea to persist the state of the business entity as a sequence of state-changing events. How to build such a system?

Data Validation 52

Data Validation Data Data Engineering Data Engineer 52

Data Engineering Podcast

FEBRUARY 18, 2024

Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? Don't miss out on our only event this year!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Confluent

MARCH 29, 2021

Self-managing a highly scalable distributed system with Apache Kafka® at its core is not an easy feat. That’s why operators prefer tooling such as Confluent Control Center for administering and […].

Kafka 130

Kafka Systems Management IT 130

Data Engineering Podcast

JULY 5, 2021

SaaSGlue is a managed service that lets you connect all of your systems, across clouds and physical infrastructure, and spanning all of your programming languages. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

Systems 100

Systems Management Data Warehouse Programming Language 100

Rockset

OCTOBER 4, 2022

Event streams, such as clickstreams, IoT data and other time series data, are common sources of data into these apps. The broad adoption of Apache Kafka has helped make these event streams more accessible. In this post, we compare two options for real-time analytics on event and CDC streams: Rockset and ClickHouse.

MySQL 52

MySQL Kafka Aggregated Data Architecture 52