Lyft Engineering

APRIL 25, 2023

Building a large scale unsupervised model anomaly detection system — Part 2 Building ML Models with Observability at Scale By Rajeev Prabhakar , Han Wang , Anindya Saha Photo by Octavian Rosca on Unsplash In our previous blog we discussed the different challenges we faced for model monitoring and our strategy for addressing some of these problems.

Systems 75

Systems Building Machine Learning Datasets 75

Netflix Tech

FEBRUARY 18, 2022

To this end, we developed a Rapid Event Notification System (RENO) to support use cases that require server initiated communication with devices in a scalable and extensible manner. In this blog post, we will give an overview of the Rapid Event Notification System at Netflix and share some of the learnings we gained along the way.

Systems 133

Systems Architecture Portfolio Designing 133

Cloudera

DECEMBER 9, 2022

Now, we shift focus on the needs of developers and addressing the challenges they face when building dataflows in the cloud. We’ve observed organizations using more and more data sources and destinations , as well as expecting a more diverse range of developers to build data movement flows. Enabling self-service for developers.

Designing 96

Designing Coding Google Cloud AWS 96

Lyft Engineering

APRIL 3, 2023

This blog post focuses on the scope and the goals of the recommendation system, and explores some of the most recent changes the Rider team has made to better serve Lyft’s riders. Introduction: Scope of the Recommendation System The recommendation system covers user experiences throughout the ride journey.

Systems 87

Systems Pipeline-centric Machine Learning Transportation 87

Robinhood

NOVEMBER 28, 2023

Together, we are building products and services that help create a financial system everyone can participate in. … At Robinhood, we’re constantly striving to enhance our products and processes, and the journey to integrate our Content Standards into our design system was no exception. Hey, I’m Jessi! Hey, I’m Jessi!

Designing 71

Designing Finance Certification Systems 71

Knowledge Hut

MARCH 28, 2024

You all must be aware of the fact that user experience (UX) is the cornerstone of successful design. In this blog, we’ll simply make it easier for you to understand the core ideas that influence UX design. Come, let’s explore these 12 UI/UX design concepts and equip you to produce long-lasting designs.

Designing 52

Designing Entertainment Certification Accessible 52

Pinterest Engineering

JANUARY 10, 2023

Liang Ma | Software Engineer, Core Eng; Wei Zhu | Software Engineer, Observability In early 2020, during a critical iOS out of memory incident (we have a blogpost for that), we realized that we didn’t have much visibility of how the app is running or a good system to look up for monitoring and troubleshooting.

Systems 54

Systems Building Software Engineer Software Engineering 54

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Knowledge Hut

MARCH 21, 2024

Most people don't pay much attention to an app's or software's UI design until something goes wrong. Because of this, we might argue that effective user interface design is essentially invisible, allowing users to work uninterrupted. Want to enhance your knowledge in UI design or pick up new skills?

Designing 52

Designing Accessible Accessibility Project 52

Knowledge Hut

MARCH 20, 2024

As we head into 2024, let's take a moment to explore the heart and soul of s oftware development - Design Principles. In this blog, we'll talk about some of the coolest software design principles you need to know in 2024. What are Software Design Principles? Why are Software Design Principles Important?

Designing 52

Designing Software Engineer Software Engineering Coding 52

LinkedIn Engineering

DECEMBER 20, 2023

At LinkedIn, trust is the cornerstone for building meaningful connections and professional relationships. In this blog post, we discuss how we are harnessing AI to help us with abuse prevention and share an overview of our infrastructure and the role it plays in identifying and mitigating abusive behavior on our platform.

Building 91

Building Algorithm Kafka Machine Learning 91

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 363

Education Project PostgreSQL Software Engineer 363

Knowledge Hut

MARCH 29, 2024

In this rapidly growing digital world, I've realized that ethical considerations in design frameworks have never been more critical than before. As a designer, I exert significant influence over how people interact with technology and data, making the imperative for ethical design important to me. What is Ethical Design?

Designing 52

Designing Software Engineer Software Engineering Algorithm 52

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Knowledge Hut

JANUARY 31, 2024

It can be difficult to design IT services that meet business needs and customer expectations. The service design module is one of the certifications within ITIL v3 Service Lifecycle workstream. ITIL service design provides a holistic design approach to aid companies deliver better services. What is ITIL Service Design?

Designing 52

Designing Process IT Portfolio 52

Cloudera

MARCH 14, 2023

We just announced the general availability of Cloudera DataFlow Designer , bringing self-service data flow development to all CDP Public Cloud customers. In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities.

Data Pipeline 81

Data Pipeline Designing Kafka Metadata 81

Pinterest Engineering

JANUARY 4, 2024

In 2020, anticipating the growing needs of the business and to simplify our storage offerings, we decided to consolidate our different key-value systems in the company into a single unified service called KVStore. RocksDB is designed to provide high performance, scalability, and efficient storage for various workloads.

Database 77

Database Building Datasets Relational Database 77

Netflix Tech

MARCH 14, 2023

We build creator tooling to enable these colleagues to focus their time and energy on creativity. We implemented a batch processing system for users to submit their requests and wait for the system to generate the output. This limited pilot system greatly reduced the time spent by our users to manually analyze the content.

Media 116

Media Building Algorithm Machine Learning 116

DoorDash Engineering

FEBRUARY 22, 2023

This article outlines the challenges we faced while building the inventory platform and how we solved them after multiple iterations of experimentation and analysis. Additionally, the Fulfillment engineering team recently released this article about how they use the CockroachDB to scale their system.

Designing 125

Designing Database Architecture SQL 125

Cloudera

JUNE 2, 2022

Over the last two years, the Cloudera DataFlow team has been hard at work building Cloudera DataFlow for the Public Cloud (CDF-PC). This blog aims to answer two questions: What is a universal data distribution service? How to onboard data into their system? I don’t care about their system. What is the modern data stack?

Systems 99

Systems Data Lake Google Cloud Data Collection 99

Knowledge Hut

DECEMBER 22, 2023

In our tech-driven world, UI and UX design elevate everyday activities, turning them into extraordinary and enjoyable experiences. As a UI/UX designer, you may embrace infinite growth, improve designs based on user input, keep up with trends and technology, and join the ranks of powerful designers. Still not sure?

Designing 52

Designing Food Certification Portfolio 52

Knowledge Hut

JANUARY 3, 2024

A major computer system component is its operating system (OS). A computer would be a little more than a useless computer without an operating system. And at least one operating system must be installed on your computer to run simple programs like browsers. What is Operating System (OS)?

Systems 52

Systems Programming Python Accessible 52

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. How have the design and goals of the product changed since you started it? Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

DataKitchen

DECEMBER 9, 2022

This can include the use of tools for data preparation, model training, and deployment, as well as technologies for monitoring and managing data-related systems and processes. This can help organizations to build trust in their data-related workflows, and to drive better outcomes from their data analytics and machine learning initiatives.

Machine Learning 52

Machine Learning Data Preparation Government Data Analytics 52

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. The individual/automation must retrieve the same after authenticating to the DataVault token service.��

Systems 55

Systems Metadata Datasets Designing 55

Pinterest Engineering

SEPTEMBER 20, 2023

These teams work together to ensure algorithmic fairness, inclusive design, and representation are an integral part of our platform and product experience. Our commitment is evidenced by our history of building products that champion inclusivity. “Everyone” has been the north star for our Inclusive AI and Inclusive Product teams.

Building 85

Building Pipeline-centric Machine Learning Datasets 85

Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. Building a real-time, contextual and trustworthy knowledge base for AI applications revolves around RAG pipelines.

Cloud 64

Cloud Building Metadata Kafka 64

Confluent

JUNE 26, 2019

Microservices have a symbiotic relationship with domain-driven design (DDD)—a design approach where the business domain is carefully modeled in software and evolved over time, independently of the plumbing that makes the system work. What technology should you use to build microservice architectures? Martin Fowler.

Kafka 109

Kafka Designing Architecture ETL Tools 109

Edureka

JUNE 27, 2023

That, my friend, is the art of UI/UX design—the magic that turns ordinary websites and applications into extraordinary works of art. UI/UX design is the perfect combination of creativity and functionality. In this enchanting blog, we will go through the concepts of UI/UX design.

Designing 52

Designing Portfolio Architecture Certification 52

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Your first 30 days are free!

Database 162

Database Technology Data Lake High Quality Data 162

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Workfall

FEBRUARY 7, 2023

With the introduction of WebAssembly, it became possible to build frontend web apps in Rust , such as the one we just built, expanding development opportunities for developers. As we know, JavaScript has never been the ideal language for creating CPU-intensive applications owing to its single-threaded design.

Building 87

Building PostgreSQL Coding AWS 87

Data Engineering Podcast

APRIL 19, 2020

Summary Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor. Eventador is a managed platform designed to let you focus on using the data that you collect, without worrying about how to make it reliable.

Building 100

Building PostgreSQL MongoDB SQL 100

The Pragmatic Engineer

SEPTEMBER 19, 2023

An SDET has the ability to analyze the functionality and architecture of a Product and thus design and implement tools that help testers test it. The SDET enjoys short project life cycles, designs and implements many tools and test frameworks in a single year, uses the latest technology and has plenty of room for innovation.

Software Engineer 283

Software Engineer Software Engineering Coding Engineering 283

Rockset

DECEMBER 19, 2023

From his early days at Quora to leading projects at Facebook and his current venture at Fennel (a real-time feature store for ML), Nikhil has traversed the evolving landscape of machine learning engineering and machine learning infrastructure specifically in the context of recommendation systems.

Systems 52

Systems Machine Learning Deep Learning Media 52

Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. It's actively developed.

Engineering 40

Engineering Bytes AWS Python 40

Pinterest Engineering

MAY 13, 2024

The subsequent blog post will delve into how we looked into our specific needs, evaluated multiple candidates and decided on the adoption of a new database technology. Yet the HBase version upgrade is a slow and painful process due to a legacy build/deploy/provisioning pipeline and compatibility issues (the last upgrade from 0.94

NoSQL 69

NoSQL MySQL Database Systems 69