Rockset

SEPTEMBER 2, 2022

I recently had the good fortune to host a small-group discussion on personalization and recommendation systems with two technical experts with years of experience at FAANG and other web-scale companies. Nikhil Garg is CEO and co-founder of Fennel AI , a startup working on building the future of real-time machine learning infrastructure.

Systems 52

Systems Building Machine Learning NoSQL 52

Yelp Engineering

MARCH 7, 2024

This blog post covers how we leverage Yelp’s extensive streaming infrastructure to build robust data abstractions for our offline and streaming data consumers. Key terminology Let’s start by covering certain key terms used throughout the post: Offline systems - data warehousing platforms such as AWS Redshift or.

Building 59

Building AWS Data Pipeline Data 59

Lyft Engineering

APRIL 24, 2024

See our previous blog post for more on the motivations behind TLC. The Technical Learning workstream offers Lyft scientists the opportunity to build or brush up their core data science skills across multiple areas. Some recent examples can be found in our tech blog: reinforcement learning and structural causal modeling.

Data Science 66

Data Science Building Computer Science Software Engineer 66

Knowledge Hut

JANUARY 18, 2024

CISSP stands for Certified Information Systems Security Professional, and it is a certification in cyber security. This professional certification is developed and offered by (ISC)2, also known as International Information Systems Security Certification Consortium. This blog will help you understand the question “what is CISSP?

Systems 52

Systems Certification Consulting Education 52

Cloudera

DECEMBER 1, 2023

It focuses on five key pillars: investing in research and development; unleashing government AI resources; setting standards and policy; building the AI workforce; and advancing trust and security. AI systems must operate within a framework that promotes ethical practices, transparency, and accountability. million), among others.

Building 104

Building Government Transportation Data Governance 104

Cloudera

JUNE 2, 2022

Over the last two years, the Cloudera DataFlow team has been hard at work building Cloudera DataFlow for the Public Cloud (CDF-PC). This blog aims to answer two questions: What is a universal data distribution service? How to onboard data into their system? I don’t care about their system. What is the modern data stack?

Systems 98

Systems Data Lake Google Cloud Data Collection 98

Uber Engineering

SEPTEMBER 29, 2021

a driver starting a trip) and system actions … The post Building Uber’s Fulfillment Platform for Planet-Scale using Google Cloud Spanner appeared first on Uber Engineering Blog. The platform handles billions of database transactions each day, ranging from user actions (e.g.,

Google Cloud 143

Google Cloud Cloud Building Database 143

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

DataKitchen

DECEMBER 9, 2022

This can include the use of tools for data preparation, model training, and deployment, as well as technologies for monitoring and managing data-related systems and processes. This can help organizations to build trust in their data-related workflows, and to drive better outcomes from their data analytics and machine learning initiatives.

Machine Learning 52

Machine Learning Data Preparation Government Data Analytics 52

Pinterest Engineering

SEPTEMBER 20, 2023

Our commitment is evidenced by our history of building products that champion inclusivity. We know from experience that building for marginalized communities helps make the product work better for everyone. To ensure an unbiased approach, we also leveraged our skin tone and hair pattern signals when building this dataset.

Building 84

Building Pipeline-centric Machine Learning Datasets 84

Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. So which static site generator to choose?

Engineering 40

Engineering Bytes AWS Python 40

Snowflake

DECEMBER 11, 2023

While digital advertisers and paid media professionals are on the hook to build ample sales pipeline and maximize return on ad spend (ROAS), they’re also expected to deliver personalized advertising content while navigating evolving privacy requirements and adhering to consumer expectations—all while extracting insights from siloed ad platforms.

Building 88

Building Cloud Media Algorithm 88

databricks

AUGUST 23, 2023

To build customer support bots, internal knowledge graphs, or Q&A systems, customers often use Retrieval Augmented Generation (RAG) applications which leverage pre-trained models.

Building 97

Building Systems 97

Data Engineering Podcast

NOVEMBER 26, 2019

In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse. What did the previous system look like? What was your design criteria for building a new platform?

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Databand.ai

AUGUST 30, 2023

Building a Winning Data Quality Strategy: Step by Step Eitan Chazbani August 30, 2023 What Is a Data Quality Strategy? This starts with building a strong business case for your data quality strategy. Some common data governance tools include data cataloging solutions, data quality monitoring tools, and data access control systems.

Building 52

Building Data Cleanse Data Governance Datasets 52

DoorDash Engineering

JANUARY 24, 2023

Instead, we pursued a complete redesign to upgrade our system, building a no-code platform and migrating the service to Kotlin, our new backend framework. We also discuss how we migrated the system to Kotlin. A key drawback: We would have a hybrid state with both the legacy system and the new system involved.

Building 72

Building Python Coding Food 72

Data Engineering Podcast

APRIL 19, 2020

Summary Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor. This was an interesting inside look at building a business on top of open source stream processing frameworks and how to reduce the burden on end users.

Building 100

Building PostgreSQL MongoDB SQL 100

Netflix Tech

OCTOBER 19, 2020

which is difficult when troubleshooting distributed systems. In our previous blog post we introduced Edgar, our troubleshooting tool for streaming sessions. This insight led us to build Edgar: a distributed tracing infrastructure and user experience. The following sections describe our journey in building these components.

Building 93

Building Transportation Metadata Java 93

Data Engineering Podcast

AUGUST 26, 2019

Summary Data engineers are responsible for building tools and platforms to power the workflows of other members of the business. What are some of the common shortcomings that you have found in out-of-the-box tools that organizations use to build their data stack?

Building 100

Building Data Analytics Media Data Engineering 100

Data Engineering Weekly

APRIL 21, 2023

Ananth and Aswin discussed a blog from BuzzFeed that shares lessons learned from building products powered by generative AI. The blog highlights how generative AI can be integrated into a company's work culture and workflow to enhance creativity rather than replace jobs.

Building 52

Building Data Architect Education Data Engineering 52

Zalando Engineering

MARCH 1, 2021

In order to provide an excellent customer experience, we must ensure our systems can technically handle high traffic events such as Cyber Week or other sales campaigns. Checkout and payments related systems are particularly important during sales events. We have published a detailed article on how Zalando prepares for the Cyberweek.

Systems 40

Systems Building Java Algorithm 40

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. Building a data lake infrastructure is a highly beneficial endeavor, not just for the immediate use of data, but also for its potential to foster the growth of analytics across the organization.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Uber Engineering

AUGUST 7, 2019

A common axiom among Uber engineers states that building new features is like fixing a car’s engine while driving it.

Systems 79

Systems Engineering Building IT 79

Cloudera

JANUARY 6, 2021

Python is used extensively among Data Engineers and Data Scientists to solve all sorts of problems from ETL/ELT pipelines to building machine learning models. Apache HBase is an effective data storage system for many workflows but accessing this data specifically through Python can be a struggle. Introduction. Example Operations .

Machine Learning 131

Machine Learning Data Science Database Building 131

Data Engineering Podcast

DECEMBER 22, 2019

To address that need more completely the team at Materialize has created an engine that allows for building queryable views of your data as it is continually updated from the stream of changes being generated by your applications. Between those use cases there are varying levels of support for fast reads on quickly changing data.

SQL 100

SQL Building Engineering Java 100

Engineering at Meta

OCTOBER 23, 2023

To optimize build and performance, we developed our own build system called Buck , which was first open-sourced in 2013. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Meta has a very large monorepo, with many different programming languages.

Programming Language 109

Programming Language Java Python Programming 109

Knowledge Hut

JUNE 20, 2023

However, to succeed in e-commerce, businesses require certain skills and tools to build and run an effective online storefront. Hence, it is recommended that you build your career with professional courses that help you acquire the necessary skills to thrive in the world of e-commerce.

Building 52

Building Media Data Analysis Certification 52

Yelp Engineering

MARCH 25, 2024

In this blog post, we highlight how phone number masking helps build consumer trust in the services marketplace at Yelp, decreases the friction in communication with service professionals, and allows for seamless switching between the Yelp app and a user’s phone.

Project 103

Project Consulting Engineering Systems 103

Data Engineering Podcast

MAY 19, 2019

Summary In recent years the traditional approach to building data warehouses has shifted from transforming records before loading, to transforming them afterwards. The data build tool (dbt) is designed to bring battle tested engineering practices to your analytics pipelines. Can you talk through how the packaging system is implemented?

Data Analytics 100

Data Analytics Building Engineering Data Warehouse 100

LinkedIn Engineering

MARCH 21, 2023

One of the most exciting parts of our work is that we get to play a part in helping progress a skills-first labor market through our team’s ongoing engineering work in building our Skills Graph. Engineering vs PyTorch Figure 6: Sample Seed Skills Graph KGBert helps build a more accurate and complex taxonomy in less time.

Building 85

Building Recruitment Machine Learning Deep Learning 85

Data Engineering Weekly

MAY 3, 2024

There is a lot of context missing in that tweet, so I decided to write a blog about it. However, modern Notebooks like Databricks seamlessly integrate with Git to build pull requests and code review processes. There is no underlying semantics for unit testing the code and data testing build-in.

Programming Language 117

Programming Language ETL Tools Data Pipeline Coding 117

Pinterest Engineering

OCTOBER 26, 2022

This means that during the retrieval stage, we use a batch of query pins to call our retrieval system to generate pin recommendations. Therefore, we can build a feedback loop to reward the queries with downstream engagement. In the future iterations, we will consider more engagement types rather than repin to build a user profile.

Building 52

Building Software Engineer Software Engineering Engineering 52

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 135

Utilities Systems Architecture Coding 135

Monte Carlo

JUNE 22, 2023

Building the platform as composable architecture Their next step was to build the product…that would be used to build data products. The platform team started thinking through which services domains would need to build, maintain, and deliver high-quality, FAIR data products to their end-users.

Building 52

Building Data Ingestion Data Business Analyst 52

Data Engineering Weekly

NOVEMBER 11, 2022

Not too long ago, I worked for a famous messaging platform; You may be typing on it as you read this blog. I took a break from data engineering and focused on system monitoring and observability for some time. Little did I know I would be building data pipelines to measure system efficiency instead of measuring business efficiency.

Systems 40

Systems High Quality Data Consulting Metadata 40

Cloudera

JANUARY 20, 2021

For more context, this demo is based on concepts discussed in this blog post How to deploy ML models to production. With this example as inspiration, I decided to build off of sensor data and serve results from a model in real-time. As a result, I decided to use an open-source Occupancy Detection Data Set to build this application.

Machine Learning 111

Machine Learning Database Data Science Building 111

Uber Engineering

FEBRUARY 16, 2018

In distributed systems, retries are inevitable. Given the scope … The post Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka appeared first on Uber Engineering Blog.

Kafka 109

Kafka Building Engineering Systems 109