Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Marc Lamberti

SEPTEMBER 22, 2023

In this article, you will learn everything about what Airflow is, what it isn’t, and its core concepts and components. Stemming from this analogy, “you” is the orchestrator in data orchestration, and the recipe is the data pipeline. When you create a data pipeline, you create a DAG.

Data Pipeline 130

Data Pipeline Python Metadata Database 130

Snowflake

DECEMBER 5, 2023

Bring gen AI to your governed enterprise data with Snowflake Cortex No matter your industry, department or role, you can leverage generative AI (gen AI) and large language models (LLMs) to increase efficiencies and uncover new solutions to business challenges.

AWS 96

AWS Amazon Web Services Government Cloud Computing 96

Data Engineering Podcast

DECEMBER 4, 2022

Summary The term "real-time data" brings with it a combination of excitement, uncertainty, and skepticism. In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

OCTOBER 16, 2022

Pathway is a streaming database engine that embeds artificial intelligence into the storage, with functionality designed to support the spatiotemporal data that is crucial for shipping and logistics. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Database 100

Database Metadata MongoDB MySQL 100

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 90

Data Process Process Metadata Finance 90

Christophe Blefari

JUNE 3, 2023

I wanted to write more about Microsoft Fabric and the states of data that were published last week but I'll do it another time. The article is saying that it lays down with Japanese new strategy to become a leader in AI technologies, by removing barriers on training data they hope to open doors. Or several needles.

Data Pipeline 130

Data Pipeline Data SQL Algorithm 130

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. Therefore, the greatest thing you can do as a novice is to work on some real-time data engineering initiatives.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Snowflake

DECEMBER 21, 2023

In this edition, Patch.tech Co-Founder and CPO Whelan Boyd talks about how frustration with clogged data pipelines sparked the idea for Patch’s code packages, which allow engineers to distribute data sets with all the built-in elements that analysts and developers need to create apps. What inspired you to start Patch?

Software Engineer 85

Software Engineer Software Engineering Database Data Pipeline 85

Lyft Engineering

NOVEMBER 29, 2023

Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data. For our customers, this means faster analytics on near real-time data and decision making. An example of how we use Druid rollup at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Data Engineering Weekly

FEBRUARY 4, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Visit rudderstack.com to learn more. What is Data Modeling, and what is not?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Weekly

MARCH 24, 2024

link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024. As we predicted in the key trends of 2023 about Apache Flink as a clear winner in the stream processing frameworks, we see Confluent offering Flink as a service.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Kafka 89

Kafka Technology IT Government 89

Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Visit rudderstack.com to learn more. There will be food, networking, and real-world talks around data engineering. link] Nvidia: What Is Sovereign AI?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Monte Carlo

FEBRUARY 8, 2024

Just as a watchmaker meticulously adjusts every tiny gear and spring in harmonious synchrony for flawless performance, modern data pipeline optimization requires a similar level of finesse and attention to detail. Learn how cost, processing speed, resilience, and data quality all contribute to effective data pipeline optimization.

Data Pipeline 52

Data Pipeline AWS Datasets Data 52

Snowflake

MARCH 14, 2024

As a cohesive ERP solution, SAP is often one of the largest data resources in an organization, containing everything from financial and transactional data to master information about customers, vendors, materials, facilities, planning and even HR. What’s the challenge with unlocking SAP data?

IT 88

IT Data Ingestion Data AWS 88

Confluent

JANUARY 24, 2024

Learn why Confluent was named as a leader among cloud data pipelines, innovating, in our opinion, every industry with real-time stream processing and analytics, cloud-native Apache Kafka®, and robust developer tooling.

Data Pipeline 57

Data Pipeline Cloud Kafka Data 57

Knowledge Hut

NOVEMBER 2, 2023

Azure Data engineering projects are complicated and require careful planning and effective team participation for a successful completion. While many technologies are available to help data engineers streamline their workflows and guarantee that each aspect meets its objectives, ensuring that everything works properly takes time.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Cloudera

JUNE 26, 2023

Here are some tips and tricks of the trade to prevent well-intended yet inappropriate data engineering and data science activities from cluttering or crashing the cluster. For data engineering and data science teams, CDSW is highly effective as a comprehensive platform that trains, develops, and deploys machine learning models.

ETL Tools 75

ETL Tools Programming Language Datasets Data Pipeline 75

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions. We expect complete and accurate data at the end of each run.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Snowflake

AUGUST 16, 2023

Applications Snowflake Native App Framework now available in AWS – public preview Snowflake Native Apps are an entirely new way to put data to work. Learn more here. Learn more here. Data comes in a continuous manner, and often a separate architecture is required to handle streaming data.

Scala 52

Scala Transportation Kafka Data Lake 52

Monte Carlo

JANUARY 3, 2024

Data teams are scrambling to answer the call. That quick check of the box feels like a step forward, but if you aren’t already thinking about how to connect LLMs with your proprietary data and business context to actually drive differentiated value, you’re behind. Hard truth #4: Your data isn’t ready yet anyway.

Technology 113

Technology Database Data Governance Data Engineering 113

Confluent

AUGUST 16, 2022

Learn how you can integrate data streams into your environment, and enrich data across your existing data pipelines using Confluent Cloud.

Data Pipeline 97

Data Pipeline Cloud Data 97

Striim

AUGUST 26, 2023

While Confluent is a well-known option for data streaming platforms, its complexity can pose significant challenges for businesses. This complexity not only has technical repercussions but also impacts business operations by leading to unreliable or missing data, which can be both costly and time-consuming to rectify.

MongoDB 52

MongoDB Google Cloud Kafka AWS 52

Netflix Tech

JANUARY 13, 2022

By Vikram Srivastava and Marcelo Mayworm Netflix has one of the most complex data platforms in the cloud on which our data scientists and engineers run batch and streaming workloads. And we can’t discount the productivity impact it causes on data platform users.

Kafka 95

Kafka Big Data Data Machine Learning 95

Workfall

JUNE 12, 2023

In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Data, the lifeblood of organisations, holds the key to unlocking untapped potential and propelling businesses forward. In this blog, we will cover: What Is CDC and Its Benefits?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Monte Carlo

APRIL 1, 2024

But, for data engineers, there’s something else that comes pretty close to the top of that list: clean data. Data cleaning is an essential step to ensure your data is safe from the adage “garbage in, garbage out.” Define Clear Data Quality Standards 2. Implement Routine Data Audits 3. Table of Contents 1.

High Quality Data 52

High Quality Data Datasets Data Data Pipeline 52

Netflix Tech

NOVEMBER 17, 2022

To do so, we continuously push the boundaries of streaming video quality and leverage the best video technologies. There are, roughly speaking, two steps to encode a video in our pipeline: Video preprocessing, which encompasses any transformation applied to the high-quality source video prior to encoding.

Media 117

Media Architecture Algorithm Designing 117

Snowflake

JUNE 28, 2023

At Snowflake, we’re helping data scientists, data engineers, and application developers build faster and more efficiently in the Data Cloud. To make it even easier to process data with Snowpark Python UDFs and Stored Procedures, we have added support for Python 3.9 and unstructured data support, now in public preview.

Pipeline-centric 56

Pipeline-centric Programming Language Government Python 56

Netflix Tech

DECEMBER 14, 2020

The case of the extra 40 ms By: John Blair , Netflix Partner Engineering The Netflix application runs on hundreds of smart TVs, streaming sticks and pay TV set top boxes. Meanwhile, a field engineer for the chip vendor had diagnosed the root cause: Netflix’s Android TV application, called Ninja, was not delivering audio data quickly enough.

Bytes 140

Bytes Engineering Manufacturing Coding 140

Data Engineering Weekly

MARCH 10, 2024

Editor’s Note: Chennai Meetup Wrap-Up & Preparation work started for DEWCon I am so grateful for the enthusiastic participants who made our Chennai Data Heroes- Community for Data Folks meetup vibrant! Big thanks to our insightful speakers, Hareshkumar Selvakumar - Talks about his work on Data Products for PayPal.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Weekly

DECEMBER 3, 2023

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Visit rudderstack.com to learn more. The blog is an excellent read to understand late-arriving data, backfilling, and incremental processing complications.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Towards Data Science

APRIL 3, 2023

Building a Mental Model for Engineers and Anyone in Between Stream Processing can be handled gently and with care, or wildly, and almost out of control! By processing a smaller set of data, more often , you effectively divide and conquer a data problem that may otherwise be cost and time prohibitive.

Process 87

Process Data Lake Systems Bytes 87

Knowledge Hut

NOVEMBER 17, 2023

Azure Data Engineers play an important role in building efficient, secure, and intelligent data solutions on Microsoft Azure's powerful platform. The position of Azure Data Engineers is becoming increasingly important as businesses attempt to use the power of data for strategic decision-making and innovation.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Cloudera

JUNE 28, 2022

In a previous blog of this series, Turning Streams Into Data Products , we talked about the increased need for reducing the latency between data generation/ingestion and producing analytical results and insights from this data. containing data that may have to be used to enrich the streaming data.

Process 80

Process Kafka SQL Machine Learning 80

Striim

NOVEMBER 30, 2022

Businesses of all scales and industries have access to increasingly large amounts of data, which need to be harnessed effectively. According to an IDG Market Pulse survey , companies collect data from 400 sources on average. However, fragmented data can slow down the delivery of great product experiences and internal operations.

AWS 52

AWS Cloud Management MySQL 52