WeCloudData

OCTOBER 19, 2021

They use Kinesis Firehose and AWS Lambda to transform and store the data the devices collect. The data is served to the client’s app via RDS and Dynamo DB. The current pipeline randomly breaks, takes a long time to process data for frontend users, DynamoDB has a rate limit.

Consulting 52

Consulting AWS Aggregated Data Data Collection 52

WeCloudData

OCTOBER 19, 2021

They use Kinesis Firehose and AWS Lambda to transform and store the data the devices collect. The data is served to the client’s app via RDS and Dynamo DB. The current pipeline randomly breaks, takes a long time to process data for frontend users, DynamoDB has a rate limit.

Consulting 52

Consulting AWS Aggregated Data Data Collection 52

WeCloudData

OCTOBER 19, 2021

Their product is a solution that saves farms over $20,000 annually by improving energy efficiency […] The post Consulting Case Study: Real-time Data Streaming Pipeline Optimization appeared first on WeCloudData. It also provides farmers with the power to control their operating costs.

Consulting 52

Consulting Utilities Data Data Pipeline 52

Towards Data Science

OCTOBER 6, 2023

An End-To-End Tutorial for Beginners Continue reading on Towards Data Science »

Data Pipeline 94

Data Pipeline Building Data Science Data 94

Data Engineering Weekly

APRIL 28, 2023

Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives. On DEW #124, we selected the following article dbt: State of Analytics Engineering dbt publishes the state of analytical [data???🤔]

Consulting 52

Consulting Kafka Lambda Architecture Engineering 52

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineering 251

Data Engineering Data Engineer Engineering Data 251

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Want to see Starburst in action?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

APRIL 23, 2023

Meroxa was created to enable teams of all sizes to deliver real-time data applications. In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows. Can you describe what Meroxa is and the story behind it?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Snowflake

MARCH 2, 2023

Snowflake enables organizations to be data-driven by offering an expansive set of features for creating performant, scalable, and reliable data pipelines that feed dashboards, machine learning models, and applications. But before data can be transformed and served or shared, it must be ingested from source systems.

Kafka 119

Kafka Data Ingestion Data Pipeline Cloud Storage 119

Start Data Engineering

DECEMBER 11, 2022

Data pipeline patterns 3.1. Streaming 3.2. Multi-hop pipelines 3.3.2. Conditional/ Dynamic pipelines 3.3.3. Disconnected data pipelines 4. Source Ordering 2.3. Sink Overwritability 3. Extraction patterns 3.1.1. Time ranged 3.1.2. Full Snapshot 3.1.3. Lookback 3.1.4. Behavioral 3.2.1. Idempotent 3.2.2.

Data Pipeline 195

Data Pipeline Designing Data 195

Christophe Blefari

APRIL 12, 2024

The fest we deserve ( credits ) I hope this Data News finds you well. This is an episode in French and we talked mainly about the eventual end of the modern data stack. Build analytics at Hive.co — The journey Oleg and his team went through to implement a modern data stack. Reminds me Kestra so much.

BI 130

BI SQL Data Coding 130

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Data Engineering Podcast

MAY 21, 2023

Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. Stream processing technologies have been around for around a decade.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. With Materialize, you can!

Building 162

Building BI SQL Machine Learning 162

Christophe Blefari

MARCH 15, 2024

Saying mainly that " Sora is a tool to extend creativity " Last point Mira has been mocked and criticised online because as a CTO she wasn't able to say on which public / licensed data Sora has been trained on. Pandera, a data validation library for dataframes, now supports Polars.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community!

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Confluent

JANUARY 20, 2022

A data pipeline is a method for getting data from one system to another, whether for analytics purposes or for storage. Learning the elements that make up this proven architecture […].

Data Pipeline 62

Data Pipeline Kafka Architecture Data 62

Data Engineering Podcast

JANUARY 15, 2023

Summary The modern data stack has made it more economical to use enterprise grade technologies to power analytics at organizations of every scale. At the Modern Data Company they created the DataOS platform as a means of driving your full analytics lifecycle through code, while providing automatic knowledge graphs and data discovery.

Coding 130

Coding Building PostgreSQL Data Lake 130

Data Engineering Podcast

MAY 28, 2023

Summary Building a data team is hard in any circumstance, but at a startup it can be even more challenging. The requirements are fluid, you probably don't have a lot of existing data talent to manage the hiring and onboarding, and there is a need to move fast. What are the concepts that the new hire needs to know?

Data Lake 162

Data Lake Machine Learning Data Warehouse Education 162

Data Engineering Podcast

MAY 14, 2023

In this episode the host Tobias Macey shares his reflections on recent experiences where the abstractions leaked and some observances on how to deal with that situation in a data platform architecture. What do you have planned for the future of your data platform? When is ELT the wrong choice?

Data Lake 147

Data Lake Machine Learning Data Warehouse AWS 147

Rockset

MARCH 18, 2024

Flink is one of the most popular stream processing technologies, ranked as a top five Apache project and backed by a diverse committer community including Alibaba and Apple. Rockset customers using Flink often share how challenging it is to self-manage Flink for streaming transformations.

Cloud 64

Cloud Building Metadata Kafka 64

LinkedIn Engineering

JUNE 26, 2023

However, we've found that this vertical self-service model doesn't work particularly well for data pipelines, which involve wiring together many different systems into end-to-end data flows. Data pipelines power foundational parts of LinkedIn's infrastructure, including replication between data centers.

Data Pipeline 112

Data Pipeline Kafka MySQL SQL 112

Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Integration 162

Data Integration BI SQL Python 162

Rockset

APRIL 3, 2020

This problem arises due to a lack of real-time data on vehicle health or inventory health, to take preemptive action or real-time action. ARCHITECTURE: The proposed template of the solution and data pipeline for fleet management would look as shown in the below diagram. The client simulates the sensor data from a vehicle.

Data Pipeline 40

Data Pipeline Systems Management NoSQL 40

Data Engineering Podcast

DECEMBER 28, 2022

Summary With all of the messaging about treating data as a product it is becoming difficult to know what that even means. Vishal Singh is the head of products at Starburst which means that he has to spend all of his time thinking and talking about the details of product thinking and its application to data.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

Data Engineering Podcast

JUNE 4, 2023

Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Lake 130

Lyft Engineering

JUNE 6, 2023

Discover how Lyft identified and fixed performance issues in our streaming pipelines. Background Every streaming pipeline is unique. When reviewing a pipeline’s performance, we ask the following questions: “Is there a bottleneck?”, “Is the pipeline performing optimally?”, “Will it continue to scale with increased load?”

Utilities 123

Utilities Coding Python Systems 123

Data Engineering Podcast

DECEMBER 25, 2022

Summary Encryption and security are critical elements in data analytics and machine learning applications. We have well developed protocols and practices around data that is at rest and in motion, but security around data in use is still severely lacking. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Your data can be stable and wrong, but then it isn't reliable. Confidence in your data is achieved through constant validation and testing.

Project 130

Project Building Data Lake Machine Learning 130

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? The Platform Integration Data Store Transformation Orchestration Presentation Transportation Observability Closing What’s changed? Over the last three years, my life has changed as well.

Building 81

Building Transportation Data Lake Metadata 81

Data Engineering Podcast

JUNE 17, 2023

In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible for downstream use cases. By incorporating column-level lineage in the data modeling process it encourages a more robust and well-informed design.

Data Lake 130

Data Lake Metadata Data Architecture Machine Learning 130

Data Engineering Podcast

APRIL 16, 2023

Summary Business intellingence has been chasing the promise of self-serve data for decades. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

APRIL 9, 2023

Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. Now that the data warehouse has taken center stage a new approach of composable customer data platforms is emerging.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

Data Engineering Podcast

NOVEMBER 12, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147