Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

ProjectPro

FEBRUARY 6, 2023

With over 10 million active subscriptions, 50 million active topics, and a trillion messages processed per day, Google Cloud Pub/Sub makes it easy to build and manage complex event-driven systems. Google Cloud Pub/Sub is a messaging service that allows apps and services to exchange event data. What is Google Pub/Sub?

Google Cloud 52

Google Cloud Cloud Cloud Storage Data Ingestion 52

Data Engineering Podcast

JUNE 12, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

DataKitchen

OCTOBER 20, 2023

.’ It assigns unique identifiers to each data item—referred to as ‘payloads’—related to each event. By offering real-time tracking mechanisms and sending targeted alerts to specific consumers, a Payload DJ can immediately notify them of any changes, delays, or issues affecting their data.

Insurance 70

Insurance Pharmaceutical Data Data Ingestion 70

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. With the increasing expecation for information to be instantly accessible, it drives the need for reliable change data capture. In fact, while only 3.5% That’s where our friends at Ascend.io

Food 100

Food MongoDB Scala MySQL 100

Knowledge Hut

MARCH 19, 2024

AWS Lambda AWS Lambda is a serverless computing service that enables developers to run code in response to events without needing to work with servers. It allows businesses to construct event-driven architectures and microservices in which functions are invoked by events like file uploads, database changes, or HTTP requests.

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Data Engineering Podcast

JULY 24, 2022

In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0 Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Cloudera

APRIL 20, 2021

Streaming Analytics is a type of data analysis that processes data streams for real-time analytics. It continuously processes data from multiple streams and performs simple calculations to complex event processing for delivering sophisticated use cases. What are the advantages of Streaming Analytics?

Hospitality 93

Hospitality Kafka Retail Data Ingestion 93

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. In fact, while only 3.5%

Food 100

Food MongoDB Scala MySQL 100

Snowflake

MARCH 21, 2023

In February, Snowflake launched new features around streaming data ingestion and data governance and improved SQL experience and performance, with enhancements to Search Optimization Service and more. Check out Felipe Hoffa’s video on how to use Snowsight to get from data to decision faster. Learn more here.

Retail 69

Retail Healthcare Data Ingestion Consulting 69

DataKitchen

OCTOBER 2, 2023

Data Teams and Their Types of Data Journeys In the rapidly evolving landscape of data management and analytics, data teams face various challenges ranging from data ingestion to end-to-end observability. It explores why DataKitchen’s ‘Data Journeys’ capability can solve these challenges.

Data Ingestion 40

Data Ingestion Data Government Datasets 40

Data Engineering Podcast

JULY 10, 2022

In this episode Chris Riccomini shares his experiences building and scaling data operations at WePay and LinkedIn, as well as the lessons he has learned working with other teams as they automated their own systems. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. In fact, while only 3.5%

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. What are the primary system requirements that have influenced the design choices? In fact, while only 3.5%

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Databand.ai

JUNE 28, 2023

“Observability” has become a bit of a buzzword so it’s probably best to define it: Data observability is the blanket term for monitoring and improving the health of data within applications and systems like data pipelines. Data observability vs. monitoring: what is the difference?

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Data Engineering Podcast

NOVEMBER 13, 2022

To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. In this episode Vishnu Venkataraman shares the journey that he and his team have taken to build and evolve their systems and improve the product offerings that they are able to support. In fact, while only 3.5%

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

OCTOBER 2, 2022

Summary Data lineage is something that has grown from a convenient feature to a critical need as data systems have grown in scale, complexity, and centrality to business. Alvin is a platform that aims to provide a low effort solution for data lineage capabilities focused on simplifying the work of data engineers.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

OCTOBER 30, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Sign up free at dataengineeringpodcast.com/rudder Data teams are increasingly under pressure to deliver. In fact, while only 3.5%

Engineering 100

Engineering MongoDB Scala MySQL 100

Striim

NOVEMBER 13, 2023

Introduction In the fast-evolving world of data integration, Striim’s collaboration with Snowflake stands as a beacon of innovation and efficiency. This mode is particularly useful for audit trails or scenarios where preserving the historical sequence of data changes is important.

Data Ingestion 59

Data Ingestion Utilities Data Integration Data 59

Data Engineering Podcast

OCTOBER 16, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Data Lake 100

Data Lake Food MongoDB Scala 100

Cloudera

AUGUST 26, 2020

With the way systems are set up, if an exchange let it go even for a few minutes it can go completely unchecked with huge financial implications. Flink could be running in the background and defining patterns and analyzing two different events too. From a data ingestion standpoint, NiFi is designed for this purpose.

Banking 82

Banking Data Ingestion Kafka Data Lake 82

Data Engineering Podcast

SEPTEMBER 18, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Hospitality 100

Hospitality Food MongoDB Scala 100

Knowledge Hut

NOVEMBER 2, 2023

These Azure data engineer projects provide a wonderful opportunity to enhance your data engineering skills, whether you are a beginner, an intermediate-level engineer, or an advanced practitioner. Who is Azure Data Engineer? Azure SQL Database, Azure Data Lake Storage).

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Cloudera

AUGUST 4, 2021

I will talk more about Predictive Maintenance in the Public Sector and how government agencies are optimizing equipment maintenance with data insight during Cloudera’s Industry Event on August 5. images, video, text, spectral data) or other input such as thermographic or acoustic signals. .

Unstructured Data 98

Unstructured Data Data Ingestion Government Machine Learning 98

Snowflake

SEPTEMBER 13, 2023

Observability Notifications for better observability – general availability With this release, the SYSTEM$SEND_EMAIL() system stored procedure is generally available for sending email notifications. Learn more about the GEOMETRY data type and ST_TRANSFORM on our blog. Learn more. Learn more here. © 2023 Snowflake Inc.

Python 74

Python SQL Data Pipeline Data Ingestion 74

Snowflake

AUGUST 16, 2023

Custom Event Billing capabilities enable you to build your own pricing strategy. You can charge customers based on their usage and specify billing events based on your preferences, such as consumed rows, ingested rows and more. What remains challenging is how streaming data is brought together with batch data.

Scala 52

Scala Transportation Kafka Data Lake 52

Ascend.io

JANUARY 2, 2024

These are the interfaces where the pipeline taps into various systems to acquire data. The sources of data can be incredibly diverse, ranging from data warehouses, relational databases, and web analytics to CRM platforms, social media tools, and IoT device sensors. In what format will the data be available?

Data Pipeline 52

Data Pipeline Building Raw Data Data Warehouse 52

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. You need to think about the whole model lifecycle.

Machine Learning 106

Machine Learning Python Kafka Java 106

Data Engineering Weekly

JUNE 4, 2023

[link] Uber: Spark Analysers: Catching Anti-Patterns In Spark Apps One of the challenges in commoditizing data processing engines like Spark is that it requires an expert user to understand and operate this system. The author writes an excellent introduction to vector databases and their applications.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Cloudera

JANUARY 20, 2021

The data journey is not linear, but it is an infinite loop data lifecycle – initiating at the edge, weaving through a data platform, and resulting in business imperative insights applied to real business-critical problems that result in new data-led initiatives. Conclusion.

Manufacturing 106

Manufacturing Data Warehouse Kafka Retail 106

Knowledge Hut

MARCH 22, 2024

It includes the service and capability portfolio that makes the device connectivity, data ingestion, analytics, and integration with other cloud services. Remote Monitoring and Control: Azure IoT allows remote monitoring and control of devices and systems in multiple locations. trillion by 2026.

Cloud Computing 52

Cloud Computing Utilities Cloud Transportation 52

Monte Carlo

MAY 26, 2023

One ecommerce platform lost around $5 million in revenue because their machine learning model that identified out of stock items and recommended substitutions was operating on thousands of temporary tables and stale data for six months. Pro-tip : Don’t confuse data freshness with data latency. This leaves two options.

Data Pipeline 98

Data Pipeline Data Data Warehouse Machine Learning 98

Knowledge Hut

APRIL 25, 2024

(with Example) Spatial analysis , commonly referred to as geospatial data science is a geographical solution that combines data science with geographic solutions like geographic information systems (GIS).In A typical data science process automatically integrates everything from analytics to data engineering.

Data Science 52

Data Science Telecommunication Transportation Big Data 52

Knowledge Hut

MAY 2, 2024

The order management system pushes the order status to the queue(could be Kafka) from where Streaming process reads every minute and picks all the orders with their status. Event Detection: Apache Spark is increasingly used in event detection like credit card fraud detection, money laundering activities etc.

Scala 52

Scala Hospitality Healthcare Retail 52

Pinterest Engineering

AUGUST 18, 2022

We considered various approaches for doing data migration and finalized the methodology based on various trade offs: Doing double writes ( writing to 2 sources of truths in sync/async fashion) from the service to both tables (HBase and TiDB) and using the TiDB backend mode in the lightning for data ingestion.

Data Ingestion 52

Data Ingestion Hadoop Database Kafka 52

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 56

Data Ingestion Kafka Java Data Pipeline 56

Rockset

JULY 28, 2022

CDC enables true real-time analytics on your application data, assuming the platform you send the data to can consume the events in real time. Options For Change Data Capture on MongoDB Apache Kafka The native CDC architecture for capturing change events in MongoDB uses Apache Kafka.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52