Workfall

SEPTEMBER 18, 2023

Reading Time: 9 minutes Imagine your data as pieces of a complex puzzle scattered across different platforms and formats. This is where the power of data integration comes into play. Meet Airbyte, the data magician that turns integration complexities into child’s play. In this blog, we will cover: What is Airbyte?

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

Scott Logic

APRIL 22, 2024

Large-model AI is becoming more and more influential in the market, and with the well-known tech giants starting to introduce easy-access AI stacks, a lot of businesses are left feeling that although there may be a use for AI in their business, they’re unable to see what use cases it might help them with.

BI 97

BI Software Engineer Software Engineering Algorithm 97

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. In any case, I need your honest feedback, what would make you consider paying for the content I create? Each time I look back and I see the amount of Fridays I've spent reading and writing I'm still surprised.

Kafka 130

Kafka Data Lake Data Storage Data 130

Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Christophe Blefari

DECEMBER 31, 2023

Before moving on to the Data News, a bit of personal news, in December, I took part in the MotherDuck meetup in Berlin. I presented what I believe to be the future from my DuckDB experiments. End of January, on the 31st I'll speak at a Modern Data Stack conf in Paris, still about DuckDB, but this time in French.

Data 100

Data Cloud Storage Datasets Python 100

FreshBI

MAY 3, 2024

The purpose of this blog is to provide an example of how a manufacturing operation can use Business Intelligence (BI) anchored in its economic engine, to inform the ERP selection process. What are first things? We use the dashboard above as a guide and work back to the data required to be furnished by the ERP. So how do I do it?

Manufacturing 52

Manufacturing BI Process Banking 52

Data Engineering Weekly

FEBRUARY 4, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Joe Reis: Definition of Data Modeling & What Data Modeling Is not Joe raised a very fundamental question in data engineering.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions. We expect complete and accurate data at the end of each run.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Data Engineering Weekly

DECEMBER 3, 2023

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Github writes an excellent blog to capture the current state of the LLM integration architecture. Lackluster AI/ML results often stem from poor data quality.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights.

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. What is the data behavior? See how it works today. Dropbox: Is this a date?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Picnic Engineering

FEBRUARY 8, 2023

In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. In this post, we will explain how our team automates the creation of new data pipeline deployments. Now, we can have a pipeline ready in minutes. What is the Saga pattern?

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Databand.ai

JULY 17, 2023

MLOps: Similarities, Differences, and How to Choose Joseph Arnold July 17, 2023 What Is DataOps? DataOps , short for Data Operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data management processes.

Data Pipeline 52

Data Pipeline Machine Learning High Quality Data BI 52

Databand.ai

AUGUST 30, 2023

DataOps Framework: 4 Key Components and How to Implement Them Eric Jones August 30, 2023 What Is a DataOps Framework? The DataOps framework is a set of practices, processes, and technologies that enables organizations to improve the speed, accuracy, and reliability of their data management and analytics operations.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

Workfall

JUNE 12, 2023

In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Data, the lifeblood of organisations, holds the key to unlocking untapped potential and propelling businesses forward. In this blog, we will cover: What Is CDC and Its Benefits?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Cloudera

JUNE 2, 2022

Since 2015, the Cloudera DataFlow team has been helping the largest enterprise organizations in the world adopt Apache NiFi as their enterprise standard data movement tool. This need has generated a market opportunity for a universal data distribution service. Why does every organization need it when using a modern data stack?

Systems 98

Systems Data Lake Google Cloud Data Collection 98

Databand.ai

MAY 19, 2023

May Product Update: Snowflake Data Quality Alerts and Groups Helen Soloveichik May 19, 2023 Databand’s May product update is now available! Snowflake Data Quality Alerts: enhanced “data-at-rest” monitoring with out-of-the-box data quality alerts on Snowflake tables.

Datasets 52

Datasets Data Data Pipeline Accessible 52

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Airflow isn’t meant to process the data. See how it works today.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Data Engineering Weekly

MARCH 19, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real time with your own JavaScript or Python code. link] Sanjeev Mohan: What Exactly is a Data Product? Is chatGPT a data product? Is Data a product? What is Data Product, indeed?

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Lyft Engineering

NOVEMBER 29, 2023

Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data. For our customers, this means faster analytics on near real-time data and decision making. An example of how we use Druid rollup at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

ThoughtSpot

OCTOBER 27, 2023

Every business that analyzes their operational (or transactional) data needs to build a custom data pipeline involving several batch or streaming jobs to extract transactional data from relational databases , transform it, and load it into the data warehouse. Create a new data connection to Amazon Redshift.

Building 70

Building MySQL Data Warehouse SQL 70

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . CDP data lifecycle integration and SDX security and governance. Key features of CDP Data Engineering.

Data Engineering 124

Data Engineering Data Engineer Engineering Pipeline-centric 124

ThoughtSpot

OCTOBER 27, 2023

Every business that analyzes their operational (or transactional) data needs to build a custom data pipeline involving several batch or streaming jobs to extract transactional data from relational databases , transform it, and load it into the data warehouse. Create a new data connection to Amazon Redshift.

Building 69

Building MySQL Data Warehouse SQL 69

Cloudera

NOVEMBER 29, 2023

What does the Model Registry offer? The Model Registry records who made changes to a model, when these changes were made, and what the changes entailed. Robust APIs : The Model Registry’s APIs facilitate integration with CI/CD pipelines and essential tools in MLOps.

Machine Learning 82

Machine Learning Metadata Cloud Project 82

dbt Developer Hub

DECEMBER 11, 2023

What features are customers using? This blog post walks through the end-to-end process we used to set up product analytics for the dbt Semantic Layer using the dbt Semantic Layer. The steps we followed were broadly: Work with engineering to understand the data sources. Load the data into our warehouse.

Finance 64

Finance BI SQL Cloud 64

DoorDash Engineering

NOVEMBER 28, 2023

It is amusing for a human being to write an article about artificial intelligence in a time when AI systems, powered by machine learning (ML), are generating their own blog posts. As shown in Figure 1, data science intersects ML in multiple ways and is paramount to DoorDash’s success.

Machine Learning 90

Machine Learning Pipeline-centric Data Science Designing 90

Lyft Engineering

JUNE 28, 2023

However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Pinterest Engineering

NOVEMBER 7, 2023

Jeff Xiang | Software Engineer, Logging Platform Vahid Hashemian | Software Engineer, Logging Platform Jesus Zuniga | Software Engineer, Logging Platform At Pinterest, data is ingested and transported at petabyte scale every day, bringing inspiration for our users to create a life they love.

Kafka 98

Kafka Java Software Engineer Software Engineering 98

RandomTrees

APRIL 12, 2024

In this blog, we delve into the intricacies of model retraining, exploring its significance, various approaches, triggers, and best practices to empower organizations in mastering this essential component of MLOps. What is Model Retraining? Data drift occurs when the statistical properties of incoming data change over time.

Machine Learning 52

Machine Learning Datasets Systems Process 52

Knowledge Hut

MARCH 28, 2024

Data science has become one of the most trending fields today. Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 This demonstrates the increasing need for Microsoft Certified Data Engineers. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

DoorDash Engineering

NOVEMBER 7, 2023

We delve here into the different options we considered, why we believe our new design is most efficient, and what we have learned as we implemented this solution. At Doordash, we value every chance to boost order conversions in the app. Figure 1: An example of a real-time push notification to prompt purchase completion.

Designing 95

Designing Coding Process Management 95

Cloudera

MARCH 2, 2023

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. Iceberg is a high-performance open table format for huge analytic data sets. This enables you to maximize utilization of streaming data at scale. Currently, Iceberg support in CSP is in technical preview mode.

Process 112

Process SQL Kafka Database 112

Databand.ai

AUGUST 30, 2023

DataOps Architecture: 5 Key Components and How to Get Started Ryan Yackel August 30, 2023 What Is DataOps Architecture? DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Engineering at Meta

OCTOBER 24, 2023

In our last blog post on automatic product deprecation , we talked about the complexities of product deprecations, and a solution Meta has built called the Systematic Code and Asset Removal Framework (SCARF). Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing dead code.

Coding 128

Coding Programming Language Python MySQL 128

Databand.ai

JULY 19, 2023

Complete Guide to Data Ingestion: Types, Process, and Best Practices Helen Soloveichik July 19, 2023 What Is Data Ingestion? Data Ingestion is the process of obtaining, importing, and processing data for later use or storage in a database. However, the ultimate goal is simple: to prepare data for immediate use.

Data Ingestion 52

Data Ingestion Process Data Cleanse Data Governance 52