High Quality Data and SQL - Data Engineering Digest

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable.

SQL

SQL Data Lake High Quality Data Data Pipeline

A Breakthrough AI-Powered SQL Assistant

Snowflake

APRIL 11, 2024

Data is the lifeblood of modern businesses, but unlocking its true insights often requires complex SQL queries. At Snowflake, we believe in making the power of data accessible to all. That’s why we prioritize simplicity, governance and quality in everything we build – including our AI-powered tools.

SQL

SQL AWS Data Analysis High Quality Data

Low Code And High Quality Data Engineering For The Whole Organization With Prophecy

Data Engineering Podcast

JULY 16, 2021

The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. No more scripts, just SQL.

High Quality Data

High Quality Data Data Engineering Data Engineer Coding

Webinars

How to Optimize the Developer Experience for Monumental Impact

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

Leading the Development of Profitable and Sustainable Products

MORE WEBINARS

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka

Kafka Data Lake High Quality Data SQL

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Engineering Podcast

MAY 5, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Building

Building Data Lake High Quality Data Machine Learning

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Building

Building Data Lake High Quality Data Machine Learning

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Non-relational Database

Non-relational Database Relational Database Database Designing

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Data Engineering Podcast

APRIL 7, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake

Data Lake High Quality Data BI Data Workflow

Addressing The Challenges Of Component Integration In Data Platform Architectures

Data Engineering Podcast

NOVEMBER 26, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Architecture

Architecture Data Lake High Quality Data SQL

Designing Data Platforms For Fintech Companies

Data Engineering Podcast

DECEMBER 31, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Designing

Designing Data Lake High Quality Data SQL

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. Data lakes are notoriously complex.

Database

Database Data Lake High Quality Data Data Workflow

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake

Data Lake Building High Quality Data AWS

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Process

Data Process Process Data Lake High Quality Data

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake

Data Lake High Quality Data Government Data Pipeline

Adding An Easy Mode For The Modern Data Stack With 5X

Data Engineering Podcast

DECEMBER 17, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Data Lake

Data Lake High Quality Data SQL Architecture

Unlocking Your dbt Projects With Practical Advice For Practitioners

Data Engineering Podcast

NOVEMBER 19, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Project

Project Data Lake High Quality Data SQL

Modern Customer Data Platform Principles

Data Engineering Podcast

JANUARY 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake

Data Lake High Quality Data NoSQL Data Warehouse

Shining Some Light In The Black Box Of PostgreSQL Performance

Data Engineering Podcast

NOVEMBER 5, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

PostgreSQL

PostgreSQL Data Lake High Quality Data SQL

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Project

Project Data Lake High Quality Data Data Workflow

Designing Data Transfer Systems That Scale

Data Engineering Podcast

DECEMBER 3, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Systems

Systems Designing Data Lake SQL

When And How To Conduct An AI Program

Data Engineering Podcast

MARCH 3, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Programming

Programming Data Lake High Quality Data Data Pipeline

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering Podcast

FEBRUARY 25, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Database

Database Technology Data Lake High Quality Data

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Data Engineering Podcast

NOVEMBER 12, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Software Engineer

Software Engineer Software Engineering Engineering Data Lake

Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

Data Engineering Podcast

DECEMBER 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Lake

Data Lake High Quality Data SQL Architecture

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake

Data Lake High Quality Data Data Pipeline Architecture

Building ETL Pipelines With Generative AI

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

Building

Building BI SQL Machine Learning

Easily Integrate Monte Carlo with Databricks via Partner Connect

Monte Carlo

OCTOBER 16, 2023

“The Databricks Lakehouse Platform enables organizations to unlock tremendous value from their data by delivering highly scalable, cost effective compute and storage. Monte Carlo helps protect that earned value by ensuring teams can then deliver that reliable high quality data at scale,” said Matt Sulkis, head of partnerships, Monte Carlo.

High Quality Data

High Quality Data SQL Accessible Accessibility

Build Your Python Data Processing Your Way And Run It Anywhere With Fugue

Data Engineering Podcast

FEBRUARY 20, 2022

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. With the Oxylabs scraper APIs you can extract data from even javascript heavy websites. Combined with their residential proxies you can be sure that you’ll have reliable and high quality data whenever you need it.

Python

Python Data Process IT Building

How we cut our tests by 80% while increasing data quality: the power of aggregating test failures in dbt

dbt Developer Hub

JANUARY 23, 2023

At Tempus , a precision medicine company specializing in oncology, high quality data is a necessary component for high quality clinical models. Aggregating test failure results using Jinja macros and pre-configured metadata to pull together high level summary tables. on BigQuery.

Metadata

Metadata High Quality Data SQL Data Integration

Is Prompt Engineering Overhyped? No—But Learn These 3 GenAI Skills Too

Monte Carlo

MARCH 7, 2024

Why prompt engineering isn’t all that and a bag of SQL queries Understand vector databases Create AI differentiation with RAG Find and solve real business problems High-quality data always lives up to the hype What is prompt engineering? Table of Contents Why is prompt engineering important?

Engineering

Engineering High Quality Data Database Architecture

Running demand forecasting machine learning models at scale

Picnic Engineering

DECEMBER 12, 2023

In some instances, calculating a moving average with SQL queries or leveraging already-known demand for products ordered by our customers will be good enough. For that use case a simple SQL query can be enough. When putting bad-quality data into machine learning models, you will get bad-quality output.

Machine Learning

Machine Learning Deep Learning Food High Quality Data

Ripple's Centralized Data Platform

Ripple Engineering

JANUARY 29, 2024

A lack of a centralized system makes building a single source of high-quality data difficult. Ripple Data Consumers query the data from the lake storage using the SQL strategy. It can lead to expensive, slow, and unmaintainable systems.

Database-centric

Database-centric Pipeline-centric NoSQL High Quality Data

How to Use DBT to Get Actionable Insights from Data?

Workfall

JULY 4, 2023

With DBT, they weave powerful SQL spells to create data models that capture the essence of their organization’s information. DBT’s superpowers include seamlessly connecting with databases and data warehouses, performing amazing transformations, and effortlessly managing dependencies to ensure high-quality data.

Data Warehouse

Data Warehouse SQL PostgreSQL Database

5 Layers of Data Lakehouse Architecture Explained

Monte Carlo

JANUARY 5, 2024

Data lakehouse architecture is an increasingly popular choice for many businesses because it supports interoperability between data lake formats. It supports ACID transactions and can run fast queries, typically through SQL commands, directly on object storage in the cloud or on-prem on structured and unstructured data.

Architecture

Architecture Data Lake Metadata Unstructured Data

Data Lakehouse Architecture Explained: 5 Layers

Monte Carlo

JANUARY 5, 2024

Data lakehouse architecture is an increasingly popular choice for many businesses because it supports interoperability between data lake formats. It supports ACID transactions and can run fast queries, typically through SQL commands, directly on object storage in the cloud or on-prem on structured and unstructured data.

Architecture

Architecture Data Lake Metadata Unstructured Data

What is dbt Testing? Definition, Best Practices, and More

Monte Carlo

AUGUST 30, 2023

Data testing is the first step in many data engineers’ journey toward reliable data. dbt (data build tool) is a SQL-based command-line tool that offers native testing features. dbt also happens to be an incredibly handy tool for running many data quality tests. This model will be saved as `.sql`

SQL

SQL Datasets Database High Quality Data

Data Engineering Weekly #107

Data Engineering Weekly

NOVEMBER 13, 2022

It moved from the speculation to the data engineers understanding the benefit of it and asking when we can get the implementation soon. I met many data leaders about Data Contracts, my project Schemata, and how the extended version we are building can help them create high-quality data. Not Endless DAGs!

Data Engineering

Data Engineering Data Engineer Engineering Kafka

From Big Data to Better Data: Ensuring Data Quality with Verity

Lyft Engineering

OCTOBER 3, 2023

High-quality data is necessary for the success of every data-driven company. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. Beginning backtest for 1 date(s) and 1 check(s).

Big Data

Big Data Metadata Data Warehouse Data

8 Data Quality Issues and How to Solve Them

Monte Carlo

AUGUST 9, 2023

Too much data Too much data might not sound like a problem (it is called big data afterall), but when rows populate out of proportion, it can slow model performance and increase compute costs. Volume tests It’s important to identify data volume changes as quickly as possible.

Finance

Finance High Quality Data SQL Data

Data Quality Testing: 7 Essential Tests

Monte Carlo

DECEMBER 19, 2022

Too much data Too much data might not sound like a problem (it is called big data afterall), but when rows populate out of proportion, it can slow model performance and increase compute costs. Essentially, does this data reflect reality? In this case, the SLI would be something like “hours since dataset refreshed.”

High Quality Data

High Quality Data Data Bytes SQL

Data Engineering Weekly #125

Data Engineering Weekly

APRIL 2, 2023

Meta: Presto - A Decade of SQL Analytics at Meta Presto and Kafka are the two systems that greatly impacted data infrastructure in the last decade. nHowever, High-Quality Data Creation and Data collaboration going to remain challenging. ","username":"ananthdurai","name":"at-ananth-at-data-folks

Data Engineering

Data Engineering Data Engineer Engineering Pipeline-centric

Tackling Real Time Streaming Data With SQL Using RisingWave

A Breakthrough AI-Powered SQL Assistant

Webinars

Trending Sources

Low Code And High Quality Data Engineering For The Whole Organization With Prophecy

Webinars

Troubleshooting Kafka In Production

Making Email Better With AI At Shortwave

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Build Your Second Brain One Piece At A Time

Designing A Non-Relational Database Engine

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Addressing The Challenges Of Component Integration In Data Platform Architectures

Designing Data Platforms For Fintech Companies

Reconciling The Data In Your Databases With Datafold

Build A Data Lake For Your Security Logs With Scanner

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Sharing Across Business And Platform Boundaries

Adding An Easy Mode For The Modern Data Stack With 5X

Unlocking Your dbt Projects With Practical Advice For Practitioners

Modern Customer Data Platform Principles

Shining Some Light In The Black Box Of PostgreSQL Performance

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Designing Data Transfer Systems That Scale

When And How To Conduct An AI Program

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

Version Your Data Lakehouse Like Your Software With Nessie

Building ETL Pipelines With Generative AI

Easily Integrate Monte Carlo with Databricks via Partner Connect

Build Your Python Data Processing Your Way And Run It Anywhere With Fugue

How we cut our tests by 80% while increasing data quality: the power of aggregating test failures in dbt

Is Prompt Engineering Overhyped? No—But Learn These 3 GenAI Skills Too

Running demand forecasting machine learning models at scale

Ripple's Centralized Data Platform

How to Use DBT to Get Actionable Insights from Data?

5 Layers of Data Lakehouse Architecture Explained

Data Lakehouse Architecture Explained: 5 Layers

What is dbt Testing? Definition, Best Practices, and More

Data Engineering Weekly #107

From Big Data to Better Data: Ensuring Data Quality with Verity

8 Data Quality Issues and How to Solve Them

Data Quality Testing: 7 Essential Tests

Data Engineering Weekly #125

Stay Connected