Data Management, Data Pipeline, High Quality Data and Project

DataOps vs. MLOps: Similarities, Differences, and How to Choose

Databand.ai

JULY 17, 2023

DataOps , short for Data Operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data management processes. It aims to streamline the entire data lifecycle—from ingestion and preparation to analytics and reporting.

Data Pipeline

Data Pipeline Machine Learning High Quality Data BI

AI Implementation: The Roadmap to Leveraging AI in Your Organization

Ascend.io

JANUARY 10, 2024

By blending these elements, we lay a solid foundation, ensuring your AI projects don’t just start strong, but also deliver real, lasting value. AI models are only as good as the data they consume, making continuous data readiness crucial. Your data pipeline platform should excel in collecting data from a wide array of sources.

Data Pipeline

Data Pipeline Government Data Governance Raw Data

Data Engineering Weekly #161

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems.

Data Engineering

Data Engineering Data Engineer Pipeline-centric Engineering

Webinars

From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

How To Get Promoted In Product Management

MORE WEBINARS

Intrinsic Data Quality: 6 Essential Tactics Every Data Engineer Needs to Know

Monte Carlo

JANUARY 10, 2024

Intrinsic data quality is the quality of data assessed independently of its use case. Extrinsic data, meanwhile, is more about the context — it’s how your data interacts with the world outside and how it fits into the larger picture of your project or organization.

Data Cleanse

Data Cleanse Data Engineering Data Engineer Engineering

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Data Engineering Podcast

MARCH 31, 2024

To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. What are the challenges/shortcomings associated with those approaches?

Project

Project Data Lake High Quality Data Data Workflow

[O’Reilly Book] Chapter 1: Why Data Quality Deserves Attention Now

Monte Carlo

AUGUST 31, 2023

As the data analyst or engineer responsible for managing this data and making it usable, accessible, and trustworthy, rarely a day goes by without having to field some request from your stakeholders. But what happens when the data is wrong? This statistic probably comes as no surprise. It certainly didn’t to us.

Data Lake

Data Lake Data Pipeline Unstructured Data Data Warehouse

How to become Azure Data Engineer I Edureka

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering

Data Engineering Data Engineer Engineering Programming Language

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to make this vision a reality. Data lakes are notoriously complex.

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned commercial companion to the open source project they are investing in that capability with a suite of new features.

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Unlocking Your dbt Projects With Practical Advice For Practitioners

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects.

Project

Project Data Lake High Quality Data SQL

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building

Building Data Lake High Quality Data Machine Learning

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database

Non-relational Database Relational Database Database Designing

Release Management For Data Platform Services And Logic

Data Engineering Podcast

MAY 12, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex. If you've learned something or tried out a project from the show then tell us about it!

Management

Management Data Lake High Quality Data Machine Learning

Forge Your Career Path with Best Data Engineering Certifications

ProjectPro

FEBRUARY 21, 2023

Due to the enormous amount of data being generated and used in recent years, there is a high demand for data professionals, such as data engineers, who can perform tasks such as data management, data analysis, data preparation, etc.

Certification

Certification Data Engineering Data Engineer Engineering

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building

Building Data Lake High Quality Data Machine Learning

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL

SQL Data Lake High Quality Data Data Pipeline

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

The primary purpose of the catalog is to inform the query engine of what data exists and where, but the Nessie project aims to go beyond that simple utility. Data lakes are notoriously complex. How have the design and goals of the project changed since it was first created? Your first 30 days are free!

Data Lake

Data Lake High Quality Data Data Pipeline Architecture

When And How To Conduct An AI Program

Data Engineering Podcast

MARCH 3, 2024

Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about how to conduct an AI program for your organization. Data lakes are notoriously complex. What are the steps to take to build confidence in the availability of the data? Your first 30 days are free!

Programming

Programming Data Lake High Quality Data Data Pipeline

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Database

Database Technology Data Lake High Quality Data

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka

Kafka Data Lake High Quality Data SQL

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex. Can you start by outlining some of the situations where reconciling data between databases is needed?

Database

Database Data Lake High Quality Data Data Workflow

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

In this episode Andrew Jefferson explains the complexities of building a robust system for data sharing, the techno-social considerations, and how the Bobsled platform that he is building aims to simplify the process. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data Government Data Pipeline

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. What are the open questions today in technical scalability of data engines? What is your heuristic for when a given research project needs to be terminated or productionized?

Data Process

Data Process Process Data Lake High Quality Data

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake

Data Lake Building High Quality Data AWS

Designing Data Transfer Systems That Scale

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor.

Systems

Systems Designing Data Lake SQL

Building An “Amazon.com” For Your Data Products

Monte Carlo

JUNE 20, 2023

While it seems like it fits the requirements, you don’t quite trust it — who knows where the data in this shifty table has been. This project is your baby; you just can’t risk it. You can think of SLOs as measures that remove uncertainty surrounding the data and serve as a primary way to define trustworthiness for its consumers.

Building

Building Data Datasets High Quality Data

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure

DataKitchen

JULY 27, 2023

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure.

Data Pipeline

Data Pipeline BI Machine Learning Data Preparation

Modern Customer Data Platform Principles

Data Engineering Podcast

JANUARY 21, 2024

In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how he is applying the capabilities of cloud data warehouses to the challenge of building more comprehensive experiences for end-users through a modern customer data platform (CDP). Data projects are notoriously complex.

Data Lake

Data Lake High Quality Data NoSQL Data Warehouse

Designing Data Platforms For Fintech Companies

Data Engineering Podcast

DECEMBER 31, 2023

In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Designing

Designing Data Lake High Quality Data SQL

Adding An Easy Mode For The Modern Data Stack With 5X

Data Engineering Podcast

DECEMBER 17, 2023

In this episode founder Tarush Aggarwal explains how the realities of the modern data stack are impacting data teams and the work that they are doing to accelerate time to value. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Data Lake

Data Lake High Quality Data SQL Architecture

Shining Some Light In The Black Box Of PostgreSQL Performance

Data Engineering Podcast

NOVEMBER 5, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

PostgreSQL

PostgreSQL Data Lake High Quality Data SQL

Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

Data Engineering Podcast

DECEMBER 10, 2023

Andrew Maguire got tired of solving that problem for each of the different roles he has ended up in, so he created the open source Anomstack project. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data.

Data Lake

Data Lake High Quality Data SQL Architecture

Addressing The Challenges Of Component Integration In Data Platform Architectures

Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Data lakes are notoriously complex.

Architecture

Architecture Data Lake High Quality Data SQL

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Data Engineering Podcast

NOVEMBER 12, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Software Engineer

Software Engineer Software Engineering Engineering Data Lake

Build Your Python Data Processing Your Way And Run It Anywhere With Fugue

Data Engineering Podcast

FEBRUARY 20, 2022

In answer to that challenge the Fugue project offers an interface to automatically translate across Pandas, Spark, and Dask execution environments without having to modify your logic. Bigeye let’s data teams measure, improve, and communicate the quality of your data to company stakeholders.

Python

Python Data Process IT Building

8 Data Ingestion Tools (Quick Reference Guide)

Monte Carlo

FEBRUARY 20, 2024

Choosing one tool over another isn’t just about the features it offers today; it’s a bet on the future of how data will flow within organizations. Consideration What to Look For Integration Capabilities Support for a diverse array of data sources and destinations, ensuring compatibility with your data ecosystem.

Data Ingestion

Data Ingestion Google Cloud Kafka AWS

What is dbt Testing? Definition, Best Practices, and More

Monte Carlo

AUGUST 30, 2023

How to run generic or schema tests in dbt Within dbt Core, you’ll encounter a set of generic, predefined tests that can be easily applied to columns in your models by adding a few lines of YAML (a human-friendly data serialization language) in your dbt project. sql` file within the `models` directory of your dbt project.

SQL

SQL Datasets Database High Quality Data

Data Engineering Weekly #107

Data Engineering Weekly

NOVEMBER 13, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering

Data Engineering Data Engineer Engineering Kafka

Best Data Observability Tools (with RFP Template and Analyst Reports)

Monte Carlo

FEBRUARY 8, 2024

Gartner While Gartner hasn’t produced a data observability magic quadrant or report ranking data observability vendors, they have named it one of the hottest emerging technologies and placed it on the 2023 Data Management Hype Cycle. That means data engineers and high quality data are integral to the solution.

BI

BI Data Pipeline Data Warehouse Data

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Data Engineering Podcast

APRIL 7, 2024

In this episode Artyom Keydunov, creator of Cube, discusses the evolution and applications of the semantic layer as a component of your data platform, and how Cube provides speed and cost optimization for your data consumers. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data BI Data Workflow

How to Treat Your Data As a Product

Monte Carlo

MARCH 28, 2024

For the past few decades, most companies have kept data in an organizational silo. Analytics teams served business units, and even as data became more crucial to decision-making and product roadmaps, the teams in charge of data pipelines were treated more like plumbers and less like partners.

Data

Data High Quality Data Machine Learning Accessible

What is DataOps? The Ultimate Guide for Data Teams

Databand.ai

JANUARY 24, 2023

DataOps is an automated, process-oriented methodology used by analytics and data teams to improve quality and reduce the cycle times of data and analytics. The goal of DataOps is to deliver value faster by creating predictable delivery and change management of data, data models, and related artifacts.”

Retail

Retail Banking Government Manufacturing

DataOps vs. MLOps: Similarities, Differences, and How to Choose

AI Implementation: The Roadmap to Leveraging AI in Your Organization

Webinars

Trending Sources

Data Engineering Weekly #161

Webinars

Intrinsic Data Quality: 6 Essential Tactics Every Data Engineer Needs to Know

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

[O’Reilly Book] Chapter 1: Why Data Quality Deserves Attention Now

How to become Azure Data Engineer I Edureka

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Making Email Better With AI At Shortwave

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Unlocking Your dbt Projects With Practical Advice For Practitioners

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Designing A Non-Relational Database Engine

Release Management For Data Platform Services And Logic

Forge Your Career Path with Best Data Engineering Certifications

Build Your Second Brain One Piece At A Time

Tackling Real Time Streaming Data With SQL Using RisingWave

Version Your Data Lakehouse Like Your Software With Nessie

When And How To Conduct An AI Program

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Troubleshooting Kafka In Production

Reconciling The Data In Your Databases With Datafold

Data Sharing Across Business And Platform Boundaries

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Build A Data Lake For Your Security Logs With Scanner

Designing Data Transfer Systems That Scale

Building An “Amazon.com” For Your Data Products

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure

Modern Customer Data Platform Principles

Designing Data Platforms For Fintech Companies

Adding An Easy Mode For The Modern Data Stack With 5X

Shining Some Light In The Black Box Of PostgreSQL Performance

Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

Addressing The Challenges Of Component Integration In Data Platform Architectures

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Build Your Python Data Processing Your Way And Run It Anywhere With Fugue

8 Data Ingestion Tools (Quick Reference Guide)

What is dbt Testing? Definition, Best Practices, and More

Data Engineering Weekly #107

Best Data Observability Tools (with RFP Template and Analyst Reports)

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

How to Treat Your Data As a Product

What is DataOps? The Ultimate Guide for Data Teams

Stay Connected