Data, Data Governance, Data Pipeline and High Quality Data

Data Engineering Weekly #161

Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering

Data Engineering Data Engineer Pipeline-centric Engineering

How Fox Facilitates Data Trust with Governance and Monte Carlo

Monte Carlo

JANUARY 16, 2024

They run on data powered by Fox. Factor in the advertising strategies, media production, partner programming, audience analytics…and you’re looking at an ocean of data that would fill even the deepest trench (we’d like a television show about that too, please!). So how does Fox’s data strategy support these complex data workflows?

Government

Government High Quality Data Data Governance Entertainment

Practical First Steps In Data Governance For Long Term Success

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of the organization. At what point does a lack of an explicit governance policy become a liability?

Data Governance

Data Governance Government Data Lake High Quality Data

Webinars

Leading the Development of Profitable and Sustainable Products

How To Get Promoted In Product Management

MORE WEBINARS

Intrinsic Data Quality: 6 Essential Tactics Every Data Engineer Needs to Know

Monte Carlo

JANUARY 10, 2024

What happens when you strip away all the noise of queries and pipelines and focus on the data itself? You get down to the intrinsic data quality. What’s the difference between intrinsic and extrinsic data quality? Intrinsic data quality is the quality of data assessed independently of its use case.

Data Cleanse

Data Cleanse Data Engineering Data Engineer Engineering

Building a Winning Data Quality Strategy: Step by Step

Databand.ai

AUGUST 30, 2023

Building a Winning Data Quality Strategy: Step by Step Eitan Chazbani August 30, 2023 What Is a Data Quality Strategy? A data quality strategy details the processes, tools, and techniques employed to ensure your company’s data is accurate, consistent, complete, and up-to-date.

Building

Building Data Cleanse Data Governance Datasets

Data Accuracy vs Data Integrity: Similarities and Differences

Databand.ai

AUGUST 30, 2023

Data Accuracy vs Data Integrity: Similarities and Differences Eric Jones August 30, 2023 What Is Data Accuracy? Data accuracy refers to the degree to which data is correct, precise, and free from errors. In other words, it measures the closeness of a piece of data to its true value. What Is Data Integrity?

Data Integration

Data Integration Data Cleanse Data Validation Data Governance

Visionary Data Quality Paves the Way to Data Integrity

Precisely

MARCH 14, 2023

Then, data clouds from providers like Snowflake and Databricks made deploying and managing enterprise-grade data solutions much simpler and more cost-effective. Now, almost any company can build a solid, cost-effective data analytics or BI practice grounded in these new cloud platforms.

Data Integration

Data Integration High Quality Data BI Data

5 Hard Truths About Generative AI for Technology Leaders

Monte Carlo

JANUARY 3, 2024

Data teams are scrambling to answer the call. That quick check of the box feels like a step forward, but if you aren’t already thinking about how to connect LLMs with your proprietary data and business context to actually drive differentiated value, you’re behind. Hard truth #4: Your data isn’t ready yet anyway.

Technology

Technology Database Data Governance Data Engineering

Data Quality Engineer: Skills, Salary, & Tools Required

Monte Carlo

JULY 27, 2023

These specialists are also commonly referred to as data reliability engineers. To be successful in their role, data quality engineers will need to gather data quality requirements (mentioned in 65% of job postings) from relevant stakeholders. About 61% request you also have a formal computer science degree.

Engineering

Engineering Healthcare Scala Data Warehouse

AI Implementation: The Roadmap to Leveraging AI in Your Organization

Ascend.io

JANUARY 10, 2024

Visual representation of Conway’s Law ( source ) Read More: The Chief AI Officer: Avoid The Trap of Conway’s Law Process: Ensuring Data Readiness The backbone of successful AI implementation is robust data management processes. AI models are only as good as the data they consume, making continuous data readiness crucial.

Data Pipeline

Data Pipeline Government Data Governance Raw Data

Data Integrity vs. Data Quality: 4 Key Differences You Can’t Confuse

Monte Carlo

JUNE 16, 2023

Data integrity and quality may seem similar at first glance, and they are sometimes used interchangeably in everyday life, but they play unique roles in successful data management. When data has integrity, it means it wasn’t altered during storage, retrieval, or processing without authorization.

Data Integration

Data Integration Hospitality Banking Healthcare

Data Observability Tools: Types, Capabilities, and Notable Solutions

Databand.ai

JULY 5, 2023

What Are Data Observability Tools? Data observability tools are software solutions that oversee, analyze, and improve the performance of data pipelines. In this article: Why Are Data Observability Tools Important? Faster Troubleshooting Data pipeline failures are expensive and damaging to organizations.

Data Pipeline

Data Pipeline Data Lake Data Warehouse Datasets

IBM Loves DataOps

DataKitchen

FEBRUARY 18, 2022

DataOps is a discipline focused on the delivery of data faster, better, and cheaper to derive business value quickly. It closely follows the best practices of DevOps although the implementation of DataOps to data is nothing like DevOps to code. Data governance is necessary in the enforcement of Data Privacy.

High Quality Data

High Quality Data Business Intelligence Data Pipeline Government

[O’Reilly Book] Chapter 1: Why Data Quality Deserves Attention Now

Monte Carlo

AUGUST 31, 2023

Data is a priority for your CEO, as it often is for digital-first companies, and she is fluent in the latest and greatest business intelligence tools. But what happens when the data is wrong? What about a frantic email from your CTO about “duplicate data” in a business intelligence dashboard? The root of data downtime?

Data Lake

Data Lake Data Pipeline Unstructured Data Data Warehouse

What is Data Accuracy? Definition, Examples and KPIs

Monte Carlo

JULY 11, 2023

An anomaly certainly given how reliable digital maps are these days, but a data quality error nonetheless, specifically related to data accuracy. A key component of data quality is data accuracy – in fact, it’s one of the 6 overarching dimensions of data quality.

Data Cleanse

Data Cleanse Datasets Data Governance Government

Data Teams and Their Types of Data Journeys

DataKitchen

OCTOBER 2, 2023

Data Teams and Their Types of Data Journeys In the rapidly evolving landscape of data management and analytics, data teams face various challenges ranging from data ingestion to end-to-end observability. It explores why DataKitchen’s ‘Data Journeys’ capability can solve these challenges.

Data Ingestion

Data Ingestion Data Government Datasets

How to become Azure Data Engineer I Edureka

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering

Data Engineering Data Engineer Engineering Programming Language

Being Data Driven At Stripe With Trino And Iceberg

Data Engineering Podcast

JUNE 16, 2024

Summary Stripe is a company that relies on data to power their products and business. In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform.

Data Lake

Data Lake High Quality Data Metadata Python

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

Zenlytic Is Building You A Better Coworker With AI Agents

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Data lakes are notoriously complex.

Building

Building Data Lake High Quality Data Business Intelligence

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action? Can you describe what the focus of Dagster+ is and the story behind it?

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Release Management For Data Platform Services And Logic

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex.

Management

Management Data Lake High Quality Data Machine Learning

Forge Your Career Path with Best Data Engineering Certifications

ProjectPro

FEBRUARY 21, 2023

With so many data engineering certifications available , choosing the right one can be a daunting task. There are over 133K data engineer job openings in the US, but how will you stand out in such a crowded job market? The answer is- by earning professional data engineering certifications! AWS or Azure? Cloudera or Databricks?

Certification

Certification Data Engineering Data Engineer Engineering

Data Migration Strategies For Large Scale Systems

Data Engineering Podcast

MAY 26, 2024

When that system is responsible for the data layer the process becomes more challenging. Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments. Can you start by sharing some of your experiences with data migration projects?

Systems

Systems Data Lake High Quality Data Google Cloud

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL

SQL Data Lake High Quality Data Data Pipeline

X-Ray Vision For Your Flink Stream Processing With Datorios

Data Engineering Podcast

JUNE 9, 2024

Summary Streaming data processing enables new categories of data products and analytics. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex.

Process

Process Data Lake High Quality Data Machine Learning

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. From a product perspective, what are the data challenges that are posed by email?

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database

Database Data Lake High Quality Data Data Workflow

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

Summary Sharing data is a simple concept, but complicated to implement well. There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data Government Data Pipeline

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. Data lakes are notoriously complex. Join us at the top event for the global data community, Data Council Austin.

Data Lake

Data Lake High Quality Data Data Pipeline Architecture

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Building

Building Data Lake High Quality Data Machine Learning

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Building

Building Data Lake High Quality Data Machine Learning

Designing Data Platforms For Fintech Companies

Data Engineering Podcast

DECEMBER 31, 2023

Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Want to see Starburst in action?

Designing

Designing Data Lake High Quality Data SQL

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. Cliff Crosland co-founded Scanner to provide fast querying of high scale log data for security auditing. SIEM) A query engine is useless without data to analyze.

Data Lake

Data Lake Building High Quality Data AWS

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database

Non-relational Database Relational Database Database Designing

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. What are the open questions today in technical scalability of data engines?

Data Process

Data Process Process Data Lake High Quality Data

Designing Data Transfer Systems That Scale

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. With Materialize, you can!

Systems

Systems Designing Data Lake SQL

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Kafka

Kafka Data Lake High Quality Data SQL

When And How To Conduct An AI Program

Data Engineering Podcast

MARCH 3, 2024

Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about how to conduct an AI program for your organization. Data lakes are notoriously complex. Join us at the top event for the global data community, Data Council Austin. Your first 30 days are free!

Programming

Programming Data Lake High Quality Data Data Pipeline

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Join us at the top event for the global data community, Data Council Austin.

Database

Database Technology Data Lake High Quality Data

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Data Engineering Podcast

MARCH 31, 2024

Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. Dagster offers a new approach to building and running data platforms and data pipelines.

Project

Project Data Lake High Quality Data Data Workflow

Adding An Easy Mode For The Modern Data Stack With 5X

Data Engineering Podcast

DECEMBER 17, 2023

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. In this episode founder Tarush Aggarwal explains how the realities of the modern data stack are impacting data teams and the work that they are doing to accelerate time to value.

Data Lake

Data Lake High Quality Data SQL Architecture

Modern Customer Data Platform Principles

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Data projects are notoriously complex.

Data Lake

Data Lake High Quality Data NoSQL Data Warehouse

Addressing The Challenges Of Component Integration In Data Platform Architectures

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex. With Materialize, you can!

Architecture

Architecture Data Lake High Quality Data SQL

Data Engineering Weekly #161

How Fox Facilitates Data Trust with Governance and Monte Carlo

Webinars

Trending Sources

Practical First Steps In Data Governance For Long Term Success

Webinars

Intrinsic Data Quality: 6 Essential Tactics Every Data Engineer Needs to Know

Building a Winning Data Quality Strategy: Step by Step

Data Accuracy vs Data Integrity: Similarities and Differences

Visionary Data Quality Paves the Way to Data Integrity

5 Hard Truths About Generative AI for Technology Leaders

Data Quality Engineer: Skills, Salary, & Tools Required

AI Implementation: The Roadmap to Leveraging AI in Your Organization

Data Integrity vs. Data Quality: 4 Key Differences You Can’t Confuse

Data Observability Tools: Types, Capabilities, and Notable Solutions

IBM Loves DataOps

[O’Reilly Book] Chapter 1: Why Data Quality Deserves Attention Now

What is Data Accuracy? Definition, Examples and KPIs

Data Teams and Their Types of Data Journeys

How to become Azure Data Engineer I Edureka

Being Data Driven At Stripe With Trino And Iceberg

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Zenlytic Is Building You A Better Coworker With AI Agents

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Release Management For Data Platform Services And Logic

Forge Your Career Path with Best Data Engineering Certifications

Data Migration Strategies For Large Scale Systems

Tackling Real Time Streaming Data With SQL Using RisingWave

X-Ray Vision For Your Flink Stream Processing With Datorios

Making Email Better With AI At Shortwave

Reconciling The Data In Your Databases With Datafold

Data Sharing Across Business And Platform Boundaries

Version Your Data Lakehouse Like Your Software With Nessie

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Build Your Second Brain One Piece At A Time

Designing Data Platforms For Fintech Companies

Build A Data Lake For Your Security Logs With Scanner

Designing A Non-Relational Database Engine

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Designing Data Transfer Systems That Scale

Troubleshooting Kafka In Production

When And How To Conduct An AI Program

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Adding An Easy Mode For The Modern Data Stack With 5X

Modern Customer Data Platform Principles

Addressing The Challenges Of Component Integration In Data Platform Architectures

Stay Connected