Accessibility, Data Governance and Data Pipeline

Practical First Steps In Data Governance For Long Term Success

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of the organization. At what point does a lack of an explicit governance policy become a liability?

Data Governance

Data Governance Government Data Lake High Quality Data

Low Friction Data Governance With Immuta

Data Engineering Podcast

DECEMBER 21, 2020

Summary Data governance is a term that encompasses a wide range of responsibilities, both technical and process oriented. One of the more complex aspects is that of access control to the data assets that an organization is responsible for managing. What is data governance?

Data Governance

Data Governance Government Data Lake Banking

Simplified End-to-End Development for Production-Ready Data Pipelines, Applications, and ML Models

Snowflake

JUNE 4, 2024

Snowflake’s new Python API (GA soon) simplifies data pipelines and is readily available through pip install snowflake. Additionally, Dynamic Tables are a new table type that you can use at every stage of your processing pipeline. Interact with Snowflake objects directly in Python. Automate or code, the choice is yours.

Data Pipeline

Data Pipeline Python SQL Government

Webinars

Leading the Development of Profitable and Sustainable Products

How To Get Promoted In Product Management

MORE WEBINARS

The Future of Data Governance: 4 Trends to Watch Out For

Monte Carlo

JANUARY 24, 2024

Data is among your company’s most valuable commodities, but only if you know how to manage it. More data, more access to data, and more regulations mean data governance has become a higher-stakes game. At the same time, data governance technologies are growing more intelligent.

Data Governance

Data Governance Government Data Lake Data Architecture

Data Governance: Framework, Tools, Principles, Benefits

Knowledge Hut

APRIL 20, 2023

Data governance refers to the set of policies, procedures, mix of people and standards that organisations put in place to manage their data assets. It involves establishing a framework for data management that ensures data quality, privacy, security, and compliance with regulatory requirements.

Data Governance

Data Governance Government Data Cleanse Data Security

Zenlytic Is Building You A Better Coworker With AI Agents

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Want to see Starburst in action?

Building

Building Data Lake High Quality Data Business Intelligence

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. What are the most interesting, innovative, or unexpected ways that you have seen Trino lakehouses used?

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

Data Pipeline Architecture Explained: 6 Diagrams and Best Practices

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline

Data Pipeline Architecture Data Lake Data Warehouse

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data BI Data Workflow

X-Ray Vision For Your Flink Stream Processing With Datorios

Data Engineering Podcast

JUNE 9, 2024

[Starburst Logo]([link] This episode is brought to you by Starburst - an end-to-end data lakehouse platform for data engineers who are battling to build and scale high quality data pipelines on the data lake. Want to see Starburst in action? Want to see Starburst in action?

Process

Process Data Lake High Quality Data Machine Learning

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. How does that change as a function of the type of data? What are the requirements around governance and auditability of data access that need to be addressed when sharing data? tabular, image, etc.)

Data Lake

Data Lake High Quality Data Government Data Pipeline

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building

Building Data Lake High Quality Data Machine Learning

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building

Building Data Lake High Quality Data Machine Learning

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Release Management For Data Platform Services And Logic

Data Engineering Podcast

MAY 12, 2024

[Starburst Logo]([link] This episode is brought to you by Starburst - an end-to-end data lakehouse platform for data engineers who are battling to build and scale high quality data pipelines on the data lake. Want to see Starburst in action? Want to see Starburst in action?

Management

Management Data Lake High Quality Data Machine Learning

Introducing Snowflake Notebooks, an End-to-End Interactive Environment for Data & AI Teams

Snowflake

JUNE 6, 2024

A look inside Snowflake Notebooks: A familiar notebook interface, integrated within Snowflake’s secure, scalable platform Keep all your data and development workflows within Snowflake’s security boundary, minimizing the need for data movement. Discover valuable business insights through exploratory data analysis. The best part?

SQL

SQL Python Data Workflow Machine Learning

Data Migration Strategies For Large Scale Systems

Data Engineering Podcast

MAY 26, 2024

Starburst Logo]([link] This episode is brought to you by Starburst - an end-to-end data lakehouse platform for data engineers who are battling to build and scale high quality data pipelines on the data lake. If you've learned something or tried out a project from the show then tell us about it!

Systems

Systems Data Lake High Quality Data Google Cloud

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.

SQL

SQL Data Lake High Quality Data Data Pipeline

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.

Non-relational Database

Non-relational Database Relational Database Database Designing

Join us at the Iceberg Summit 2024

Cloudera

MAY 10, 2024

Leveraging Cloudera’s platform, powered by Iceberg, organizations can transform their data and analytics capabilities with open data lakehouses, making the most of data from across the entire enterprise with unique tools and no unnecessary data movement or transformations along the way.

Government

Government Data Governance Data Pipeline Big Data

DataOps Framework: 4 Key Components and How to Implement Them

Databand.ai

AUGUST 30, 2023

It emphasizes the importance of collaboration between different teams, such as data engineers, data scientists, and business analysts, to ensure that everyone has access to the right data at the right time. This includes data ingestion, processing, storage, and analysis.

Data Governance

Data Governance Data Pipeline Government Data Cleanse

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Data Engineering Podcast

MARCH 31, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.

Project

Project Data Lake High Quality Data Data Workflow

When And How To Conduct An AI Program

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Programming

Programming Data Lake High Quality Data Data Pipeline

Modern Customer Data Platform Principles

Data Engineering Podcast

JANUARY 21, 2024

How have the architectural shifts changed the ways that organizations interact with their customer data? What are the governance policy and enforcement challenges that are added with the expansion of access and responsibility? How have the responsibilities shifted across different roles?

Data Lake

Data Lake High Quality Data NoSQL Data Warehouse

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Database

Database Technology Data Lake High Quality Data

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Database

Database Data Lake High Quality Data Data Workflow

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data Data Pipeline Architecture

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Engineering Podcast

JANUARY 7, 2024

[Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.

Data Process

Data Process Process Data Lake High Quality Data

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

[Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.

Data Lake

Data Lake Building High Quality Data AWS

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

[Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.

Kafka

Kafka Data Lake High Quality Data SQL

The last (but not least)”ops” you need for your data : DataGovops

François Nguyen

JANUARY 18, 2021

To finish the trilogy (Dataops, MLops), let’s talk about DataGovOps or how you can support your Data Governance initiative. Last part, it was added the data security and privacy part. Every data governance policy about this topic must be read by a code to act in your data platform (access management, masking, etc.)

Data Governance

Data Governance Metadata Government Data Pipeline

Addressing The Challenges Of Component Integration In Data Platform Architectures

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Go to [materialize.com]([link] today and get 2 weeks free!

Architecture

Architecture Data Lake High Quality Data SQL

Designing Data Platforms For Fintech Companies

Data Engineering Podcast

DECEMBER 31, 2023

How does that influence the architectural design/capabilities for data platforms in those organizations? Data governance is a notoriously challenging problem. What are the most interesting, innovative, or unexpected approaches to data management that you have seen in the fintech sector?

Designing

Designing Data Lake High Quality Data SQL

Toward a Data Mesh (part 2) : Architecture & Technologies

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? How can we interoperate between the data domains ? We want interoperability for any data stored versus we have to think how to store the data in a specific node to optimize the processing.

Technology

Technology Architecture Google Cloud Metadata

How DataOS Nails Gartner’s Magic Quadrant for Data Integration

The Modern Data Company

JANUARY 22, 2024

Founded on the principle of empowering every stakeholder to make data-driven decisions, Modern’s journey is intricately tied to the ideals of data democratization. Data engineering excellence Modern offers robust solutions for building, managing, and operationalizing data pipelines.

Data Integration

Data Integration Metadata Government Unstructured Data

Who Is Responsible For Data Quality? 5 Different Answers From Real Data Teams

Monte Carlo

JUNE 6, 2023

This post will focus on the most common team ownership models including: data engineering, data reliability engineering, analytics engineering, data quality analysts, and data governance teams. Table of Contents Why is important to answer who is responsible for data quality?

Data Governance

Data Governance Government Data Data Engineering

Back to the Financial Regulatory Future

Cloudera

FEBRUARY 15, 2024

Some of the key benefits of a modern data architecture for regulatory compliance include: Enhanced data governance and compliance: Modern data architecture incorporates data governance practices and security controls to ensure data privacy, regulatory compliance, and protection against unauthorized access or breaches.

Insurance

Insurance Banking Data Architecture Data Ingestion

5 Data Quality Tools—What They Are & When You Need Them

Monte Carlo

MARCH 13, 2024

Let’s dig in and explore the landscape of the top so-called “data quality tools” — what they are, what they’re not, and whether they’re the right first step towards more reliable data. Governance helps companies set important standards and achieve higher levels of data security, data accessibility, and data quality.

Data Governance

Data Governance Government Metadata Data

Data Observability: How Blinkist Prevents Broken Data Pipelines at Scale with Monte Carlo

Monte Carlo

FEBRUARY 22, 2021

Gopi Krishnamurthy, Director of Engineering, leads the team responsible for data engineering, infrastructure, cloud center-of-excellence, growth, and monetization. For Blinkist, having trustworthy and reliable data is foundational to the success of their business. Image courtesy of Blinkist. “At Image courtesy of Blinkist.

Data Pipeline

Data Pipeline Data Government Data Engineering

Which Team Should Own Data Quality?

Towards Data Science

JUNE 8, 2023

This post will focus on the most common team ownership models including: data engineering, data reliability engineering, analytics engineering, data quality analysts, and data governance teams. Why is data quality ownership important? The governance team treats every team output as a data product.

Data Governance

Data Governance Government Generalist Data Engineering

Data Lineage Tools: Key Capabilities and 5 Notable Solutions

Databand.ai

JULY 19, 2023

However, their importance has grown significantly in recent years due to the increasing complexity of data architectures and the growing need for data governance and compliance. In this article: Why Are Data Lineage Tools Important? Atlan Atlan offers a modern approach to data governance.

Pipeline-centric

Pipeline-centric Data Governance Metadata Government

Self Service is Simply Efficient – Cloudera DataFlow Designer GA announcement

Cloudera

MARCH 14, 2023

Data leaders will be able to simplify and accelerate the development and deployment of data pipelines, saving time and money by enabling true self service. It is no secret that data leaders are under immense pressure. Publishing: Users have access to a central catalog where they can more easily manage versioning of flows.

Designing

Designing Data Pipeline Data Governance Government

Designing Data Transfer Systems That Scale

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor.

Systems

Systems Designing Data Lake SQL

Practical First Steps In Data Governance For Long Term Success

Low Friction Data Governance With Immuta

Webinars

Trending Sources

Simplified End-to-End Development for Production-Ready Data Pipelines, Applications, and ML Models

Webinars

The Future of Data Governance: 4 Trends to Watch Out For

Data Governance: Framework, Tools, Principles, Benefits

Zenlytic Is Building You A Better Coworker With AI Agents

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Pipeline Architecture Explained: 6 Diagrams and Best Practices

Making Email Better With AI At Shortwave

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

X-Ray Vision For Your Flink Stream Processing With Datorios

Data Sharing Across Business And Platform Boundaries

Build Your Second Brain One Piece At A Time

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Release Management For Data Platform Services And Logic

Introducing Snowflake Notebooks, an End-to-End Interactive Environment for Data & AI Teams

Data Migration Strategies For Large Scale Systems

Tackling Real Time Streaming Data With SQL Using RisingWave

Designing A Non-Relational Database Engine

Join us at the Iceberg Summit 2024

DataOps Framework: 4 Key Components and How to Implement Them

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

When And How To Conduct An AI Program

Modern Customer Data Platform Principles

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Reconciling The Data In Your Databases With Datafold

Version Your Data Lakehouse Like Your Software With Nessie

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Build A Data Lake For Your Security Logs With Scanner

Troubleshooting Kafka In Production

The last (but not least)”ops” you need for your data : DataGovops

Addressing The Challenges Of Component Integration In Data Platform Architectures

Designing Data Platforms For Fintech Companies

Toward a Data Mesh (part 2) : Architecture & Technologies

How DataOS Nails Gartner’s Magic Quadrant for Data Integration

Who Is Responsible For Data Quality? 5 Different Answers From Real Data Teams

Back to the Financial Regulatory Future

5 Data Quality Tools—What They Are & When You Need Them

Data Observability: How Blinkist Prevents Broken Data Pipelines at Scale with Monte Carlo

Which Team Should Own Data Quality?

Data Lineage Tools: Key Capabilities and 5 Notable Solutions

Self Service is Simply Efficient – Cloudera DataFlow Designer GA announcement

Designing Data Transfer Systems That Scale

Stay Connected