Accessibility, Data Pipeline and Data Workflow

Snowflake’s New Python API Empowers Data Engineers to Build Modern Data Pipelines with Ease

Snowflake

APRIL 17, 2024

Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. Dive in to experience how the enhanced Python API streamlines your data workflows and unlocks the full potential of Python within Snowflake.

Data Pipeline

Data Pipeline Python Data Engineering Data Engineer

New Fivetran connector streamlines data workflows for real-time insights

ThoughtSpot

SEPTEMBER 6, 2023

Those coveted insights live at the end of a process lovingly known as the data pipeline. The pathway from ETL to actionable analytics can often feel disconnected and cumbersome, leading to frustration for data teams and long wait times for business users.

Data Workflow

Data Workflow Raw Data Data Lake Business Intelligence

4 Ways to Tackle Data Pipeline Optimization

Monte Carlo

FEBRUARY 8, 2024

Just as a watchmaker meticulously adjusts every tiny gear and spring in harmonious synchrony for flawless performance, modern data pipeline optimization requires a similar level of finesse and attention to detail. Learn how cost, processing speed, resilience, and data quality all contribute to effective data pipeline optimization.

Data Pipeline

Data Pipeline AWS Datasets Data

Webinars

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

Leading the Development of Profitable and Sustainable Products

How To Get Promoted In Product Management

MORE WEBINARS

Zenlytic Is Building You A Better Coworker With AI Agents

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Unfortunately this often turns into an exercise in frustration for everyone involved due to complex workflows and hard-to-understand dashboards. Want to see Starburst in action?

Building

Building Data Lake High Quality Data Business Intelligence

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data BI Data Workflow

Making Data Pipelines Self-Serve For Everyone With Shipyard

Data Engineering Podcast

JUNE 1, 2021

Summary Every part of the business relies on data, yet only a small team has the context and expertise to build and maintain workflows and data pipelines to transform, clean, and integrate it. RudderStack’s smart customer data pipeline is warehouse-first. task-based, data assets, etc.)

Data Pipeline

Data Pipeline Data Warehouse Data Workflow Data

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database

Non-relational Database Relational Database Database Designing

Why You Shouldn’t Use Notebooks for Production Data Pipelines

Ascend.io

AUGUST 18, 2023

This not only jeopardizes the integrity and robustness of production environments but also compounds challenges for both data scientists and engineers. This article delves into the reasons behind our assertion: data science notebooks are not your best choice for production data pipelines. What Are Jupyter Notebooks?

Data Pipeline

Data Pipeline Programming Language Data Science Coding

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

What Is Data Pipeline Automation?

Ascend.io

MARCH 17, 2023

These engineering functions are almost exclusively concerned with data pipelines, spanning ingestion, transformation, orchestration, and observation — all the way to data product delivery to the business tools and downstream applications. Pipelines need to grow faster than the cost to run them.

Data Pipeline

Data Pipeline Datasets Data Software Engineer

What Is Data Pipeline Automation?

Ascend.io

MARCH 17, 2023

These engineering functions are almost exclusively concerned with data pipelines, spanning ingestion, transformation, orchestration, and observation — all the way to data product delivery to the business tools and downstream applications. Pipelines need to grow faster than the cost to run them.

Data Pipeline

Data Pipeline Datasets Data Software Engineer

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Dagster offers a new approach to building and running data platforms and data pipelines. Your first 30 days are free!

Project

Project Data Lake High Quality Data Data Workflow

Data Pipeline Architecture Explained: 6 Diagrams and Best Practices

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline

Data Pipeline Architecture Data Lake Data Warehouse

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building

Building Data Lake High Quality Data Machine Learning

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building

Building Data Lake High Quality Data Machine Learning

Data Pipeline vs. ETL: Which Delivers More Value?

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline

Data Pipeline ETL Tools Pipeline-centric Data Warehouse

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Dagster offers a new approach to building and running data platforms and data pipelines. How does that change as a function of the type of data?

Data Lake

Data Lake High Quality Data Government Data Pipeline

Data pipeline asset management with Dataflow

Netflix Tech

FEBRUARY 9, 2022

JAR) form to be executed as part of the user defined data pipeline. data pipeline ?—?a DAG) for the purpose of transforming data using some business logic. Netflix homegrown CLI tool for data pipeline management. workflow ?—?see scala-workflow/main.sch.yaml. SQL) or compiled (e.g.

Data Pipeline

Data Pipeline Scala Management Python

Release Management For Data Platform Services And Logic

Data Engineering Podcast

MAY 12, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake.

Management

Management Data Lake High Quality Data Machine Learning

Data Migration Strategies For Large Scale Systems

Data Engineering Podcast

MAY 26, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Systems

Systems Data Lake High Quality Data Google Cloud

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database

Database Data Lake High Quality Data Data Workflow

Build vs Buy Data Pipeline Guide

Monte Carlo

APRIL 24, 2023

Investing in something like dbt makes creating and managing complex models faster and more reliable for expanding engineering and practitioner teams because dbt relies on modular SQL and engineering best practices to make data transforms more accessible. Missed Nishith’s 5 considerations? When should you use dbt Core?

Data Pipeline

Data Pipeline Building Data Ingestion BI

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Dagster offers a new approach to building and running data platforms and data pipelines.

SQL

SQL Data Lake High Quality Data Data Pipeline

Designing Data Transfer Systems That Scale

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Datafold : ![Datafold]([link]

Systems

Systems Designing Data Lake SQL

When And How To Conduct An AI Program

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Programming

Programming Data Lake High Quality Data Data Pipeline

Modern Customer Data Platform Principles

Data Engineering Podcast

JANUARY 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. How have the architectural shifts changed the ways that organizations interact with their customer data? How have the responsibilities shifted across different roles?

Data Lake

Data Lake High Quality Data NoSQL Data Warehouse

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Database

Database Technology Data Lake High Quality Data

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Data Pipeline Architecture

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Process

Data Process Process Data Lake High Quality Data

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake

Data Lake Building High Quality Data AWS

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Kafka

Kafka Data Lake High Quality Data SQL

Addressing The Challenges Of Component Integration In Data Platform Architectures

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Architecture

Architecture Data Lake High Quality Data SQL

3. Psyberg: Automated end to end catch up

Netflix Tech

NOVEMBER 14, 2023

Now, let’s explore the state of our pipelines after incorporating Psyberg. Pipelines After Psyberg Let’s explore how different modes of Psyberg could help with a multistep data pipeline. This ensures that the next instance of the workflow will pick up newer updates.

Metadata

Metadata Data Pipeline Scala Data Workflow

Toward a Data Mesh (part 2) : Architecture & Technologies

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. We want interoperability for any data stored versus we have to think how to store the data in a specific node to optimize the processing. We want to have our hands free and be totally devoted to devops principles.

Technology

Technology Architecture Google Cloud Metadata

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Data Engineering Podcast

NOVEMBER 12, 2023

Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data lakes are notoriously complex. What are some of the primary ways that developers interact with Tabnine during their development workflow? webapps vs. data pipelines vs. exploratory analysis, etc.)

Software Engineer

Software Engineer Software Engineering Engineering Data Lake

DataOps Framework: 4 Key Components and How to Implement Them

Databand.ai

AUGUST 30, 2023

It emphasizes the importance of collaboration between different teams, such as data engineers, data scientists, and business analysts, to ensure that everyone has access to the right data at the right time. This includes data ingestion, processing, storage, and analysis.

Data Governance

Data Governance Data Pipeline Government Data Cleanse

Designing Data Platforms For Fintech Companies

Data Engineering Podcast

DECEMBER 31, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Designing

Designing Data Lake High Quality Data SQL

ETL for Snowflake: Why You Need It and How to Get Started

Ascend.io

DECEMBER 19, 2023

That’s what we call a data pipeline. It could just as well be ‘ELT for Snowflake’ The key takeaway is that these terms are representative of the actual activity being undertaken: the construction and management of data pipelines within the Snowflake environment.

ETL Tools

ETL Tools IT Data Pipeline Data Warehouse

DataOps Tools: Key Capabilities & 5 Tools You Must Know About

Databand.ai

AUGUST 30, 2023

Each type of tool plays a specific role in the DataOps process, helping organizations manage and optimize their data pipelines more effectively. Data Integration Data integration is the process of collecting, transforming, and consolidating data from various sources.

Data Cleanse

Data Cleanse Data Pipeline Data Ingestion Data Validation

Data Orchestration: Defining, Understanding, and Applying

Ascend.io

DECEMBER 11, 2023

Here’s the deal: for data to truly drive your business forward, you need a reliable and scalable system to keep it moving without hiccups. In other words, you need data orchestration. In this article, we’ll break down what data orchestration is, its significance, and how it differs from data pipeline orchestration.

Data Workflow

Data Workflow Data Pipeline Data Lake Data

Adding An Easy Mode For The Modern Data Stack With 5X

Data Engineering Podcast

DECEMBER 17, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake

Data Lake High Quality Data SQL Architecture

Unlocking Your dbt Projects With Practical Advice For Practitioners

Data Engineering Podcast

NOVEMBER 19, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Project

Project Data Lake High Quality Data SQL

Snowflake’s New Python API Empowers Data Engineers to Build Modern Data Pipelines with Ease

New Fivetran connector streamlines data workflows for real-time insights

Webinars

Trending Sources

4 Ways to Tackle Data Pipeline Optimization

Webinars

Zenlytic Is Building You A Better Coworker With AI Agents

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Making Data Pipelines Self-Serve For Everyone With Shipyard

Designing A Non-Relational Database Engine

Why You Shouldn’t Use Notebooks for Production Data Pipelines

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Making Email Better With AI At Shortwave

What Is Data Pipeline Automation?

What Is Data Pipeline Automation?

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Data Pipeline Architecture Explained: 6 Diagrams and Best Practices

Build Your Second Brain One Piece At A Time

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Pipeline vs. ETL: Which Delivers More Value?

Data Sharing Across Business And Platform Boundaries

Data pipeline asset management with Dataflow

Release Management For Data Platform Services And Logic

Data Migration Strategies For Large Scale Systems

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Reconciling The Data In Your Databases With Datafold

Build vs Buy Data Pipeline Guide

Tackling Real Time Streaming Data With SQL Using RisingWave

Designing Data Transfer Systems That Scale

When And How To Conduct An AI Program

Modern Customer Data Platform Principles

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Version Your Data Lakehouse Like Your Software With Nessie

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Build A Data Lake For Your Security Logs With Scanner

Troubleshooting Kafka In Production

Addressing The Challenges Of Component Integration In Data Platform Architectures

3. Psyberg: Automated end to end catch up

Toward a Data Mesh (part 2) : Architecture & Technologies

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

DataOps Framework: 4 Key Components and How to Implement Them

Designing Data Platforms For Fintech Companies

ETL for Snowflake: Why You Need It and How to Get Started

DataOps Tools: Key Capabilities & 5 Tools You Must Know About

Data Orchestration: Defining, Understanding, and Applying

Adding An Easy Mode For The Modern Data Stack With 5X

Unlocking Your dbt Projects With Practical Advice For Practitioners

Stay Connected