Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. Data stacks are becoming more and more complex.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Towards Data Science

OCTOBER 25, 2023

As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. You probably work closely with the colleague who builds the data pipeline that you were consuming. There was only one data team, two at most.

Data Pipeline 82

Data Pipeline SQL Data Architecture Data 82

ProjectPro

JUNE 16, 2022

An AWS data pipeline helps businesses move and unify their data to support several data-driven initiatives. It enables flow from a data lake to an analytics database or an application to a data warehouse. This blog will teach you about AWS Data Pipeline, its architecture, components, and benefits.

Data Pipeline 59

Data Pipeline AWS Amazon Web Services Data Consolidation 59

Data Engineering Podcast

FEBRUARY 6, 2022

In order to make geospatial analytics more maintainable and scalable there has been an increase in the number of database engines that provide extensions to their SQL syntax that supports manipulation of spatial data. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

SQL 100

SQL Building IT Google Cloud 100

Ascend.io

MARCH 17, 2023

These engineering functions are almost exclusively concerned with data pipelines, spanning ingestion, transformation, orchestration, and observation — all the way to data product delivery to the business tools and downstream applications. Pipelines need to grow faster than the cost to run them.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineer 52

Ascend.io

MARCH 17, 2023

These engineering functions are almost exclusively concerned with data pipelines, spanning ingestion, transformation, orchestration, and observation — all the way to data product delivery to the business tools and downstream applications. Pipelines need to grow faster than the cost to run them.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineer 52

Towards Data Science

SEPTEMBER 10, 2023

Data Teams are out of technical scope —  Tuning clusters or complex configurations (Databricks, Snowflake) is a time consuming task where data teams would rather be focusing on actual pipelines and SQL code. Two very popular scenarios are: data size growing over time, and code changes.

Data Pipeline 89

Data Pipeline Data Utilities Coding 89

Data Engineering Podcast

JUNE 25, 2023

Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights.

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Monte Carlo

APRIL 24, 2023

Data transformation Once you’ve ingested your raw data, it needs to be transformed into something structured before it can be fed into your BI tooling and data products. Generally speaking, most transformations are performed using SQL, leveraging open source solutions like dbt Core, or utilizing a managed service like dbt Cloud.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Hevo

FEBRUARY 9, 2023

Data Build Tool (DBT) is a powerful, SQL-based open-source tool. It helps organizations quickly build and maintain data transformations in their data pipelines. Building data pipelines is challenging, as we […]

Data Pipeline 52

Data Pipeline Building Data Warehouse Data Storage 52

Ascend.io

MAY 24, 2023

This article is based on a presentation given by Sarwat Fatima , Principal Data Engineer at Biome Analytics, at the Data Pipeline Automation Summit 2023. Dive right into Sarwat’s full presentation at the Data Pipeline Automation Summit 2023. Table of Contents Data is the lifeblood of the healthcare industry.

Healthcare 52

Healthcare Data Pipeline Hospitality Datasets 52

Ascend.io

JULY 28, 2023

Data pipelines are integral to business operations, regardless of whether they are meticulously built in-house or assembled using various tools. As companies become more data-driven, the scope and complexity of data pipelines inevitably expand. Ready to fortify your data management practice?

Data Pipeline 52

Data Pipeline Architecture Lambda Architecture Data Architecture 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Cloudera

FEBRUARY 9, 2023

Cloudera Stream Processing aims to take real-time data analytics to the next level. We’re excited to highlight job monitoring with notifications, a new feature for SQL Stream Builder (SSB). The sudden failing of a complex data pipeline can lead to devastating consequences — especially if it goes unnoticed.

SQL 78

SQL Kafka Aggregated Data Architecture 78

DataKitchen

JULY 27, 2023

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure.

Data Pipeline 52

Data Pipeline BI Machine Learning Data Preparation 52

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Multiple open source projects and vendors have been working together to make this vision a reality.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Cloudera

JUNE 2, 2021

Data pipelines are in high demand in today’s data-driven organizations. As critical elements in supplying trusted, curated, and usable data for end-to-end analytic and machine learning workflows, the role of data pipelines is becoming indispensable.

Data Pipeline 90

Data Pipeline Data Warehouse Machine Learning Data Architect 90

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Ascend.io

MAY 8, 2023

The costs of developing and running data pipelines are coming under increasing scrutiny because the bills for infrastructure and data engineering talent are piling up. For data teams, it is time to ask: “How can we have an impact on these runaway costs and still deliver unprecedented business value?”

Data Pipeline 52

Data Pipeline Data Ingestion Metadata Data 52

Christophe Blefari

MARCH 15, 2024

With yato you give a folder with SQL queries and it guesses the DAG and runs the queries in the right order. BigQuery supports DELETE to delete partitions in a SQL query. Arrow doing a lot of the data operation heavy lifting. Give a lot of insights on the market. this is more common sense but always works.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Towards Data Science

APRIL 6, 2023

Today’s post follows the same philosophy: fitting local and cloud pieces together to build a data pipeline. And, when it comes to data engineering solutions, it’s no different: They have databases, ETL tools, streaming platforms, and so on — a set of tools that makes our life easier (as long as you pay for them). not sponsored.

AWS 79

AWS Data Pipeline Amazon Web Services Python 79

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Snowflake

AUGUST 24, 2023

The family of companies, a part of Radian Group, relies on data science and machine learning to build and deploy models using a variety of data from different sources to bring technology to all parts of the home-buyer experience.

Data Pipeline 52

Data Pipeline Python Programming Language Data Validation 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Engineering at Meta

NOVEMBER 30, 2022

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. A limiting type system: Initially, we used only the fixed set of built-in Hive data types ( string , integer , boolean , etc. )

SQL 76

SQL Data Warehouse Metadata Coding 76

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Towards Data Science

MARCH 6, 2023

On-premise and cloud working together to deliver a data product Photo by Toro Tseleng on Unsplash Developing a data pipeline is somewhat similar to playing with lego, you mentalize what needs to be achieved (the data requirements), choose the pieces (software, tools, platforms), and fit them together. Google Cloud.

Google Cloud 85

Google Cloud Cloud Storage Data Pipeline Cloud 85

Data Engineering Podcast

APRIL 14, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Towards Data Science

OCTOBER 17, 2023

By: Jeff Chou, Stewart Bryson Image by Los Muertos Crew Databricks’ SQL warehouse products are a compelling offering for companies looking to streamline their production SQL queries and warehouses. What are Databricks’ SQL warehouse offerings? If cost is a concern to you, you can run SQL queries in jobs compute as well!

SQL 52

SQL Cloud Data Science AWS 52

Data Engineering Podcast

MARCH 24, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

Snowflake

MARCH 21, 2023

In Snowflake’s 6 Data Science Trends in 2023 , one of the more prominent trends we identified was the emerging use of unified tools and infrastructure for SQL and Python. Challenges with siloed data architecture SQL and Python are among the most popular languages in the modern data stack for transformations, analysis, and ML.

SQL 52

SQL Python Programming Language Scala 52

Towards Data Science

DECEMBER 23, 2023

Photo by Ian Taylor on Unsplash This tutorial guides you through an analytics use case, analyzing semi-structured data with Spark SQL. We’ll start with the data engineering process, pulling data from an API and finally loading the transformed data into a data lake (represented by MinIO ).

SQL 75

SQL Data Analytics Hadoop Raw Data 75

Monte Carlo

AUGUST 31, 2023

Every data science team has that moment where it realizes it has outgrown its initial architecture. That moment came quickly for our team and our SQL-based feature store. Examples include: when the data arrived , how many rows were loaded , rate of NULL values, and more signals of unreliable data.

SQL 64

SQL Data Science Machine Learning Algorithm 64

Data Engineering Podcast

APRIL 7, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162