Data Engineering Podcast

JULY 17, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. No more scripts, just SQL. No more scripts, just SQL.

Metadata 100

Metadata BI Data Warehouse Government 100

Christophe Blefari

SEPTEMBER 28, 2023

With a handy warehouse at the center and multiple SaaS tools revolving around to answer useful—sometimes not—use-cases. Historically, data pipelines were designed with an ETL approach, storage was expensive and we had to transform the data before using it. Following an E(T)LT approach.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Data Engineering Podcast

JULY 31, 2022

Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. How have the design and goals of the product changed or evolved? Can you describe how Manta is implemented?

IT 100

IT Metadata MongoDB MySQL 100

Monte Carlo

JULY 27, 2023

The data quality engineer will then need to design or optimize data architectures and pipelines to ensure these requirements are met (mentioned in 70% of job postings). This is a continuous process as data environments are complex, interdependent, and constantly changing. Ability to perform root cause analysis on defects.

Engineering 52

Engineering Healthcare Scala Data Warehouse 52

Ascend.io

DECEMBER 19, 2023

You’re extracting and loading data first, then transforming it in Snowflake’s cloud data warehouse. The GUI often allows for easy visualization and monitoring of data pipelines, which can be particularly beneficial for complex workflows. that you can combine to create custom data workflows.

ETL Tools 52

ETL Tools IT Data Pipeline Data Warehouse 52

Knowledge Hut

SEPTEMBER 25, 2023

You can learn deeply about the Azure learning journey that leads to the position of an Azure Data Engineer Associate on the Microsoft Azure platform. Who is an Azure Data Engineer? As an Azure Data Engineer, you will be expected to design, implement, and manage data solutions on the Microsoft Azure cloud platform.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Workfall

JUNE 12, 2023

CDC also plays a crucial role in data integration and ETL processes. It captures incremental changes from transactional databases or other sources, efficiently loading them into data warehouses or data lakes. This table will serve as the source of data, and any changes made here will be reflected in your data warehouse.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Hepta Analytics

FEBRUARY 14, 2022

We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design. When the business intelligence needs change, they can go query the raw data again. The purpose of the data is not determined.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Knowledge Hut

NOVEMBER 2, 2023

An Azure Data Engineer is a professional who is in charge of designing, implementing, and maintaining data processing systems and solutions on the Microsoft Azure cloud platform. A Data Engineer is responsible for designing the entire architecture of the data flow while taking the needs of the business into account.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Monte Carlo

SEPTEMBER 20, 2022

Here’s how Prefect , Series B startup and creator of the popular data orchestration tool, harnessed the power of data observability to preserve headcount, improve data quality and reduce time to detection and resolution for data incidents. But a growing company means growing data needs. Image courtesy of Prefect.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Knowledge Hut

DECEMBER 28, 2023

This blog will guide us through the Azure Data Engineer certification path , equipping us with insights necessary for this transformative journey. Who is an Azure Data Engineer? An Azure Data Engineer is responsible for designing, implementing and managing data solutions on Microsoft Azure.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

dbt Developer Hub

MARCH 23, 2023

Easy and intuitive code : Because audit_helper relies on dbt macros, it was designed to be an intuitive tool that runs on simple SQL queries. Optional: Disable the print_table() command, so the model can be materialized on your data warehouse.

Data Warehouse 64

Data Warehouse Generalist SQL Coding 64

Data Engineering Podcast

JANUARY 1, 2022

In this episode Emily Riederer shares her work to create a controlled vocabulary for managing the semantic elements of the data managed by her team and encoding it in the schema definitions in her data warehouse. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Warehouse 100

Data Warehouse BI Data Workflow Data 100

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

DataKitchen

JANUARY 3, 2022

There’s a recent trend toward people creating data lake or data warehouse patterns and calling it data enablement or a data hub. DataOps expands upon this approach by focusing on the processes and workflows that create data enablement and business analytics. DataOps Process Hub.

Business Analyst 98

Business Analyst Data Lake Consulting Data Analytics 98

Netflix Tech

OCTOBER 27, 2020

Usually Data scientists and engineers write Extract-Transform-Load (ETL) jobs and pipelines using big data compute technologies, like Spark or Presto , to process this data and periodically compute key information for a member or a video. The processed data is typically stored as data warehouse tables in AWS S3.

Data Warehouse 97

Data Warehouse Datasets Data Big Data 97

Cloudera

FEBRUARY 11, 2019

Cloudera provides a unified platform with multiple data apps and tools, big data management, hybrid cloud deployment flexibility, admin tools for platform provisioning and control, and a shared data experience for centralized security, governance, and metadata management.

Data Management 43

Data Management Management Metadata Government 43

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

ProjectPro

JANUARY 20, 2016

Apache Pig helps SQL server professionals create parallel data workflows. Apache pig eases data manipulation over multiple data sources using a combination of tools. Using Hive SQL professionals can use Hadoop like a data warehouse.

Hadoop 52

Hadoop SQL Java Big Data 52

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. By improving data quality, organizations can generate more accurate and reliable insights from their data analytics processes.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. In this episode Satish Jayanthi explains how he is building a framework to allow enterprises to move quickly while maintaining guardrails for data workflows.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Monte Carlo

MAY 14, 2024

Let’s revisit how several of those key table formats have emerged and developed over time: Apache Avro : Developed as part of the Hadoop project and released in 2009, Apache Avro provides efficient data serialization with a schema-based structure.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

OCTOBER 29, 2021

In this episode Ahmed Elsamadisi shares his journey to find a more flexible and universal data model in the form of the "activity schema" that is powering the Narrator platform, and how it has allowed his customers to perform self-service exploration of their business domains without being blocked by schema evolution in the data warehouse.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Pipeline 100

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

JUNE 1, 2021

In this episode he explains the utility of a low code solution that lets non engineers create their own self-serve pipelines, how the Shipyard platform is designed to make that possible, and how it allows engineers to create reusable tasks to satisfy the specific needs of the business. task-based, data assets, etc.)

Data Pipeline 100

Data Pipeline Data Warehouse Data Workflow Data 100

Data Engineering Podcast

NOVEMBER 14, 2021

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. What are the tradeoffs that you see in each approach?

Data Warehouse 100

Data Warehouse BI Data Workflow Data 100

dbt Developer Hub

APRIL 24, 2023

Nonetheless, Alteryx may have difficulties to cope with the complexity increase within an organization’s data pipeline, and it can become a suboptimal tool when companies start dealing with large and complex data transformations. The graphic interface of Alteryx Designer is presented in Figure 1.

BI 83

BI Data Workflow SQL Data Pipeline 83

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Monte Carlo

APRIL 28, 2023

Data Engineers : Data engineers who work with data pipelines and ETL processes use DuckDB to enhance the performance of their data processing tasks. They can leverage DuckDB’s efficient query execution and integration with data formats like Parquet to optimize their data workflows.

Data Warehouse 52

Data Warehouse Programming Language Datasets Data Pipeline 52

Ascend.io

MAY 31, 2023

They are different tools with different capabilities, designed to handle different tasks. This article delves into the “data pipeline vs ETL” discussion, clearly defining their differences and similarities, and why understanding these nuances is critical to an effective data strategy.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Meltano

OCTOBER 5, 2022

We’ll explain what a data engineer is, what the job entails, and how to become a data engineer. Plus, we’ll explain how data engineers use Meltano, our DataOps platform, for efficient data management. What Is Data Engineering? A data engineer’s focus is mainly on raw data, formats, security, and storage.

Data Engineering 52

Data Engineering Data Engineer Engineering Raw Data 52

Monte Carlo

MAY 25, 2023

While data orchestration tools might not be required for a pipeline to be considered “functional,” they’re nonetheless an essential component of the modern data stack, and serve as the connective tissue among various data warehouses. Automating data workflows. What is Apache Airflow? What is Prefect?

Data Pipeline 52

Data Pipeline Data Workflow Data Data Governance 52

Data Engineering Weekly

JANUARY 15, 2023

. 🎯 I defined the modern data stack sometime back as; @sarahmk125 MDS is a set of vendor tools that solve niche data problems (lineage, orchestration, quality) with the side effect of creating a disjointed data workflow that makes data folks lives more complicated.","username":"ananthdurai","name":"at-ananth-at-data-folks

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Meltano

OCTOBER 5, 2022

Think about data operations as a factory assembly line where a warehouse engineer optimizes and automates processes to increase productivity and product quality. In the same way, a DataOps engineer designs the data assembly line that enables data scientists to derive insights from data analytics faster and with fewer errors.

Engineering 52

Engineering Raw Data Data Pipeline ETL Tools 52

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. Database-centric In bigger organizations, Data engineers mainly focus on data analytics since the data flow in such organizations is huge.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

AltexSoft

NOVEMBER 7, 2022

Before we start, all those who are new to data engineering can watch our video explaining its general concepts. How data engineering works. Apache Airflow is an open-source Python -based workflow orchestrator that enables you to design, schedule, and monitor data pipelines. What is Apache Airflow?

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52