Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Knowledge Hut

SEPTEMBER 25, 2023

They are in charge of designing data storage systems that scale, perform, and are economical enough to satisfy the organization's requirements. They guarantee that the data is efficiently cleaned, converted, and loaded. Work together with data scientists and analysts to understand the needs for data and create effective data workflows.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

dbt Developer Hub

MARCH 23, 2023

While we can surely rely on that overview to validate the final refactored model with its legacy counterpart, it can be less useful while we are in the middle of the process of rebuilding a data workflow, where we need to track down which are exactly the columns that are causing incompatibility issues and what is wrong with them.

Data Warehouse 64

Data Warehouse Generalist SQL Coding 64

Databand.ai

AUGUST 30, 2023

Poor data quality can lead to incorrect or misleading insights, which can have significant consequences for an organization. DataOps tools help ensure data quality by providing features like data profiling, data validation, and data cleansing.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

dbt Developer Hub

APRIL 24, 2023

One example of a popular drag-and-drop transformation tool is Alteryx which allows business analysts to transform data by dragging and dropping operators in a canvas. In this sense, dbt may be a more suitable solution to building resilient and modular data pipelines due to its focus on data modeling.

BI 83

BI Data Workflow SQL Data Pipeline 83

Monte Carlo

MARCH 24, 2023

Sure, terabytes or even petabytes of data are involved, but generally it’s not the size of the data but everything surrounding the data–workflows, access permissions, layers of dependencies–that pose data migration risks. Here are some tips for mitigating some of the risks of data migration.

Data Warehouse 52

Data Warehouse AWS Data Governance Cloud 52

DataKitchen

APRIL 13, 2021

Piperr.io — Pre-built data pipelines across enterprise stakeholders, from IT to analytics, tech, data science and LoBs. Prefect Technologies — Open-source data engineering platform that builds, tests, and runs data workflows. Genie — Distributed big data orchestration service by Netflix.

Consulting 123

Consulting Machine Learning Data Science Data Pipeline 123