Medium Data Engineering

FEBRUARY 9, 2023

Run Apache Airflow without having to manage the infrastructure Continue reading on Medium »

Cloud Storage 52

Cloud Storage Data Workflow Google Cloud Cloud 52

Medium Data Engineering

APRIL 23, 2023

Pre-requisites: Continue reading on Medium »

Data Workflow 52

Data Workflow Data Data Engineering Data Engineer 52

KDnuggets

JANUARY 4, 2023

These Python packages would improve your data workflow.

Python 115

Python Data Workflow Data Data Science 115

Medium Data Engineering

APRIL 11, 2023

Azure Data Factory is a cloud-based data integration service that allows users to create, schedule, and orchestrate data workflows. Continue reading on Medium »

Data 52

Data Data Workflow Data Integration Cloud 52

Medium Data Engineering

MAY 21, 2023

In the world of Data Engineering, the way we handle data workflows plays a crucial role in achieving real-time processing and improved… Continue reading on Medium »

Data Process 40

Data Process Process Data Workflow Data 40

Medium Data Engineering

MARCH 27, 2023

Apache Airflow and Dagster are open-source platforms for managing data workflows. Choose Apache Airflow for dynamic task generation and… Continue reading on CodeX »

Data Workflow 52

Data Workflow Data Management Data Engineering 52

Medium Data Engineering

MARCH 10, 2023

Apache Airflow is an open-source platform used for orchestrating complex data workflows. It offers a powerful toolset for managing… Continue reading on Medium »

Data Workflow 52

Data Workflow Data Management Data Engineering 52

Hevo

FEBRUARY 28, 2023

As raw data comes in various forms and sizes, it is essential to have a proper system to handle big data. One of the significant challenges is referencing data points as the complexities increase in data workflows.

Raw Data 52

Raw Data Data Workflow Big Data Data 52

dbt Developer Hub

APRIL 24, 2023

One example of a popular drag-and-drop transformation tool is Alteryx which allows business analysts to transform data by dragging and dropping operators in a canvas. In this sense, dbt may be a more suitable solution to building resilient and modular data pipelines due to its focus on data modeling.

SQL 83

SQL Data Workflow BI Data Pipeline 83

The Pragmatic Engineer

NOVEMBER 16, 2022

7 November Domino Data Lab (data workflow platform, Series E). In Mexico, the company had a large initiative with private drivers and a fleet of Tesla cars. That operation is in talks of selling to other investors. Meta - 13% layoffs. Widely-reported, e.g. here. 25% of their workforce laid off. Verified.

Data Workflow 130

Data Workflow Software Engineer Software Engineering Data Science 130

Monte Carlo

MAY 25, 2023

Benefits of Data Orchestration Data orchestration can yield significant benefits for organizations—from driving efficiency across the data team to improving relationships across business domains. Automating data workflows. Data quality issues introduced by faster data workflows.

Data 52

Data Data Pipeline Data Workflow Data Engineering 52

Preset

FEBRUARY 21, 2022

CrateDB is a distributed SQL database that excels at IoT and Time Series data workflows. In this post, we'll showcase how CrateDB and Superset can be used together.

SQL 52

SQL Data Workflow Database Data 52

Data Engineering Podcast

AUGUST 28, 2022

What are the different concerns that need to be included in a stack that supports fully automated data workflows? There was recently an interesting article suggesting that the "left-to-right" approach to data workflows is backwards.

Data Engineering 100

Data Engineering Data Engineer Engineering Data 100

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. In this episode Satish Jayanthi explains how he is building a framework to allow enterprises to move quickly while maintaining guardrails for data workflows.

Data Warehouse 100

Data Warehouse Data Engineering Data Engineer SQL 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. As you can see, this is in the code part where you are building your data pipelines, a misnomer because this is an over simplification. The other benefit is you can also use parameters and build a generic workflows to be re-used.

Architecture 130

Architecture Data Technology Metadata 130

Data Engineering Weekly

DECEMBER 29, 2022

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. How happy are you with your data catalogs?

Data Warehouse 64

Data Warehouse Data Metadata Data Engineering 64

Monte Carlo

APRIL 28, 2023

Data Engineers : Data engineers who work with data pipelines and ETL processes use DuckDB to enhance the performance of their data processing tasks. They can leverage DuckDB’s efficient query execution and integration with data formats like Parquet to optimize their data workflows.

Data Warehouse 52

Data Warehouse Database Data Pipeline Datasets 52

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows.

Data 130

Data Metadata Data Engineering Data Engineer 130

Data Engineering Weekly

JANUARY 15, 2023

. 🎯 I defined the modern data stack sometime back as; @sarahmk125 MDS is a set of vendor tools that solve niche data problems (lineage, orchestration, quality) with the side effect of creating a disjointed data workflow that makes data folks lives more complicated.","username":"ananthdurai","name":"at-ananth-at-data-folks

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Data Engineering Podcast

NOVEMBER 14, 2021

data warehouse as a chokepoint vs quality checks on extract) What are the difficulties that engineers and stakeholders encounter when identifying and defining information that is necessary to identify issues in their workflows?

Data 100

Data Data Warehouse Data Engineering Data Engineer 100

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information.

Data 100

Data Systems Data Warehouse Software Engineer 100

Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Meltano

JULY 7, 2022

Those best practices, such as version control, a unified development environment, and continuous, built-in testing, can also be brought to data workflows. Now it’s Data’s Turn The data profession is about five years behind software engineering. Data is just very different from dev.

Software Engineer 52

Software Engineer Software Engineering BI Data Workflow 52

Data Engineering Weekly

MARCH 11, 2023

[link] Data Engineering Weekly Data Catalog - A Broken Promise Data catalogs are the most expensive data integration systems you never intended to build.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Data Engineering Podcast

MAY 22, 2022

With an eye to making data workflows more accessible to everyone in an organization Raj Bains and his team at Prophecy designed a powerful and extensible low-code platform that lets technical and non-technical users scale data flows without forcing everyone into the same layers of abstraction.

Data 100

Data Data Engineering Data Engineer SQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

What are the contributing factors that lead to fragmentation of visibility for data workflows at different stages? What are the sources of information that you use to build a cohesive view of an organization’s data assets? What are the lifecycle stages of a data asset that are most often overlooked or un-maintained?

Metadata 100

Metadata Data Building Data Engineering 100

Data Engineering Podcast

OCTOBER 29, 2021

How does Narrator fit within the data workflows of an organization? How does Narrator fit within the data workflows of an organization? What are the challenges that you have seen organizations encounter when attempting to make analytics a self-serve capability? What are the use cases that you are focused on?

Data Warehouse 100

Data Warehouse Data Engineering Data Engineer Coding 100

Data Engineering Podcast

OCTOBER 22, 2021

In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are already doing in your data warehouse modeling with dbt, and how they are focusing on bridging the divide between data teams and business teams and the requirements that they have for data workflows.

Engineering 100

Engineering Data Data Warehouse SQL 100

Data Engineering Podcast

JULY 31, 2022

Your host is Tobias Macey and today I’m interviewing Ernie Ostic about Manta, an automated data lineage service for managing visibility and quality of your data workflows Interview Introduction How did you get involved in the area of data management? Can you describe what Manta is and the story behind it?

Metadata 100

Metadata Data Machine Learning Unstructured Data 100

Data Engineering Podcast

JANUARY 1, 2022

Can you start by discussing some of the anti-patterns that you have encountered in data warehouse naming conventions and how it relates to the modeling approach? star/snowflake schema, data vault, etc.) What are some of the types of contracts that can, and should, be defined and enforced in data workflows?

Data Warehouse 100

Data Warehouse Data Data Engineering Data Engineer 100

Meltano

FEBRUARY 22, 2023

Testing a data application is similar to testing any software application in many ways, just with a strong focus on testing data-related issues. But testing problems like failing data workflows, mismatches in data reconciliation after ETL, and data quality issues means that you’re not only testing the code but also the data itself.

Data 52

Data Data Workflow Database Coding 52

Meltano

FEBRUARY 22, 2023

Testing a data application is similar to testing any software application in many ways, just with a strong focus on testing data-related issues. But testing problems like failing data workflows, mismatches in data reconciliation after ETL, and data quality issues means that you’re not only testing the code but also the data itself.

Data 52

Data Data Workflow Database Coding 52

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility.

Data Integration 100

Data Integration Data Data Engineering Data Engineer 100

Data Engineering Podcast

NOVEMBER 27, 2021

Your host is Tobias Macey and today I’m interviewing Mark Hookey about Demyst Data, a platform for operationalizing external data Interview Introduction How did you get involved in the area of data management? What are some examples of data sets that an organization might want to use in their analytics?

Data 100

Data Data Warehouse SQL Data Engineering 100

Snowflake

MARCH 27, 2023

The Flow Diagram tool provides a wide range of analytical and data processing components, including various data sources and sinks, spatial joins and filters, clustering, isoline creation, and tools to leverage spatial indexes like H3 and Quadbin. Workflows automates not only geospatial processes, but other data workflows as well.

SQL 52

SQL Generalist Data Ingestion Raw Data 52

Data Engineering Podcast

JULY 17, 2022

Shipyard]([link] Shipyard is an orchestration platform that helps data teams build out solid data operations from the get-go by connecting data tools and streamlining data workflows.

Data Management 130

Data Management Data Metadata Management 130

Data Engineering Podcast

JANUARY 30, 2022

Pandas is a tool that spans data processing and data science. What are some of the ways that data engineers should think about writing their code to make it accessible to data scientists for supporting collaboration across data workflows? Pandas is a tool that spans data processing and data science.

Data Engineering 100

Data Engineering Data Engineer Engineering Data 100