Medium Data Engineering

SEPTEMBER 28, 2023

Apache Airflow is an open-source platform designed to automate, schedule, and orchestrate complex data workflows. Continue reading on Medium »

Data Workflow 52

Data Workflow Data Designing Data Engineering 52

Towards Data Science

AUGUST 22, 2023

Optimising big data workflows orchestration Continue reading on Towards Data Science »

AWS 80

AWS Data Workflow Data Science Big Data 80

Medium Data Engineering

FEBRUARY 9, 2023

Run Apache Airflow without having to manage the infrastructure Continue reading on Medium »

Cloud Storage 52

Cloud Storage Data Workflow Google Cloud Cloud 52

Medium Data Engineering

MARCH 15, 2023

Big data processing is a complex task that often requires specialized skills and infrastructure. AWS Wrangler is a powerful tool that… Continue reading on Medium »

AWS 40

AWS Big Data Data Workflow Data 40

Medium Data Engineering

SEPTEMBER 1, 2023

As a data or machine learning engineer, you often need to automate complex data workflows that involve ETL/ELT tasks or ML model training… Continue reading on Medium »

Architecture 52

Architecture Data Workflow Machine Learning Engineering 52

Medium Data Engineering

APRIL 23, 2023

Pre-requisites: Continue reading on Medium »

Data Workflow 52

Data Workflow Data Data Engineering Data Engineer 52

Medium Data Engineering

JUNE 16, 2023

Introduction: Data processing is a critical component of any modern organization’s data workflow. When it comes to processing data, two… Continue reading on Medium »

Data Process 52

Data Process Bytes Process Data 52

KDnuggets

JANUARY 4, 2023

These Python packages would improve your data workflow.

Python 104

Python Data Workflow Data Data Science 104

Medium Data Engineering

APRIL 11, 2023

Azure Data Factory is a cloud-based data integration service that allows users to create, schedule, and orchestrate data workflows. Continue reading on Medium »

Data 52

Data Data Workflow Data Integration Cloud 52

Medium Data Engineering

MARCH 27, 2023

Apache Airflow and Dagster are open-source platforms for managing data workflows. Choose Apache Airflow for dynamic task generation and… Continue reading on CodeX »

Data Workflow 52

Data Workflow Data Management Data Engineering 52

Medium Data Engineering

MARCH 10, 2023

Apache Airflow is an open-source platform used for orchestrating complex data workflows. It offers a powerful toolset for managing… Continue reading on Medium »

Data Workflow 52

Data Workflow Data Management Data Engineering 52

Hevo

FEBRUARY 28, 2023

As raw data comes in various forms and sizes, it is essential to have a proper system to handle big data. One of the significant challenges is referencing data points as the complexities increase in data workflows.

Raw Data 52

Raw Data Data Workflow Big Data Data 52

Medium Data Engineering

MAY 21, 2023

In the world of Data Engineering, the way we handle data workflows plays a crucial role in achieving real-time processing and improved… Continue reading on Medium »

Data Process 40

Data Process Process Data Workflow Data 40

Databand.ai

AUGUST 30, 2023

Data Orchestration Data orchestration refers to the coordination and management of data workflows, from data ingestion to data processing and analysis. DataOps tools should offer powerful data orchestration capabilities, allowing organizations to build, schedule, and monitor data workflows with ease.

Data Pipeline 52

Data Pipeline Data Ingestion Data Analytics Data Validation 52

The Pragmatic Engineer

NOVEMBER 16, 2022

7 November Domino Data Lab (data workflow platform, Series E). In Mexico, the company had a large initiative with private drivers and a fleet of Tesla cars. That operation is in talks of selling to other investors. Meta - 13% layoffs. Widely-reported, e.g. here. 25% of their workforce laid off. Verified.

Data Workflow 130

Data Workflow Software Engineer Software Engineering Data Science 130

dbt Developer Hub

APRIL 24, 2023

One example of a popular drag-and-drop transformation tool is Alteryx which allows business analysts to transform data by dragging and dropping operators in a canvas. In this sense, dbt may be a more suitable solution to building resilient and modular data pipelines due to its focus on data modeling.

SQL 83

SQL Data Workflow BI Data Pipeline 83

Preset

FEBRUARY 21, 2022

CrateDB is a distributed SQL database that excels at IoT and Time Series data workflows. In this post, we'll showcase how CrateDB and Superset can be used together.

SQL 52

SQL Data Workflow Database Data 52

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Architecture Data Storage 52

Data Engineering Podcast

SEPTEMBER 17, 2023

Can you describe the workflow for building autonomous linkages across data assets that are modelled as JSON-LD? What are the most interesting, innovative, or unexpected ways that you have seen JSON-LD used for data workflows? When is JSON-LD the wrong choice? When is JSON-LD the wrong choice?

Building 189

Building Data SQL Python 189

Data Engineering Podcast

AUGUST 28, 2022

What are the different concerns that need to be included in a stack that supports fully automated data workflows? There was recently an interesting article suggesting that the "left-to-right" approach to data workflows is backwards.

Data Engineering 100

Data Engineering Data Engineer Engineering Data 100

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. In this episode Satish Jayanthi explains how he is building a framework to allow enterprises to move quickly while maintaining guardrails for data workflows.

Data Warehouse 100

Data Warehouse Data Engineering Data Engineer SQL 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. As you can see, this is in the code part where you are building your data pipelines, a misnomer because this is an over simplification. The other benefit is you can also use parameters and build a generic workflows to be re-used.

Architecture 130

Architecture Data Technology Metadata 130

Monte Carlo

MAY 25, 2023

Benefits of Data Orchestration Data orchestration can yield significant benefits for organizations—from driving efficiency across the data team to improving relationships across business domains. Automating data workflows. Data quality issues introduced by faster data workflows.

Data 52

Data Data Pipeline Data Workflow Data Engineering 52

Data Engineering Weekly

DECEMBER 29, 2022

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. How happy are you with your data catalogs?

Data Warehouse 64

Data Warehouse Data Metadata Data Engineering 64

Knowledge Hut

SEPTEMBER 25, 2023

Data Engineer Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure. Work together with data scientists and analysts to understand the needs for data and create effective data workflows.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Christophe Blefari

SEPTEMBER 28, 2023

When it comes to incrementally change a data platform this is a bit different, you need to find what is going wrong and what could be improved. A lot of companies started with Postgres + dbt + Metabase, don't be ashamed.

Data 147

Data SQL Big Data Hadoop 147

Databand.ai

AUGUST 30, 2023

One key aspect of data orchestration is the automation of data pipeline tasks. By automating repetitive tasks, such as data extraction, transformation, and loading (ETL), organizations can streamline their data workflows and reduce the risk of human error.

Data Pipeline 52

Data Pipeline Data Governance Data Ingestion Government 52

Workfall

JULY 18, 2023

Additionally, they offer a way to keep your transformed data up to date without redoing the entire transformation process. By using DBT materializations, you can streamline your data workflows and focus on gaining insights rather than spending excessive time on repetitive calculations.

Data 78

Data Datasets Data Workflow Data Engineering 78

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows.

Data 130

Data Metadata Data Engineering Data Engineer 130

Data Engineering Podcast

NOVEMBER 14, 2021

data warehouse as a chokepoint vs quality checks on extract) What are the difficulties that engineers and stakeholders encounter when identifying and defining information that is necessary to identify issues in their workflows?

Data 100

Data Data Warehouse Data Engineering Data Engineer 100

Data Engineering Weekly

JANUARY 15, 2023

. 🎯 I defined the modern data stack sometime back as; @sarahmk125 MDS is a set of vendor tools that solve niche data problems (lineage, orchestration, quality) with the side effect of creating a disjointed data workflow that makes data folks lives more complicated.","username":"ananthdurai","name":"at-ananth-at-data-folks

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information.

Systems 100

Systems Data Data Warehouse Software Engineer 100

Data Engineering Podcast

MAY 22, 2022

With an eye to making data workflows more accessible to everyone in an organization Raj Bains and his team at Prophecy designed a powerful and extensible low-code platform that lets technical and non-technical users scale data flows without forcing everyone into the same layers of abstraction.

Data 100

Data Data Engineering Data Engineer SQL 100

Meltano

JULY 7, 2022

Those best practices, such as version control, a unified development environment, and continuous, built-in testing, can also be brought to data workflows. Now it’s Data’s Turn The data profession is about five years behind software engineering. Data is just very different from dev.

Software Engineer 52

Software Engineer Software Engineering BI Data Workflow 52

Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Data Engineering Podcast

OCTOBER 22, 2021

In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are already doing in your data warehouse modeling with dbt, and how they are focusing on bridging the divide between data teams and business teams and the requirements that they have for data workflows.

Engineering 100

Engineering Data Data Warehouse SQL 100

Data Engineering Podcast

OCTOBER 29, 2021

How does Narrator fit within the data workflows of an organization? How does Narrator fit within the data workflows of an organization? What are the challenges that you have seen organizations encounter when attempting to make analytics a self-serve capability? What are the use cases that you are focused on?

Data Warehouse 100

Data Warehouse Data Engineering Data Engineer Coding 100