Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Knowledge Hut

SEPTEMBER 25, 2023

Microsoft Azure is a cloud computing platform that gives businesses fantastic services. This demonstrates how in-demand Microsoft Certified Data Engineers are becoming. To start a successful career in cloud computing, you must have a solid conceptual foundation.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Knowledge Hut

MARCH 28, 2024

Role Level: Intermediate Responsibilities Design and develop big data solutions using Azure services like Azure HDInsight, Azure Databricks, and Azure Data Lake Storage. Implement data ingestion, processing, and analysis pipelines for large-scale data sets.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Knowledge Hut

SEPTEMBER 29, 2023

Why Should You Get an Azure Data Engineer Certification? Becoming an Azure data engineer allows you to seamlessly blend the roles of a data analyst and a data scientist. One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Monte Carlo

JUNE 14, 2023

Why is data pipeline architecture important? Some organizations choose to still leverage an ETL pattern in the cloud, particularly for production pipelines where data contracts can help reduce data downtime. Singer – An open source tool for moving data from a source to a destination.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Ascend.io

MAY 19, 2023

In the data world, this disruption manifested in the form of cloud computing with technologies such as Redshift, Snowflake, and Spark. In our case, data ingestion, transformation, orchestration, reverse ETL, and observability. This is the modern data stack as we know it today. Sounds familiar?

Data Pipeline 52

Data Pipeline Data Engineering Data Engineer Media 52