Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Monte Carlo

APRIL 24, 2023

In this article, we’ll dive deep into the data presentation layers of the data stack to consider how scale impacts our build versus buy decisions, and how we can thoughtfully apply our five considerations at various points in our platform’s maturity to find the right mix of components for our organizations unique business needs.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Monte Carlo

JUNE 14, 2023

Why is data pipeline architecture important? Databricks – Databricks, the Apache Spark-as-a-service platform, has pioneered the data lakehouse, giving users the options to leverage both structured and unstructured data and offers the low-cost storage features of a data lake.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Snowflake

MARCH 27, 2023

It seems everyone has a handful of such shapes in their raw data, and in the past they had to fix those shapes outside of Snowflake before ingesting them. Introducing CARTO Workflows Snowflake’s powerful data ingestion and transformation features help many data engineers and analysts who prefer SQL.

Generalist 52

Generalist Raw Data Business Analyst SQL 52

AltexSoft

MARCH 14, 2023

Data storage The tools mentioned in the previous section are instrumental in moving data to a centralized location for storage, usually, a cloud data warehouse, although data lakes are also a popular option. But this distinction has been blurred with the era of cloud data warehouses.

IT 59

IT Data Warehouse Data Governance Data Lake 59