Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Monte Carlo

APRIL 24, 2023

Data ingestion When we think about the flow of data in a pipeline, data ingestion is where the data first enters our platform. This data ingestion process can be accomplished by either querying the source directly, using upstream systems to publish events, or some combination of the two.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Monte Carlo

JUNE 14, 2023

Databricks – Databricks, the Apache Spark-as-a-service platform, has pioneered the data lakehouse, giving users the options to leverage both structured and unstructured data and offers the low-cost storage features of a data lake. Singer – An open source tool for moving data from a source to a destination.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

AltexSoft

SEPTEMBER 21, 2023

The Elastic Stacks Elasticsearch is integral within analytics stacks, collaborating seamlessly with other tools developed by Elastic to manage the entire data workflow — from ingestion to visualization. This means that Elasticsearch can be easily integrated into different modern data stacks.

Engineering 52

Engineering NoSQL Programming Language Java 52

phData: Data Engineering

JANUARY 3, 2022

This commonly introduces: Database or Data Warehouse API/EDI Integrations ETL software Business intelligence tooling By leveraging off-the-shelf tooling, your company separates disciplines by technology. This helps drive requirements and determines the right validation at the right time for the data.

IT 52

IT AWS Software Engineer Software Engineering 52

AltexSoft

MARCH 14, 2023

Data uses Here comes why you need this whole MDS thing in the first place — the data use component, or how the data is actually utilized. There are two main areas of use within this component: the first is data analytics and business intelligence and the second is data science.

IT 59

IT Data Warehouse Data Governance Data Lake 59