François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. It will be illustrated with our technical choices and the services we are using in the Google Cloud Platform. There are certainly many other ways to do the same with another technologies.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Knowledge Hut

MARCH 28, 2024

Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data. Implemented and managed data storage solutions using Azure services like Azure SQL Database , Azure Data Lake Storage, and Azure Cosmos DB.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Hepta Analytics

FEBRUARY 14, 2022

This week, we got to think about our data ingestion design. We looked at the following: How do we ingest – ETL vs ELT Where do we store the data – Data lake vs data warehouse Which tool to we use to ingest – cronjob vs workflow engine NOTE : This weeks task requires good internet speed and good compute.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Monte Carlo

JUNE 14, 2023

The modern data stack era , roughly 2017 to present data, saw the widespread adoption of cloud computing and modern data repositories that decoupled storage from compute such as data warehouses, data lakes, and data lakehouses.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

AltexSoft

NOVEMBER 7, 2022

DevOps tasks — for example, creating scheduled backups and restoring data from them. Airflow is especially useful for orchestrating Big Data workflows. Airflow is not a data processing tool by itself but rather an instrument to manage multiple components of data processing. When Airflow won’t work.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

phData: Data Engineering

JANUARY 3, 2022

This commonly introduces: Database or Data Warehouse API/EDI Integrations ETL software Business intelligence tooling By leveraging off-the-shelf tooling, your company separates disciplines by technology. Instead of manually working with the data, software, and data products, extract the data to unlock the ability to generate insights.

IT 52

IT AWS Software Engineer Software Engineering 52