François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How can we interoperate between the data domains ? How do we govern all these data products and domains ? It will be illustrated with our technical choices and the services we are using in the Google Cloud Platform.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Hepta Analytics

FEBRUARY 14, 2022

Disadvantages of a data lake are: Can easily become a data swamp data has no versioning Same data with incompatible schemas is a problem without versioning Has no metadata associated It is difficult to join the data Data warehouse stores processed data, mostly structured data.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Monte Carlo

SEPTEMBER 20, 2022

Here’s how Prefect , Series B startup and creator of the popular data orchestration tool, harnessed the power of data observability to preserve headcount, improve data quality and reduce time to detection and resolution for data incidents. This left Dylan’s team with a gap to fill.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

AltexSoft

NOVEMBER 7, 2022

DevOps tasks — for example, creating scheduled backups and restoring data from them. Airflow is especially useful for orchestrating Big Data workflows. Airflow is not a data processing tool by itself but rather an instrument to manage multiple components of data processing. Metadata database. Since the 2.0

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Monte Carlo

JUNE 14, 2023

Airbyte – An open source platform that easily allows you to sync data from applications. Data streaming ingestion solutions include: Apache Kafka – Confluent is the vendor that supports Kafka, the open source event streaming platform to handle streaming analytics and data ingestion.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

AltexSoft

SEPTEMBER 21, 2023

Accessible via a unified API, these new features enhance search relevance and are available on Elastic Cloud. The Elastic Stacks Elasticsearch is integral within analytics stacks, collaborating seamlessly with other tools developed by Elastic to manage the entire data workflow — from ingestion to visualization.

Engineering 52

Engineering NoSQL Programming Language Java 52