Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 3) DataOPS at AstraZeneca The AstraZeneca team talks about data ops best practices internally established and what worked and what didn’t work!!!

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services.

Technology 130

Technology Architecture Google Cloud Metadata 130

Knowledge Hut

SEPTEMBER 26, 2023

Key Features of Azure Synapse Here are some of the key features of Azure Synapse: Cloud Data Service: Azure Synapse operates as a cloud-native service, residing within the Microsoft Azure cloud ecosystem. This cloud-centric approach ensures scalability, flexibility, and cost-efficiency for your data workloads.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

DataKitchen

NOVEMBER 4, 2021

Cloud computing has made it much easier to integrate data sets, but that’s only the beginning. Creating a data lake has become much easier, but that’s only ten percent of the job of delivering analytics to users. It often takes months to progress from a data lake to the final delivery of insights.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. Open question: how to seed data in a staging environment? Test system with A/A test. Be adaptable.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

RandomTrees

NOVEMBER 29, 2023

So, a data vault model forms the core for a data vault approach and is a data modeling design pattern used to build a data warehouse for organizations adopting enterprise-scalable analytics as and for its solutions. The data from many data bases are sent to the data warehouse through the ETL processes.

Data Warehouse 52

Data Warehouse Data Lake Database-centric Data Cleanse 52