Data Engineering Weekly

APRIL 2, 2023

Meta: Presto - A Decade of SQL Analytics at Meta Presto and Kafka are the two systems that greatly impacted data infrastructure in the last decade. The model only determines the degree to which that maximum quality is realized; in a sense, the model is a lossy compiler for the data.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DataKitchen

NOVEMBER 4, 2021

It’s too hard to change our IT data product. Can we create high-quality data in an “answer-ready” format that can address many scenarios, all with minimal keyboarding? . “I I get cut off at the knees from a data perspective, and I am getting handed a sandwich of sorts and not a good one!”. The DataOps Advantage .

Process 98

Process Data Process Pharmaceutical Data Lake 98

Maxime Beauchemin

JANUARY 20, 2017

The modern data warehouse is a more public institution than it was historically, welcoming data scientists, analysts, and software engineers to partake in its construction and operation. Data is simply too centric to the company’s activity to have limitation around what roles can manage its flow.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Ripple Engineering

JANUARY 29, 2024

  A lack of a centralized system makes building a single source of high-quality data difficult. Ripple Data Consumers query the data from the lake storage using the SQL strategy. It can lead to expensive, slow, and unmaintainable systems.

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Today’s LLMs can already process enormous amounts of unstructured data, automating much of the monotonous work of data science. Can I see the pipeline? Can I see the data source?’”

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52