Data Engineering Weekly

NOVEMBER 13, 2022

It moved from the speculation to the data engineers understanding the benefit of it and asking when we can get the implementation soon. I met many data leaders about Data Contracts, my project Schemata, and how the extended version we are building can help them create high-quality data.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Monte Carlo

MAY 17, 2023

One successful CMS company uses these custom monitors to keep track of a process that loads data to BigQuery that is gathered by a cron on the server farm. Because the sheer amount of data takes several days to process, they designed a query that checks rows loaded on a day over day and week over week basis to make sure the cron is on track.

Data 52

Data Data Pipeline Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

One successful CMS company uses these custom monitors to keep track of a process that loads data to BigQuery that is gathered by a cron on the server farm. Because the sheer amount of data takes several days to process, they designed a query that checks rows loaded on a day over day and week over week basis to make sure the cron is on track.

Data Pipeline 52

Data Pipeline Data Data Engineering Data Engineer 52