Data Engineering Podcast

JUNE 26, 2022

In this episode Malcolm Hawker shares his years of experience working in this domain to explore the combination of technical and social skills that are necessary to make an MDM project successful both at the outset and over the long term. Random data doesn’t do it — and production data is not safe (or legal) for developers to use.

Data Management 100

Data Management Management MongoDB Scala 100

Towards Data Science

OCTOBER 7, 2023

This is considerably more streamlined than the traditional approach of a Site Reliability Engineering team manually investigating and triaging incidents. Teams can use standalone vector databases like Pinecone or Zilliz , or use vector embedding capabilities in their existing data storage solutions like Databricks and Snowflake.

Unstructured Data 75

Unstructured Data Finance SQL Media 75

Monte Carlo

OCTOBER 4, 2023

This is considerably more streamlined than the traditional approach of a Site Reliability Engineering team manually investigating and triaging incidents. Teams can use standalone vector databases like Pinecone or Zilliz , or use vector embedding capabilities in their existing data storage solutions like Databricks and Snowflake.

Unstructured Data 72

Unstructured Data Finance SQL Media 72

Monte Carlo

MARCH 29, 2022

You may not have heard the term data downtime, but I’m willing to bet you’ve experienced it and the cost of bad data firsthand. Duplicate tables wreaking havoc in your Snowflake warehouse, all titled some variation of “Mikes_Table_GOOD-V3.”? Urgent ping from your CEO about “missing data” in a critical report? No harm, no foul.

Media 52

Media Google Cloud Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Another data team noticed they had multiple queries timing out in Snowflake. The issue was with a Lambda that took data from S3 and pushed it into Snowflake. They originally built this specific function in a way that stored the Snowflake credentials in code. Over one weekend they had 21 different tasks that were hanging.

Data 52

Data Data Pipeline Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Another data team noticed they had multiple queries timing out in Snowflake. The issue was with a Lambda that took data from S3 and pushed it into Snowflake. They originally built this specific function in a way that stored the Snowflake credentials in code. Over one weekend they had 21 different tasks that were hanging.

Data Pipeline 52

Data Pipeline Data Data Engineering Data Engineer 52

phData: Data Engineering

JANUARY 3, 2022

You’re likely familiar with the term “Big Data” — and the scale of this market is continuously growing. The data that is generated and collected from your business likely has compliance requirements such as SOC2 or Personally Identifiable Information (PII) that you’re legally required to protect. This is a 42.2%

Data Engineering 52

Data Engineering Data Engineer Engineering Data Governance 52