Data Engineering Podcast

JUNE 5, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. And don’t forget to thank them for their continued support of this show!

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

OCTOBER 9, 2022

You wake up to a Slack message from your CEO, who’s upset because the company’s revenue dashboard is broken. Summary For any business that wants to stay in operation, the most important thing they can do is understand their customers. And don’t forget to thank them for their continued support of this show!

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 3, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. And don’t forget to thank them for their continued support of this show!

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

MARCH 29, 2021

There are many aspects of data quality management and it’s always a treat to learn from people who are dedicating their time and energy to solving it for everyone. In this episode Maarten Masschelein and Tom Baeyens share the work they are doing at Soda to bring everyone on board to make your data clean and reliable.

Management 100

Management Data Warehouse BI Data Pipeline 100

dbt Developer Hub

JANUARY 14, 2023

After an initial round of transformation (described in the next section), the deduplicated data is loaded into BigQuery where I can query it from the Google Sheets client and manually review the deduplication. For this I sent data to Slack via a dlt decorator that posts the error on failure and the metadata on success.

Data Ingestion 52

Data Ingestion Google Cloud Python Cloud 52

Striim

JANUARY 4, 2023

Companies need to analyze large volumes of datasets, leading to an increase in data producers and consumers within their IT infrastructures. These companies collect data from production applications and B2B SaaS tools (e.g., Mailchimp). This data makes its way into a data repository, like a data warehouse (e.g., change data capture)?

PostgreSQL 52

PostgreSQL Data Warehouse Data Lake Data 52

Monte Carlo

DECEMBER 13, 2023

Over the last five years, data observability has leveled up from industry buzzword to a must-have element of every data stack. With data observability, teams can identify and proactively resolve incidents of data downtime — moments when data is partial, erroneous, missing, or otherwise inaccurate. The outcome?

BI 52

BI Data Ingestion SQL Data Warehouse 52

Data Engineering Podcast

AUGUST 5, 2019

And for your machine learning workloads, they just announced dedicated CPU instances. Summary Data is only valuable if you use it for something, and the first step is knowing that it is available. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

dbt Developer Hub

APRIL 30, 2023

My next gig was in consulting where I bootstrapped my way into data engineering and had to learn the whole gamut below. I was using this thing called audit command language (ACL) to automate debits equal credits for accounting analytics (yes, it’s as tedious as it sounds). It was late in the AM. I’m thinking about 2 am. I feel like a.5x

Data Engineering 64

Data Engineering Data Engineer Engineering Finance 64

Monte Carlo

SEPTEMBER 20, 2022

With Prefect leveraging BigQuery for its data warehouse, Fivetran for ETL operations, dbt open source for data modeling, Looker for data exploration and visualization, and their own Prefect flows for internal processes, achieving end-to-end data reliability was no small task. But a growing company means growing data needs.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Monte Carlo

APRIL 27, 2023

Sigma Workbook Pages will automatically surface as lineage nodes and catalog entries once users have connected their Monte Carlo environment to Snowflake, BigQuery, Redshift, or Athena. Each interconnected layer can introduce and be impacted by poor data quality. What’s worse, we weren’t catching the issues on our own.”

BI 52

BI Business Intelligence Metadata Data Warehouse 52

Monte Carlo

MAY 24, 2022

billion on cloud databases such as Snowflake, Databricks, and BigQuery in 2021. PagerDuty protected their customer success machine learning models from drift. Last year, BigQuery , Snowflake , and Databricks annual revenues each neared or exceeded $1b, and in 2021, they continue to grow at astounding rates.

High Quality Data 52

High Quality Data Data Lake Media Data 52

Monte Carlo

MAY 31, 2022

Not only does the data inform critical internal decision making, but it is also public facing via key features such as machine learning powered video recommendations and a key part of their value proposition to their enterprise customers. We’re actually expanding our machine learning teams and going more in that direction.

BI 52

BI Data Warehouse Unstructured Data Data Schemas 52

Christophe Blefari

MARCH 2, 2024

I'll give a talk—in French—about how to put in production machine learning at a small scale. Before I forget I've appeared on The Joe Reis Show , we chatted with Joe about data engineering teaching, why it is hard and about generative AI that will change education for ever. Mistral perdant.

Data 162

Data Python Database BI 162

dbt Developer Hub

NOVEMBER 28, 2021

Let’s export log events into BigQuery. “I They were often written in naked python scripts that only ran a SQL query + wrote data to BigQuery. More fundamentally, I think it’s important to call out that Airflow + dbt are spiritually aligned in purpose. In a Google Cloud Storage bucket. Let’s use GCP’s logging alerts to send emails.

Google Cloud 52

Google Cloud SQL Cloud Consulting 52

Databand.ai

DECEMBER 13, 2022

Whether you’re a data engineering pro looking to stay up to date on the latest trends or new to the space and want to learn more, following the right leaders and joining the right conversations can make all the difference when it comes to plugging into the data engineering community. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Monte Carlo

APRIL 12, 2022

Data teams love the idea of automating data engineering processes in principle. After all, who doesn’t want to move faster and eliminate the time consuming, boring aspects of their job? But even time-strapped, technically savvy engineers will sometimes squirm when the suggestion is made to automate a specific task. We’ve felt it ourselves.

Data Engineering 52

Data Engineering Data Engineer Process Engineering 52

Christophe Blefari

MAY 12, 2023

In Minds of Data I'll met people from the data ecosystem in order to learn more about them. I hope you will enjoy it as much as me. Let's start with with my newly created podcast Minds of Data. I really liked Benoit and Samy presentation about Cloud Composer —Managed Airflow on GCP. for IT security or medicine).

Data 130

Data Data Storage SQL Coding 130

Data Engineering Podcast

MAY 8, 2022

From being an early engineer on the Dremel project, to helping launch and manage BigQuery, on to helping enterprises adopt Google’s data products he learned all of the critical details of how to run services used by data platform teams. Now he is the consumer of many of the tools that his work inspired.

Google Cloud 100

Google Cloud Hadoop SQL Software Engineer 100

dbt Developer Hub

NOVEMBER 28, 2021

Most organizations spend at least some of their time contributing to an open source project. 100% of them, though, depend in some way on the output of open source communities. In fact we all do. It's your entire day, even if you aren't a software engineer. But there are tactical uses for this data as well.

Raw Data 52

Raw Data Metadata Datasets Database 52

Data Engineering Weekly

MAY 28, 2023

Editor’s Note: DEW featured in AirByte’s State of the Data & Slack’s usage of Kafka DEW has been recognized as the number one individually run data newsletter in the industry, according to the latest AirByte poll ! Sign up free to test out the tool today. I'm more than happy to collaborate and help the community.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Podcast

JULY 30, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Go to dataengineeringpodcast.com/97things today to get your copy!

BI 100

BI SQL Metadata Kafka 100

Data Engineering Podcast

MAY 29, 2022

By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more.

Architecture 100

Architecture Database Data Architecture PostgreSQL 100

Data Engineering Podcast

FEBRUARY 27, 2022

In this episode Tobias Macey, the host of the show, reflects on his plans for building a data platform and what he has learned from running the podcast that is influencing his choices. Summary Building a data platform is a complex journey that requires a significant amount of planning to do well.

Designing 100

Designing Metadata Data Lake Relational Database 100

Data Engineering Podcast

SEPTEMBER 19, 2021

Summary Building, scaling, and maintaining the operational components of a machine learning workflow are all hard problems. Add the work of creating the model itself, and it’s not surprising that a majority of companies that could greatly benefit from machine learning have yet to either put it into production or see the value.

Machine Learning 100

Machine Learning Data Warehouse Banking Metadata 100

Data Engineering Podcast

JUNE 11, 2021

He shares some of the common patterns for building pipelines to power business intelligence dashboards, machine learning applications, and data warehouses. Now they offer the technologies that they run internally to external users of their cloud platform. And don’t forget to thank them for their continued support of this show!

Google Cloud 100

Google Cloud Cloud Big Data Ecosystem BI 100

Data Engineering Podcast

APRIL 10, 2022

By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more.

Data Integration 100

Data Integration SQL Data Engineering Data Engineer 100

Monte Carlo

JULY 26, 2023

One lesson we learned was not to jump on market opportunities without focusing on the tech. When we first started acquiring companies we saw that this app team would want to go with BigQuery or this team would want to go with Redshift. Some of the data system integration projects after an acquisition can extend for 18 to 24 months.”

Machine Learning 52

Machine Learning Media Data Warehouse AWS 52

Data Engineering Podcast

SEPTEMBER 1, 2021

By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more.

Data Lake 100

Data Lake Cloud AWS SQL 100

Monte Carlo

APRIL 4, 2023

Monte Carlo, which maintains rich integrations with data warehouses and lakes like Snowflake, Databricks, Google BigQuery, and Amazon Redshift, business intelligence tools like Looker, Tableau, and Mode, and ETL tools like Airflow and dbt, extends data quality coverage at ingestion with our native Fivetran integration.

ETL Tools 52

ETL Tools Data Pipeline Business Intelligence Data Warehouse 52

Monte Carlo

APRIL 4, 2023

Monte Carlo, which maintains rich integrations with data warehouses and lakes like Snowflake, Databricks, Google BigQuery, and Amazon Redshift, business intelligence tools like Looker, Tableau, and Mode, and ETL tools like Airflow and dbt, extends data quality coverage at ingestion with our native Fivetran integration.

ETL Tools 52

ETL Tools Data Pipeline Business Intelligence Data Warehouse 52

dbt Developer Hub

OCTOBER 17, 2022

For years working in data and analytics engineering roles, I treasured the daily camaraderie sharing a small office space with talented folks using a range of tools - from analysts using SQL and Excel to data scientists working in Python. But logistically, it just couldn’t be done in a scalable way.

SQL 52

SQL Python Programming Language Data Analysis 52

AltexSoft

NOVEMBER 7, 2022

If this is not a big deal, read on to learn more about Airflow concepts and architecture which, in turn, predefine its pros and cons. It still remains a leading workflow management tool adopted by thousands of companies, from tech giants to startups. How data engineering works. How data engineering works. What is Apache Airflow?

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Monte Carlo

NOVEMBER 15, 2022

Data teams that run on top of AWS and GCP can reap the benefits of Power BI to generate fast insights leveraging data from Snowflake, AWS Redshift, BigQuery, Databricks, and other leading data warehouse and lake providers. Read on to learn more about our new integration.

BI 52

BI Data Warehouse Telecommunication Business Intelligence 52

Monte Carlo

SEPTEMBER 23, 2021

It may even be magnified by a few early morning phone calls and urgent Slack pings. But, if said data is a report used by your Chief Revenue Officer to determine next quarter’s forecast, chances are this data will make a very, very large sound. Sound familiar? Fortunately, data doesn’t have to be silent. What is data observability?

BI 52

BI Data Metadata Data Pipeline 52

Monte Carlo

APRIL 21, 2021

In 2019, when Daniel joined the company, Resident had been using BigQuery to analyze data about ad performance, tracking marketing spend, cost-of-goods, and sales. “We It doesn’t have to be this way. Direct-to-consumer mattress brands may not be the first category that comes to mind when discussing data-driven companies.

Google Cloud 40

Google Cloud Data Retail BI 40

Databand.ai

DECEMBER 29, 2022

Meetups, Slack) Frequency of posting and engagement with content Diversity of representation As a reminder, this is not a forced ranked or exhaustive list. So who should you be following? When it comes to LinkedIn (and beyond) here are our recommendations for the top 25 data strategy influencers and content creators to keep tabs on.

BI 52

BI Consulting Data Science Data Governance 52