Data Engineering Podcast

SEPTEMBER 3, 2018

Using master data management to build a data catalog helps you answer these questions reliably and simplify the process of building your business intelligence reports. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

Data Management 100

Data Management Management Relational Database Business Intelligence 100

Snowflake

APRIL 8, 2024

In 2020, Snowflake announced a new global competition to recognize the work of early-stage startups building their apps — and their businesses — on Snowflake, offering up to $250,000 in investment as the top prize. SignalFlare.ai The judges will ask questions and deliberate live before naming the 2024 Grand Prize winner.

Pipeline-centric 126

Pipeline-centric Food Healthcare Unstructured Data 126

Data Engineering Podcast

JULY 27, 2020

In this episode he shares his approach to testing complex systems, the common challenges that are faced by engineers who build them, and why it is important to understand their limitations. I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help.

Systems 100

Systems Building Scala Java 100

The Modern Data Company

DECEMBER 29, 2022

In addition, business users can leverage drag-and-drop engineering to build pipelines without extensive coding. If the finance industry can leverage an operational layer, fractured data ecosystems will become a challenge of the past.

Banking 52

Banking Pipeline-centric Building Finance 52

The Modern Data Company

FEBRUARY 2, 2024

Ever wondered why building data-driven applications feels like an uphill battle? It’s not just you – turning raw data into something meaningful can be a real challenge. This approach doesn’t just solve existing problems; it paves the way for a new era of efficiency and effectiveness in data management.

Raw Data 75

Raw Data Utilities Government Data 75

Data Engineering Podcast

SEPTEMBER 30, 2018

Summary There are countless sources of data that are publicly available for use. The team at Enigma builds a knowledge graph for use in your own data projects. This is a great episode to listen to for ideas on how to organize a data engineering organization. How do you manage the software lifecycle for your ETL code?

Building 100

Building Data Lake AWS Machine Learning 100

Data Engineering Podcast

OCTOBER 30, 2022

In this episode he shares his experiences working with organizations to adopt analytics engineering patterns and the ways that Optimus and dbt were combined to let data analysts deliver insights without the roadblocks of complex pipeline management. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Engineering 100

Engineering MongoDB Scala MySQL 100

Christophe Blefari

JULY 22, 2023

How to chat with data in Snowflake using ChatGPT, dbt, and Streamlit — Less boring, obviously when you put ChatGPT and dbt in the same sentence it creates buzz instantly. This is an interesting demo of how you can quickly build a chat experience—using OpenAI—on top of you data models.

Data 130

Data Database Big Data Data Engineering 130

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. Can you describe how Planetscale is implemented?

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Christophe Blefari

JUNE 16, 2023

The power of pre-commit and SQLFluff —SQL is a query programming language used to retrieve information from data storages, and like any other programming language, you need to enforce checks at all times. Malloy's Near Term Roadmap — I've shared recently Malloy demo , which was awesome. but I missed it).

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform. Tired of deploying bad data? What does Riskified do?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

The Modern Data Company

JANUARY 31, 2023

Another year, another chance to learn more about the world of data. In 2023, The Modern Data Company (Modern) hopes to reach more companies and organizations with our data operating system, build incredible value from existing and upcoming data assets, and share insights into major shifts in what it means to be data-driven.

Retail 98

Retail Healthcare Data Architecture Data Pipeline 98

Data Engineering Podcast

JUNE 5, 2022

Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems.

Data Security 100

Data Security Metadata MongoDB Scala 100

The Modern Data Company

FEBRUARY 15, 2023

Maintaining a clear strategy for gathering, storing, and labeling data is challenging. Limited resources: Data management has always been resource intensive, but not all organizations can maintain a full data team. Without suitable resources for company-wide data management, it’s easier to fall behind.

Government 98

Government Data Governance Raw Data Data 98

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

DECEMBER 21, 2020

If you are starting down the path of implementing a data governance strategy then this episode will provide a great overview of what is involved. If you hand a book to a new data engineer, what wisdom would you add to it? What is data governance? What are some of the common blind spots when it comes to data governance?

Data Governance 130

Data Governance Government Data Lake Banking 130

Data Engineering Podcast

JUNE 12, 2022

Summary Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Data Engineering Podcast

JANUARY 4, 2021

Summary As more organizations are gaining experience with data management and incorporating analytics into their decision making, their next move is to adopt machine learning. As a result the feature store is becoming a required piece of the data platform. What types of data can feature stores manage?

Python 100

Python Data Lake Machine Learning Computer Science 100

Data Engineering Podcast

FEBRUARY 24, 2020

Summary Misaligned priorities across business units can lead to tensions that drive members of the organization to build data and analytics projects without the guidance or support of engineering or IT staff. The availability of cloud platforms and managed services makes this a viable option, but can lead to downstream challenges.

IT 100

IT Data Lake Data Pipeline Media 100

Data Engineering Podcast

AUGUST 10, 2020

Summary Event based data is a rich source of information for analytics, unless none of the event structures are consistent. The team at Iteratively are building a platform to manage the end to end flow of collaboration around what events are needed, how to structure the attributes, and how they are captured.

Data Collection 100

Data Collection Data Engineering Data Engineer Education 100

Data Engineering Podcast

JANUARY 11, 2021

In this episode he explains how the TerminusDB system is architected to provide a versioned graph storage engine that allows for branching and merging of data sets, how that opens up new possibilities for individuals and teams to work together on building new data repositories. When is TerminusDB the wrong choice?

PostgreSQL 100

PostgreSQL Python Data Lake Computer Science 100

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Data Engineering Podcast

SEPTEMBER 16, 2018

The default in most cases is Google Analytics, but this can be limiting when you wish to perform detailed analysis of the captured data. To address this problem, Alex Dean co-founded Snowplow Analytics to build an open source platform that gives you total control of your website traffic data.

Google Cloud 100

Google Cloud Hadoop Business Intelligence Consulting 100

Data Engineering Podcast

AUGUST 24, 2020

In this episode Mars Lan and Pardhu Gunnam explain how they designed the platform, how it integrates into their data platforms, and how it is being used to power data discovery and analytics at LinkedIn. If you hand a book to a new data engineer, what wisdom would you add to it? Who is the target audience for DataHub?

Metadata 100

Metadata Management Kafka Data Engineering 100

The Modern Data Company

FEBRUARY 26, 2024

With each new product launch and market expansion, the data architecture that once supported its growth now threatened to be its Achilles’ heel. Mixing metaphors aside, this startup knows that to sustain growth and outmaneuver competitors, it needs to evolve its approach to data management.

Architecture 52

Architecture Data Architecture Data Technology 52

Data Engineering Podcast

OCTOBER 9, 2018

This was a deep dive on how to build a successful company around a powerful platform, and how that platform simplifies operations for enterprise grade data management. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

PostgreSQL 100

PostgreSQL BI Data Warehouse Machine Learning 100

Data Engineering Podcast

SEPTEMBER 28, 2020

Summary Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. This was a fascinating conversation with an energetic and enthusiastic engineer and founder about the challenges and opportunities in the realm of streaming data. When is Red Panda the wrong choice?

Kafka 100

Kafka BI Big Data Data Engineering 100

Data Engineering Podcast

JULY 1, 2019

It was an interesting conversation about how he stress tested the Instaclustr managed service for benchmarking an application that has real-world utility. What were some of the most interesting or unexpected lessons that you learned in the process of building this anomaly detection system?

Kafka 100

Kafka Finance Media Architecture 100

Data Engineering Podcast

JULY 14, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 100

Machine Learning Media Cloud Data 100

Data Engineering Podcast

SEPTEMBER 23, 2018

He also explains which layers are useful for the different members of the business, and which pitfalls to look out for along the path to a mature and flexible data platform. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 9, 2018

They also explain some of the types of data that you can use with Chaos Search, how to load it into S3, and when you might want to choose it over Amazon Athena for our serverless data analysis. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

IT 100

IT PostgreSQL Scala AWS 100

Workfall

JULY 25, 2023

It acts as an intermediary between your Google Spreadsheet and the applications you build, enabling you to interact with your Google Sheets data programmatically. With Sheety API, you can perform CRUD (Create, Read, Update, Delete) operations on the data in your Google Sheets using simple HTTP requests.

Python 76

Python Programming Language Coding Accessible 76