Data Engineering Podcast

NOVEMBER 6, 2022

Summary A lot of the work that goes into data engineering is trying to make sense of the "data exhaust" from other applications and services. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

DECEMBER 18, 2022

Summary One of the reasons that data work is so challenging is because no single person or team owns the entire process. This introduces friction in the process of collecting, processing, and using data. In order to reduce the potential for broken pipelines some teams have started to adopt the idea of data contracts.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

NOVEMBER 6, 2022

Summary Despite the best efforts of data engineers, data is as messy as the real world. Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically required custom development and training machine learning models.

MongoDB 130

MongoDB Scala MySQL Machine Learning 130

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

DECEMBER 28, 2022

Summary With all of the messaging about treating data as a product it is becoming difficult to know what that even means. Vishal Singh is the head of products at Starburst which means that he has to spend all of his time thinking and talking about the details of product thinking and its application to data.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

Data Engineering Podcast

DECEMBER 4, 2022

Summary The term "real-time data" brings with it a combination of excitement, uncertainty, and skepticism. In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

AUGUST 21, 2022

Summary Data has permeated every aspect of our lives and the products that we interact with. In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences.

Lambda Architecture 100

Lambda Architecture MongoDB Scala MySQL 100

Data Engineering Podcast

DECEMBER 25, 2022

Summary Encryption and security are critical elements in data analytics and machine learning applications. We have well developed protocols and practices around data that is at rest and in motion, but security around data in use is still severely lacking.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

OCTOBER 23, 2022

Applying those same practices to data can prove challenging due to the number of systems that need to be included to implement a complete feature. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Project 130

Project Metadata MongoDB MySQL 130

Data Engineering Podcast

NOVEMBER 20, 2022

Sifflet is a platform that brings your entire data stack into focus to improve the reliability of your data assets and empower collaboration across your teams. In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

AUGUST 28, 2022

For data engineers, this is a monumental undertaking. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Atlan is the metadata hub for your data ecosystem.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Podcast

DECEMBER 4, 2022

Summary Business intelligence is the foremost application of data in organizations of all sizes. Zing Data is building a mobile native platform for business intelligence. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Data Engineering Podcast

NOVEMBER 27, 2022

Summary The most expensive part of working with massive data sets is the work of retrieving and processing the files that contain the raw information. FeatureBase (formerly Pilosa) avoids that overhead by converting the data into bitmaps. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 100

Data Lake Data Warehouse MongoDB Data Pipeline 100

Data Engineering Podcast

NOVEMBER 27, 2022

Summary The data ecosystem has been growing rapidly, with new communities joining and bringing their preferred programming languages to the mix. This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. Atlan is the metadata hub for your data ecosystem. Missing data?

Data Process 100

Data Process Process Metadata Business Intelligence 100

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.

Metadata 130

Metadata MongoDB Scala MySQL 130

Data Engineering Podcast

JUNE 19, 2022

Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

OCTOBER 30, 2022

In its current incarnation it has become a ubiquitous need for analytics and opportunities to answer questions with data. In this episode Amir Orad discusses the Sisense platform and how it facilitates the embedding of analytics and data insights in every aspect of organizational and end-user experiences.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

AUGUST 21, 2022

Summary The position of Chief Data Officer (CDO) is relatively new in the business world and has not been universally adopted. In this episode Tracy Daniels, CDO of Truist, shares her journey into the position, her responsibilities, and her relationship to the data professionals in her organization.

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. In this episode Inbar Yogev and Lior Winner share the journey that they and their teams at Riskified have been on for their data platform. They also discuss how they have established a guild system for training and supporting data professionals in the organization.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

DECEMBER 29, 2022

Summary Making effective use of data requires proper context around the information that is being used. Rehgan Avon co-founded AlignAI to help address this challenge through a more purposeful platform designed to collect and distribute the knowledge of how and why data is used in a business. Struggling with broken pipelines?

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Regardless of how data is being used, it is critical that the information is trusted. The practice of data reliability engineering has gained momentum recently to address that question. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Building 100

Building Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

Data Engineering Podcast

JULY 31, 2022

Summary Exploratory data analysis works best when the feedback loop is fast and iterative. The Arkouda project is a Python interface built on top of the Chapel compiler to bring back those interactive speeds for exploratory analysis on horizontally scalable compute that parallelizes operations on large volumes of data.

Data Analysis 100

Data Analysis MongoDB Scala Algorithm 100

Data Engineering Podcast

SEPTEMBER 18, 2022

Summary There is a constant tension in business data between growing siloes, and breaking them down. Even when a tool is designed to integrate information as a guard against data isolation, it can easily become a silo of its own, where you have to make a point of using it to seek out information.

Metadata 100

Metadata Building MongoDB MySQL 100

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

JUNE 5, 2022

Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

JULY 3, 2022

Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. does exactly that.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

DECEMBER 18, 2022

Summary The data ecosystem has seen a constant flurry of activity for the past several years, and it shows no signs of slowing down. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 100

Data Lake Data Warehouse Data Pipeline MongoDB 100

Knowledge Hut

DECEMBER 21, 2023

Data Science is the world's most rapidly growing sector and data engineers are at the forefront. In this article, we will understand the promising data engineer career outlook and what it takes to succeed in this role. What is Data Engineering? What are the Data Engineer Career Opportunities?

Data Engineering 52

Data Engineering Data Engineer Engineering MongoDB 52

Knowledge Hut

MAY 23, 2024

Data science has become the fastest-growing field and a buzzword that every industry knows and leverages. This vast stream of interdisciplinary domains deals with data in different ways. It helps companies understand data and obtain meaningful insights from it. Each of these tools comes with a set of some of these usages.

Data Science 52

Data Science MongoDB Programming Language Hadoop 52

Data Engineering Podcast

JULY 3, 2022

Summary The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform architectures for managing data and analytical workflows. Atlan is the metadata hub for your data ecosystem.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

AUGUST 13, 2022

Summary Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking for. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Metadata 100

Metadata MongoDB Scala MySQL 100

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

AUGUST 6, 2022

Summary Data mesh is a frequent topic of conversation in the data community, with many debates about how and when to employ this architectural pattern. The team at AgileLab have first-hand experience helping large enterprise organizations evaluate and implement their own data mesh strategies.

Metadata 100

Metadata MongoDB MySQL Business Intelligence 100

Data Engineering Podcast

AUGUST 28, 2022

Summary AirBnB pioneered a number of the organizational practices that have become the goal of modern data teams. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 31, 2022

Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100