Data Engineering Podcast

JUNE 3, 2019

He also discusses how the abstractions provided by DataCoral allows his data scientists to remain productive without requiring dedicated data engineers. If you are either considering how to build a data pipeline or debating whether to migrate your existing ETL to a service this is definitely worth listening to for some perspective.

Media 100

Media Data Pipeline Machine Learning Data Science 100

Data Engineering Podcast

DECEMBER 2, 2019

Summary The financial industry has long been driven by data, requiring a mature and robust capacity for discovering and integrating valuable sources of information. This was a great conversation for an inside look at how to build and maintain a data driven culture.

Data Engineering 100

Data Engineering Data Engineer Engineering Machine Learning 100

Data Engineering Podcast

AUGUST 12, 2019

Summary The extract and load pattern of data replication is the most commonly needed process in data engineering workflows. Fivetran is a platform that does the hard work for you and replicates information from your source systems into whichever data warehouse you use. Integration of multiple data sources (e.g.

Media 100

Media Data Warehouse Data Big Data 100

Data Engineering Podcast

DECEMBER 8, 2019

Summary Data warehouses have gone through many transformations, from standard relational databases on powerful hardware, to column oriented storage engines, to the current generation of cloud-native analytical engines. If you are evaluating your options for building or migrating a data platform, then this is definitely worth a listen.

Data Warehouse 100

Data Warehouse Cloud AWS Relational Database 100

Data Engineering Podcast

NOVEMBER 26, 2019

As they scaled the volume of customers and data they began running into the limitations of their initial architecture. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Data Engineering Podcast

DECEMBER 30, 2019

In order to support their customers, they need to capture, process, and analyze massive amounts of timeseries data with a high degree of uptime and reliability. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council.

Process 100

Process Building Hadoop Java 100

Data Engineering Podcast

MARCH 25, 2019

Summary Data integration is one of the most challenging aspects of any data platform, especially as the variety of data sources and formats grow. The CluedIn team experienced this issue first-hand in their previous roles, leading them to build a business aimed at building a managed data fabric for the enterprise.

Building 100

Building Data Lake Machine Learning Kafka 100

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? This was a fascinating conversation with someone who has spent his entire career working on simplifying complex data problems. Managing and auditing access to your servers and databases is a problem that grows in difficulty alongside the growth of your teams.

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

Data Engineering Podcast

FEBRUARY 18, 2019

Summary Distributed storage systems are the foundational layer of any big data stack. Alluxio is a distributed virtual filesystem which integrates with multiple persistent storage systems to provide a scalable, in-memory storage layer for scaling computational workloads independent of the size of your data.

Systems 100

Systems Java Media Algorithm 100

Data Engineering Podcast

AUGUST 26, 2019

Summary Data engineers are responsible for building tools and platforms to power the workflows of other members of the business. Benn Stancil is the chief analyst at Mode Analytics and in this episode he explains the set of considerations and requirements that data analysts need in their tools and.

Building 100

Building Data Analytics Media Data Engineering 100

Data Engineering Podcast

JULY 28, 2019

Summary The ETL pattern that has become commonplace for integrating data from multiple sources has proven useful, but complex to maintain. For a small number of sources it is a tractable problem, but as the overall complexity of the data ecosystem continues to expand it may be time to identify new ways to tame the deluge of information.

Data Integration 100

Data Integration Metadata Media Architecture 100

Data Engineering Podcast

FEBRUARY 24, 2019

As data engineers we are responsible for building and managing the platforms that power these models. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Deep Learning 100

Deep Learning Data Engineering Data Engineer Engineering 100

Data Engineering Podcast

APRIL 15, 2019

Summary Database indexes are critical to ensure fast lookups of your data, but they are inherently tied to the database engine. Pilosa is rewriting that equation by providing a flexible, scalable, performant engine for building an index of your data to enable high-speed aggregate analysis.

Big Data 100

Big Data Relational Database Media Database 100

Data Engineering Podcast

MARCH 31, 2019

However, many of them are and as data engineers we can help to keep our projects on the path to success. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Project 100

Project IT Media Data Warehouse 100

Data Engineering Podcast

OCTOBER 7, 2019

Summary The process of exposing your data through a SQL interface has many possible pathways, each with their own complications and tradeoffs. One of the recent options is Rockset, a serverless platform for fast SQL analytics on semi-structured and structured data. Visit Datacoral.com today to find out more.

Structured Data 100

Structured Data Cloud SQL Programming Language 100

Data Engineering Podcast

NOVEMBER 11, 2019

Summary The practice of data management is one that requires technical acumen, but there are also many policy and regulatory issues that inform and influence the design of our systems. This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC.

Designing 100

Designing Data Pipeline Programming Language Data 100

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Data Engineering Podcast

AUGUST 5, 2019

Summary Data is only valuable if you use it for something, and the first step is knowing that it is available. As organizations grow and data sources proliferate it becomes difficult to keep track of everything, particularly for analysts and data scientists who are not involved with the collection and management of that information.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Data Engineering Podcast

MARCH 4, 2019

In order to gain a full understanding of what your users are doing and how best to serve them you may need to send data to multiple services, each with their own tracking code or APIs. Managing and auditing access to your servers and databases is a problem that grows in difficulty alongside the growth of your teams.

Google Cloud 100

Google Cloud Kafka Media Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 30, 2019

Summary Building an end-to-end data pipeline for your machine learning projects is a complex task, made more difficult by the variety of ways that you can structure it. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, and Data Council.

Data Pipeline 100

Data Pipeline Machine Learning Media Data 100

Data Engineering Podcast

NOVEMBER 18, 2019

Summary With the constant evolution of technology for data management it can seem impossible to make an informed decision about whether to build a data warehouse, or a data lake, or just leave your data wherever it currently rests. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo!

Data Lake 100

Data Lake Scala Data Warehouse Hadoop 100

Data Engineering Podcast

OCTOBER 21, 2019

In this episode Dipti Borkar explains how the emerging category of data orchestration tools fills this need, some of the existing projects that fit in this space, and some of the ways that they can work together to simplify projects such as cloud migration and hybrid cloud environments.

Cloud 100

Cloud Data Lake Hadoop Programming Language 100

Data Engineering Podcast

JUNE 24, 2019

Summary Building a data platform that works equally well for data engineering and data science is a task that requires familiarity with the needs of both roles. Data engineering platforms have a strong focus on stateful execution and tasks that are strictly ordered based on dependency graphs.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Science 100

Data Engineering Podcast

APRIL 28, 2019

However, managing the database layer is still a separate concern. He also covers some of the challenges of managing stateful services in Kubernetes and how the fast pace of the community has contributed to the evolution of KubeDB. Summary Kubernetes is a driving force in the renaissance around deploying and running applications.

Database 100

Database PostgreSQL MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 28, 2019

Summary Despite the fact that businesses have relied on useful and accurate data to succeed for decades now, the state of the art for obtaining and maintaining that information still leaves much to be desired. In an effort to create a better abstraction for building data applications Nick Schrock created Dagster.

Building 100

Building Data Pipeline Programming Language Metadata 100

Data Engineering Podcast

NOVEMBER 4, 2019

Summary As data engineers the health of our pipelines is our highest priority. Unfortunately, there are countless ways that our dataflows can break or degrade that have nothing to do with the business logic or data transformations that we write and maintain. and Facebook, scaling from terabytes to petabytes of analytic data.

Programming Language 100

Programming Language Kafka Media Data Engineering 100

Data Engineering Podcast

APRIL 22, 2019

Summary One of the biggest challenges for any business trying to grow and reach customers globally is how to scale their data storage. One of the unique aspects of Fauna that is worth drawing attention to is the first class support for temporality that simplifies querying of historical states of the data.

Database 100

Database Cloud NoSQL Scala 100

Data Engineering Podcast

MAY 26, 2019

Summary Some problems in data are well defined and benefit from a ready-made set of tools. For everything else, there’s Pachyderm, the platform for data science that is built to scale. Alluxio is an open source, distributed data orchestration layer that makes it easier to scale your compute and your storage independently.

Data Science 100

Data Science Data Pipeline Data Kafka 100

Data Engineering Podcast

JULY 1, 2019

It was an interesting conversation about how he stress tested the Instaclustr managed service for benchmarking an application that has real-world utility. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Kafka 100

Kafka Finance Media Architecture 100

Data Engineering Podcast

SEPTEMBER 22, 2019

Summary Object storage is quickly becoming the unifying layer for data intensive applications and analytics. Modern, cloud oriented data warehouses and data lakes both rely on the durability and ease of use that it provides. Interview Introduction How did you get involved in the area of data management?

AWS 100

AWS Google Cloud Cloud Storage Data Lake 100

Data Engineering Podcast

MAY 6, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Systems 100

Systems MongoDB NoSQL Database 100

Data Engineering Podcast

SEPTEMBER 18, 2019

Summary The conventional approach to analytics involves collecting large amounts of data that can be cleaned, followed by a separate step for analysis and interpretation. Unfortunately this strategy is not viable for handling real-time, real-world use cases such as traffic management or supply chain logistics.

Data Lake 100

Data Lake Hadoop BI Kafka 100

Data Engineering Podcast

MAY 19, 2019

Summary In recent years the traditional approach to building data warehouses has shifted from transforming records before loading, to transforming them afterwards. The data build tool (dbt) is designed to bring battle tested engineering practices to your analytics pipelines.

Data Analytics 100

Data Analytics Building Engineering Data Warehouse 100

Data Engineering Podcast

JULY 14, 2019

Summary Successful machine learning and artificial intelligence projects require large volumes of data that is properly labelled. The challenge is that most data is not clean and well annotated, requiring a scalable data labeling process. But, a new way of integrating data and improving streams has evolved.

Machine Learning 100

Machine Learning Media Cloud Data 100

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Podcast

DECEMBER 22, 2019

Data warehouses are optimized for batched writes and complex analytical queries. Between those use cases there are varying levels of support for fast reads on quickly changing data. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council.

SQL 100

SQL Building Engineering Java 100

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. It was interesting to learn about some of the custom data types and performance optimizations that are included. Integrating data across the enterprise has been around for decades – so have the techniques to do it.

Data Warehouse 100

Data Warehouse MySQL Data Lake Hadoop 100