Data Engineering Podcast

JULY 5, 2021

Deploying, managing, and scaling that orchestration can consume a large fraction of a data team’s energy so it is important to pick something that provides the power and flexibility that you need. What have been the pros and cons of working with your family on this project?

Systems 100

Systems Management Data Warehouse Programming Language 100

Knowledge Hut

DECEMBER 28, 2023

In this blog, we'll talk about intriguing and real-time sample Hadoop projects with source codes that can help you take your data analysis to the next level. Why Are Hadoop Projects So Important? The Apache Hadoop framework provides tools for efficient analysis, resource management, and parallel processing of datasets.

Hadoop 52

Hadoop Project Datasets Big Data 52

Data Engineering Podcast

MAY 6, 2019

Summary The database market continues to expand, offering systems that are suited to virtually every use case. In this episode Ryan Worl explains how it is architected, how to use it for your applications, and provides examples of system design patterns that can be built on top of it. Can you describe how FoundationDB is architected?

Systems 100

Systems MongoDB NoSQL Database 100

Knowledge Hut

NOVEMBER 2, 2023

As an AWS Cloud Practitioner with experience in delivering multiple AWS cloud practitioner projects, I vividly recall assisting a startup to prove the scalability of their AI solution on AWS during one of my early projects. In this dynamic profession, there is no alternative to project-based learning.

AWS 52

AWS Coding Cloud Project 52

Data Engineering Podcast

OCTOBER 15, 2023

Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Process 182

Process Building SQL BI 182

ProjectPro

FEBRUARY 16, 2023

With so much riding on the efficiency of ETL processes for data engineering teams, it is essential to take a deep dive into the complex world of ETL on AWS to take your data management to the next level. AWS refers to Amazon Web Service, the most widely used cloud computing system.

AWS 52

AWS Data Management ETL Tools Management 52

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Can you share your definition of "behavioral data" and how it is differentiated from other sources/types of data?

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

MARCH 18, 2023

He also explains why data security is distinct from application security and some methods for reducing the challenge of working across different data systems. From March 28-30th 2023, they'll play host to hundreds of attendees, 100 top speakers, and dozens of startups that are advancing data science, engineering and AI.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Knowledge Hut

SEPTEMBER 29, 2023

In the below blog, we are going to read more about some of these AWS Solution Architect projects. Why You Should Work on AWS Projects? The AWS projects can aid in the development of skills in cloud computing as well as other cutting-edge and critical technologies such as IoT, AI, and others. Who is an AWS Solution Architect?

AWS 52

AWS Project Coding Cloud Computing 52

Data Engineering Podcast

MARCH 27, 2022

Summary Data governance is a practice that requires a high degree of flexibility and collaboration at the organizational and technical levels. The growing prominence of cloud and hybrid environments in data management adds additional stress to an already complex endeavor.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

JUNE 12, 2022

In this episode Rod Christensen shares the story behind Aparavi and how you can use it to cut costs and gain value for the long tail of your unstructured data. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. With the increasing expecation for information to be instantly accessible, it drives the need for reliable change data capture. What do you have planned for the future of CDC at Fivetran?

Food 100

Food MongoDB Scala MySQL 100

Knowledge Hut

MAY 29, 2024

Alongside, top Business Intelligence and Visualization courses will support you in representing data through insightful visuals to support organizational goals. Considered as one of the best datasets for data visualization. It captures information on various events, emotions, and narratives reported in the media.

Datasets 52

Datasets Transportation Entertainment Media 52

Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Data Engineering Podcast

AUGUST 10, 2021

In this episode Mark Grover explains what he is building at Stemma, how it expands on the success of the Amundsen project, and why trust is the most important asset for data teams. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack.

IT 130

IT Building Data Warehouse Python 130

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. When is a C360 project the wrong choice?

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

MAY 10, 2021

Summary Building an API for real-time data is a challenging project. The team at Tinybird wants to make it easy to turn a continuous stream of data into a production ready API or data product. What are some of the custom tools or systems that you have built to help deal with them?

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. This is an ambitious and exciting project, so listen and try it out today. Start trusting your data with Monte Carlo today! No more scripts, just SQL.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

MAY 8, 2022

Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm.

Database 100

Database Data Lake BI Business Intelligence 100

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Data Engineering Podcast

FEBRUARY 17, 2020

Summary One of the biggest challenges in building reliable platforms for processing event pipelines is managing the underlying infrastructure. At Snowplow Analytics the complexity is compounded by the need to manage multiple instances of their platform across customer environments.

AWS 100

AWS Google Cloud Kafka Media 100

Data Engineering Podcast

MAY 1, 2022

In this episode Doron Porat and Liran Yogev explain how they arrived at their current architecture, the capabilities that they are optimizing for, and the complex process of identifying and evaluating new components to integrate into their systems. Can you describe what Yotpo is and the role that data plays in the organization?

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Data Engineering Podcast

APRIL 5, 2021

In this episode Prukalpa Sankar shares her experiences across multiple attempts at building a system that brings everyone onto the same page, ultimately bringing her to found Atlan. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

MAY 29, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. What are the factors (e.g.

Cloud 100

Cloud Data Warehouse Data Lake Data 100

Data Engineering Podcast

JUNE 8, 2021

Summary The way to build maintainable software and systems is through composition of individual pieces. One such component that has gone above and beyond its originally envisioned use case is BookKeeper, a distributed storage system that is optimized for durability and speed. What are some of the ways that BookKeeper is being used?

Data Warehouse 100

Data Warehouse Hadoop Metadata Architecture 100

Data Engineering Podcast

OCTOBER 30, 2022

It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus.

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

MARCH 8, 2021

Summary A majority of the time spent in data engineering is copying data between systems to make the information available for different purposes. This introduces challenges such as keeping information synchronized, managing schema evolution, building transformations to match the expectations of the destination systems.

IT 100

IT Data Warehouse MongoDB Kafka 100

ProjectPro

JULY 23, 2021

AWS (Amazon Web Services) is the world’s leading and widely used cloud platform, with over 200 fully featured services available from data centers worldwide. This blog presents some of the most unique and innovative AWS projects from beginner to advanced levels. Applications of AWS Projects Why should you work on AWS Projects?

AWS 52

AWS Project Amazon Web Services Cloud Computing 52

Data Engineering Podcast

JUNE 22, 2021

Summary The database is the core of any system because it holds the data that drives your entire experience. We spend countless hours designing the data model, updating engine versions, and tuning performance. Go to dataengineeringpodcast.com/census today to get a free 14-day trial. Can you describe how OtterTune is implemented?

Database 100

Database MySQL PostgreSQL Data Warehouse 100

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. What are the goals of the project? Can you describe what Milvus is and the story behind it?

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. How does this differ from "business as usual" in the data industry?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

SEPTEMBER 14, 2020

In this episode Dale Kim shares how Hazelcast is implemented, the use cases that it enables, and how it complements on-disk data management systems. If you hand a book to a new data engineer, what wisdom would you add to it? What are the benefits and tradeoffs of in-memory computation for data-intensive workloads?

Process 100

Process Unstructured Data Metadata Data Engineering 100

Data Engineering Podcast

AUGUST 19, 2019

Summary Managing big data projects at scale is a perennial problem, with a wide variety of solutions that have evolved over the past 20 years. One of the early entrants that predates Hadoop and has since been open sourced is the HPCC (High Performance Computing Cluster) system.

Big Data 100

Big Data Hadoop Data Lake Media 100

ProjectPro

OCTOBER 6, 2021

Flexera’s State of Cloud report highlighted that 41% of the survey respondents showed the most interest in using Google Cloud Platform for their future cloud computing projects. The benefits it offers start from data management and manipulation to machine learning tools on the GCP platform.

Google Cloud 52

Google Cloud Project Data Lake Healthcare 52

Cloudera

DECEMBER 19, 2023

Some of the primary operational problems highlighted at the PCN Government Innovation event include: Civil Government : A major challenge facing the civil government is the inefficient and cumbersome procurement process. Form an ethics review board to oversee AI projects and ensure they align with ethical values.

Government 94

Government Education Unstructured Data Datasets 94

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. What are your plans for Data Observability?

Metadata 100

Metadata BI Data Warehouse Government 100