Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Can you describe what your conception of a data contract is? Closing Announcements Thank you for listening!

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Data Engineering Podcast

DECEMBER 29, 2022

In this episode she shares the strategic and tactical elements of how to make more effective use of the technical and organizational resources that are available to you for getting work done with data. Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines.

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

NOVEMBER 16, 2020

Summary The core mission of data engineers is to provide the business with a way to ask and answer questions of their data. This often takes the form of business intelligence dashboards, machine learning models, or APIs on top of a cleaned and curated data set. What is the story behind the name?

Data Management 100

Data Management Management BI Business Intelligence 100

Data Engineering Podcast

JANUARY 15, 2023

In this episode Srujan Akula explains how the system is implemented and how you can start using it today with your existing data systems. Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Your flagship (only?)

Coding 130

Coding Building PostgreSQL Data Lake 130

Data Engineering Podcast

JANUARY 8, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data and analytics leaders, 2023 is your year to sharpen your leadership skills, refine your strategies and lead with purpose. Monte Carlo]([link] Struggling with broken pipelines? Missing data?

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Ascend.io

APRIL 28, 2023

Data pipelines are having a moment — at least, that is, within the data world. That’s because as more and more businesses are adopting a data-driven mindset, the movement of data into and within organizations has never been a bigger priority.

Data Pipeline 52

Data Pipeline Data Data Warehouse Data Engineering 52

Monte Carlo

AUGUST 25, 2023

Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider. So let’s get to the bottom of the big question: what kind of data storage layer will provide the strongest foundation for your data platform?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Data Engineering Weekly

MAY 26, 2024

link] Meta: Composable data management at Meta Meta writes about its transition to a composable data management system to improve interoperability, reusability, and engineering efficiency. The idea will infact open flexibility in programming data over the SQL-based data warehouse systems.

Data Engineering 57

Data Engineering Data Engineer Engineering Google Cloud 57

Data Engineering Podcast

NOVEMBER 27, 2022

In this episode Wes McKinney shares the ways that Arrow and its related projects are improving the efficiency of data systems and driving their next stage of evolution. Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines.

Data Process 100

Data Process Process Metadata Business Intelligence 100

Data Engineering Podcast

APRIL 26, 2021

Summary The reason for collecting, cleaning, and organizing data is to make it usable by the organization. One of the most common and widely used methods of access is through a business intelligence dashboard. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Data Engineering Podcast

OCTOBER 22, 2021

Summary The market for business intelligence has been going through an evolutionary shift in recent years. Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. No more scripts, just SQL. What are the main goals of the project?

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

NOVEMBER 4, 2021

Summary The precursor to widespread adoption of cloud data warehouses was the creation of customer data platforms. Acting as a centralized repository of information about how your customers interact with your organization they drove a wave of analytics about how to improve products based on actual usage data.

Data Warehouse 100

Data Warehouse Business Intelligence ETL System Data Lake 100

Data Engineering Podcast

NOVEMBER 20, 2022

He explains the constraints that he and his team are faced with and the various challenges that they have overcome to build useful data products on top of a legacy platform where they don’t control the end-to-end systems. Just connect it to your database/data warehouse/data lakehouse/whatever you’re using and let them do the rest.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

DECEMBER 25, 2022

In order to condense that acquired knowledge into a format that is useful to everyone Scott Hirleman turns the tables in this episode and asks Tobias about the tactical and strategic aspects of his experiences applying those lessons to the work of building a data platform from scratch. Closing Announcements Thank you for listening!

Building 100

Building Metadata Business Intelligence Data Lake 100

Towards Data Science

NOVEMBER 4, 2023

I’d like to discuss some popular Data engineering questions: Modern data engineering (DE). Does your DE work well enough to fuel advanced data pipelines and Business intelligence (BI)? Are your data pipelines efficient? Data warehouse exmaple. What is it?

Data Engineering 82

Data Engineering Data Engineer Engineering BI 82

Data Engineering Podcast

APRIL 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse Data Lake Data Pipeline BI 100

Data Engineering Podcast

DECEMBER 19, 2021

In this episode Pardhu Gunnam and Mars Lan explain how they have designed the architecture and user experience to allow everyone to collaborate on the data lifecycle and provide opportunities for automation and extensible workflows. Start trusting your data with Monte Carlo today! No more scripts, just SQL.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

JANUARY 1, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. No more scripts, just SQL.

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

Data Engineering Podcast

DECEMBER 11, 2021

In this episode he explains how it can be used to build rich and useful profiles of your users, the system architecture that powers it, and some of the ways that it is being integrated into an organization’s broader data ecosystem. Start trusting your data with Monte Carlo today! No more scripts, just SQL.

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Data Engineering Podcast

JANUARY 15, 2022

Summary Applications of data have grown well beyond the venerable business intelligence dashboards that organizations have relied on for decades. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. Who is your target audience?

Engineering 100

Engineering Electronics ETL Tools Data Pipeline 100

Data Engineering Podcast

DECEMBER 4, 2021

The whole process of hiring is an important organizational skill to cultivate and this is an interesting exploration of the specific challenges involved in finding data professionals. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

NOVEMBER 27, 2021

If you are having trouble answering questions for your business with the data that you generate and collect internally, then it is definitely worthwhile to explore the information available from external sources. Start trusting your data with Monte Carlo today! No more scripts, just SQL.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

JULY 10, 2022

They also discuss how they have established a guild system for training and supporting data professionals in the organization. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Tired of deploying bad data? Need to automate data pipelines with less red tape?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

JUNE 26, 2022

In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Data Engineering Podcast

DECEMBER 26, 2021

In this episode Maxime Beauchemin returns to revisit what it means to be a data engineer and how the role has changed over the past 5 years. In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Warehouse 100

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 20, 2021

Summary The focus of the past few years has been to consolidate all of the organization’s data into a cloud data warehouse. As a result there have been a number of trends in data that take advantage of the warehouse as a single focal point. Start trusting your data with Monte Carlo today!

Data Warehouse 100

Data Warehouse Data Lake Business Intelligence Data Engineering 100

Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. Start trusting your data with Monte Carlo today! Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads?

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

NOVEMBER 20, 2021

In this episode he shares the journey that he and his team at Elementl have taken to understand the state of the ecosystem and how they can provide a foundational layer for a holistic data platform. Start trusting your data with Monte Carlo today! The data you’re looking for is already in your data warehouse and BI tools.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

JUNE 12, 2022

In this episode she shares her thoughts and insights on how to be intentional about establishing your own data team. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Unstruk is the DataOps platform for your unstructured data.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Cloudera

AUGUST 30, 2022

Agencies are plagued by a wide range of data formats and storage environments—legacy systems, databases, on-premises applications, citizen access portals, innumerable sensors and devices, and more—that all contribute to a siloed ecosystem and the data management challenge. . Modern data architectures. Forrester ).

Data Architecture 78

Data Architecture Architecture Data Lake NoSQL 78

Data Engineering Podcast

SEPTEMBER 19, 2021

This brings with it a unique set of challenges for data collection, data management, and analytical capabilities. In this episode Jillian Rowe shares her experience of working in the field and supporting teams of scientists and analysts with the data infrastructure that they need to get their work done.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Lake 100

Knowledge Hut

DECEMBER 21, 2023

In this article, we will understand the promising data engineer career outlook and what it takes to succeed in this role. What is Data Engineering? Data engineering is the method to collect, process, validate and store data. It involves building and maintaining data pipelines, databases, and data warehouses.

Data Engineering 52

Data Engineering Data Engineer Engineering MongoDB 52