phData: Data Engineering

APRIL 4, 2023

Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format. Amazon S3 is an object storage service from Amazon Web Services (AWS) that offers industry-leading scalability, data availability, security, and performance.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. Can you describe what Cuelake is and the story behind it?

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Podcast

MARCH 27, 2022

Privacera is an enterprise grade solution for cloud and hybrid data governance built on top of the robust and battle tested Apache Ranger project. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

AUGUST 31, 2020

I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Knowledge Hut

NOVEMBER 2, 2023

As an AWS Cloud Practitioner with experience in delivering multiple AWS cloud practitioner projects, I vividly recall assisting a startup to prove the scalability of their AI solution on AWS during one of my early projects. In this dynamic profession, there is no alternative to project-based learning.

AWS 52

AWS Coding Cloud Project 52

Data Engineering Podcast

FEBRUARY 19, 2023

Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. If you're fed up with the 'Modern Data Stack', give TimeXtender a try.

IT 147

IT Data Lake Metadata Data Warehouse 147

Ascend.io

APRIL 22, 2024

As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Ascend has always been at the forefront of innovation, and with Project Inception, we’re setting a new standard.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

MAY 22, 2022

Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Knowledge Hut

NOVEMBER 2, 2023

Azure Data engineering projects are complicated and require careful planning and effective team participation for a successful completion. While many technologies are available to help data engineers streamline their workflows and guarantee that each aspect meets its objectives, ensuring that everything works properly takes time.

Data Engineering 52

Data Engineering Data Engineer Project Coding 52

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares the design of the project and how it fits into your development practices. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.

Systems 100

Systems Metadata Building MongoDB 100

Knowledge Hut

JUNE 26, 2023

If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. Get ready to delve into fascinating data engineering project concepts and explore a world of exciting data engineering projects in this article.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

MARCH 16, 2020

In this episode Adam Kocoloski shares the history of the project, how it works under the hood, and how the new design will improve the project for our new era of computation. This was an interesting conversation about the challenges of maintaining a large and mission critical project and the work being done to evolve it.

Building 100

Building Data Warehouse NoSQL Data Lake 100

Data Engineering Weekly

JULY 7, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Hudi seems to be a de facto choice for CDC data lake features. Notion migrated the insert heavy workload from Snowflake to Hudi.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

dbt Developer Hub

OCTOBER 3, 2024

Apache Iceberg is a high-performance open table format developed for modern data lakes. It was designed for large-scale datasets, and within the project, there are many ways to interact with it. To have this conversation, we need to start with the same foundational understanding of Iceberg. And this work doesn’t stop at Iceberg.

Metadata 86

Metadata Data Lake Data Storage Accessibility 86

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. or any other destination you choose.

Database 100

Database MySQL Data Lake MongoDB 100

ProjectPro

OCTOBER 6, 2021

Flexera’s State of Cloud report highlighted that 41% of the survey respondents showed the most interest in using Google Cloud Platform for their future cloud computing projects. Beginner Level GCP Sample Projects Ideas 1. Google Cloud Platform is an online vendor of multiple cloud services which can be used publicly.

Google Cloud 52

Google Cloud Project Data Lake Healthcare 52

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. Dremio has its ancestry in the Drill project.

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

MAY 19, 2019

Summary In recent years the traditional approach to building data warehouses has shifted from transforming records before loading, to transforming them afterwards. The data build tool (dbt) is designed to bring battle tested engineering practices to your analytics pipelines. When is it the wrong choice?

Data Analytics 100

Data Analytics Building Engineering Data Warehouse 100

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. RudderStack helps you build a customer data platform on your warehouse or data lake. or any other destination you choose.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 30, 2018

Summary There are countless sources of data that are publicly available for use. The team at Enigma builds a knowledge graph for use in your own data projects. This is a great episode to listen to for ideas on how to organize a data engineering organization.

Building 100

Building Data Lake AWS Machine Learning 100

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Edureka

MAY 31, 2023

With its seamless integration with various Azure services, it has become a preferred choice for both industrial projects and streaming OTT platforms. It supports a wide variety of input data sources, including Azure Event Hubs, Azure IoT Hub, Azure Blob Storage, and others. Joins: This allows you to combine data from different sources.

Project 52

Project Manufacturing Retail Data Lake 52

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. Can you describe what Zingg is and the story behind it?

MongoDB 130

MongoDB Scala MySQL Machine Learning 130

Data Engineering Podcast

SEPTEMBER 7, 2020

In this episode he explains how it is designed to allow for querying and combining data where it resides, the use cases that such an architecture unlocks, and the innovative ways that it is being employed at companies across the world. What are the tradeoffs of using Presto on top of a data lake vs a vertically integrated warehouse solution?

Architecture 100

Architecture Data Architecture SQL Engineering 100

Cloudera

JUNE 6, 2024

Recently, Cloudera, alongside OCBC, were named winners in the“ Best Big Data and Analytics Infrastructure Implementation ” category at The Asian Banker’s Financial Technology Innovation Awards 2024. With adoption growing, organizations need to understand that without good quality and trusted data, there is no way to implement AI.

Banking 87

Banking Data Lake Finance Data Governance 87

ProjectPro

JULY 22, 2016

For the same cost, organizations can now store 50 times as much data as in a Hadoop data lake than in a data warehouse. Data lake is gaining momentum across various organizations and everyone wants to know how to implement a data lake and why.

Data Lake 40

Data Lake Data Warehouse Hadoop Unstructured Data 40

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services.

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

MAY 8, 2022

In this episode Jon Herke shares how TigerGraph customers are taking advantage of those capabilities to achieve meaningful discoveries in their fields, the utilities that it provides for modeling and managing your connected data, and some of his own experiences working with the platform before joining the company.

Database 100

Database Data Lake BI Business Intelligence 100

Data Engineering Podcast

OCTOBER 30, 2022

It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus.

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 4) Building Data Products and why should you? Part 1: Why did we need to build our own SIEM?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Podcast

JUNE 12, 2022

In this episode Rod Christensen shares the story behind Aparavi and how you can use it to cut costs and gain value for the long tail of your unstructured data. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Data Engineering Podcast

APRIL 16, 2022

In this episode Demetrios Brinkmann and David Aponte share their perspectives on this rapidly changing space and what they have learned from their work building the MLOps community through blog posts, podcasts, and discussion forums. What are some of the ways that requirements become bespoke to a given organization or project?

Data Engineering 100

Data Engineering Data Engineer Engineering IT 100

Data Engineering Podcast

FEBRUARY 9, 2020

If you’re struggling with unwieldy dimensional models, slow moving projects, or challenges integrating new data sources then listen in on this conversation and then give data vault a try for yourself. What are some of the foundational skills and knowledge that are necessary for effective modeling of data warehouses?

Data Lake 100

Data Lake Data Warehouse Hadoop NoSQL 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. When is a C360 project the wrong choice?

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database 100

Database MySQL Cloud MongoDB 100