Data Engineering Podcast

MARCH 17, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. The Machine Learning Podcast helps you go from idea to production with machine learning. Want to see Starburst in action? Don't forget to check out our other shows.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. The Machine Learning Podcast helps you go from idea to production with machine learning.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 3, 2023

Andrei Tserakhau has dedicated his careeer to this problem, and in this episode he shares the lessons that he has learned and the work he is doing on his most recent data transfer system at DoubleCloud. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

NOVEMBER 5, 2023

In this episode Lukas Fittl shares some hard-won wisdom about the causes and solution of many performance bottlenecks and the work that he is doing to shine some light on PostgreSQL to make it easier to understand how to keep it running smoothly. The Machine Learning Podcast helps you go from idea to production with machine learning.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 19, 2023

In this episode they share their hard-won wisdom about how to build and scale your dbt projects. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. While it is easy to adopt, there are many potential pitfalls.

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 12, 2023

In this episode Eran Yahav shares the journey that he has taken in building this product and the ways that it enhances the ability of humans to get their work done, and when the humans have to adapt to the tool. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

DECEMBER 10, 2023

In this episode he shares what it is, how it works, and how you can start using it today to get notified when the critical metrics in your business aren't quite right. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Weekly

MARCH 17, 2024

link] Jarkko Moilanen: Exploring the Frontier of Data Products - Seeking Insights on Emerging Standards Data products are becoming a central part of the data pipeline strategy, bringing a new approach to building data as an asset. Please share your thoughts in the comments. Data quality cost = (avg.

Data Engineering 70

Data Engineering Data Engineer Engineering Data Governance 70

Christophe Blefari

MAY 12, 2023

They shared good practices on how to manage Composer in the cloud, things like: Use the same configuration for staging and prod Use a secret manager to manage your Airflow connections Use IAM restrictions in the DAGs bucket Use operators and define the company policy around it Define clear policies to govern your Airflow Also Airflow 2.6

Data 130

Data Data Storage SQL Coding 130

Data Engineering Podcast

JULY 23, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack Modern data teams are using Hex to 10x their data impact. This allows data teams to both dive deep to find insights and then share their work in an easy-to-read format to the whole org. Don't forget to check out our other shows.

Building 130

Building SQL Machine Learning Data Ingestion 130

Data Engineering Podcast

FEBRUARY 11, 2023

In this episode Aneesh Karve shares the journey that Quilt has taken to provide an approachable interface for working with versioned data in S3 that empowers everyone to collaborate. What are some of the alternative approaches/products that they are coming from? Can you describe how Quilt is implemented?

PostgreSQL 130

PostgreSQL SQL Machine Learning Data Warehouse 130

Christophe Blefari

MAY 6, 2023

In the Github repo they are sharing a YAML template describing what's in the contract. Lakehouse at Walmart — Samuel from Walmart describe the research they did and why they picked Hudi over Delta in order to implement a Lakehouse architecture. This article gives a great overview of the product.

Programming Language 100

Programming Language Data Python Data Warehouse 100

Data Engineering Weekly

DECEMBER 10, 2023

Capital One shares its view as part of a case study from Harvard Business Review. link] Thoughtworks: Anonymesh - Data sharing meets privacy and security During my stay in India, I had multiple conversations with fintech data practitioners in India, and many pointed out the challenges with data sharing while preserving privacy & security.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Knowledge Hut

MARCH 29, 2024

You can autoscale your workloads and collaborate on shared projects in an interactive Azure Databricks workspace. Below are some features of Azure Databricks : Collaborative & Interactive Workspace - With Azure Databricks you can quickly explore data and share insights, build models collaboratively.

Data Lake 52

Data Lake Scala Machine Learning SQL 52

ThoughtSpot

FEBRUARY 9, 2023

Unity Catalog enforces security permissions from any client that connects through JDBC/ODBC or Delta Sharing. ThoughtSpot now supports Unity Catalog by integrating directly with Databricks Unity Catalog enabled clusters and warehouses, allowing users to select tables from multiple catalogs that they have access to.

Government 82

Government SQL Machine Learning Cloud 82

Monte Carlo

JANUARY 16, 2024

Zero-copy data sharing 4. Data warehouses like Snowflake or Google BigQuery have made improvements integrating streaming data capabilities while Databricks has added ACID properties via delta tables and its new Unity Catalog. Zero-copy data sharing Snowflake zero-copy cloning hints at the future of data warehousing.

Data Lake 52

Data Lake Data Warehouse Unstructured Data AWS 52

Data Engineering Weekly

SEPTEMBER 3, 2023

link] Kyle Weller: Delta, Hudi, Iceberg — A Benchmark Compilation The LakeHouse architecture brings the best of the database and the data lake into the data infrastructure. Following a benchmark result between Delta & Iceberg , Apache Hudi added its benchmark in the same repo. Which one to choose?

Data Engineering 73

Data Engineering Data Engineer Engineering Data 73

Monte Carlo

JUNE 26, 2023

Ensuring data quality is absolutely critical for ML practitioners to deliver ML products that drive real-world value for their organizations and external customers, more so now than ever before with the rise of generative AI and more widely applicable ML use cases. Data observability isn’t just helping customers at the storage layer either.

Data Lake 52

Data Lake Metadata Bytes Google Cloud 52

Monte Carlo

JUNE 26, 2023

Movements in Data Sharing And we haven’t even started yet! It’s also a natural and smart extension of their Marketplace strategy that will not only help customers share machine learning apps, but LLM models as well. Data sharing was a large focus of both companies last year.

BI 59

BI Machine Learning Data Science Cloud 59

Cloudera

MARCH 17, 2023

The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data. Some of the popular table formats are Apache Iceberg, Delta Lake, Hudi, and Hive ACID. I thought this was a fairly comprehensive list.

Education 60

Education Unstructured Data Data Lake Data Warehouse 60

Scribd Technology

JULY 20, 2022

We also took advantage of the time to have multiple discussions with our colleagues at Databricks about their product and engineering roadmap, and where we can work together to improve the future of Delta Lake, Unity catalog, and more.

Kafka 52

Kafka Data Engineering IT 52

Data Engineering Podcast

JULY 10, 2022

In this episode Inbar Yogev and Lior Winner share the journey that they and their teams at Riskified have been on for their data platform. By the time errors have made their way into production, it’s often too late and damage is done. ML engineers, data engineers, data scientists, data product managers, etc.)

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Monte Carlo

JUNE 28, 2023

and Uniform The New Public Preview Features For Unity Catalog The Conclusion: Delta Sharing and Lakehouse Apps Keynote Kick-Off: Welcome To Generation AI Following a puff of smoke and red lazer light showm Databricks CEO, Ali Ghodsi, stormed the stage. Delta Lake 3.0 We are going to lead the revolution. Very cool stuff.

Data Warehouse 52

Data Warehouse Scala Unstructured Data Government 52

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore).

Big Data 78

Big Data Data Management Management Metadata 78

Data Engineering Podcast

JULY 3, 2022

In this episode Colleen Tartow shares her insights into the motivating factors and benefits of the most prominent patterns that are in the popular narrative; data mesh and the modern data stack. By the time errors have made their way into production, it’s often too late and damage is done. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB MySQL 100

Monte Carlo

MARCH 28, 2023

And so it almost seems unfair that new ideas are already springing up to disrupt the disruptors: Zero-ETL has data ingestion in its sights AI and Large Language Models could transform transformation Data product containers are eyeing the table’s thrown as the core building block of data Are we going to have to rebuild everything (again)?

Data Warehouse 126

Data Warehouse Raw Data Data Pipeline Software Engineer 126

Scott Logic

OCTOBER 18, 2023

Many respondents shared that they expect WebAssembly to deliver on the “write once and run anywhere” promise that was originally made by Java. (If From the above chart we can see that 41% of respondents are using WebAssembly in production, with a further 28% piloting or planning to use it in the next year.

Programming Language 145

Programming Language Coding Java Datasets 145

Data Engineering Podcast

MAY 15, 2022

In this episode he shares his insights and advice on how to approach such an undertaking in your own organization. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies.

Data Lake 100

Data Lake Building BI Business Intelligence 100

Scribd Technology

JULY 19, 2021

Delta Lake is integral to our data platform which is why we have invested heavily in delta-rs to support our non-JVM Delta Lake needs. This year I had the opportunity to share the progress of delta-rs at Data and AI Summit. Suffice it to say, there’s a lot of really interesting work to be done!

Python 40

Python Kafka Data Engineering Data Engineer 40

Data Engineering Podcast

MAY 6, 2018

about the challenges of running machine learning models in production and how his team tracks key metrics and samples production data to re-train and re-deploy those models for better accuracy and more robust operation. Links Spark DataBricks DataBricks Delta Applecart Stepan Pushkarev from Hydrosphere.io What is Hydropshere.io?

Data Science 100

Data Science Machine Learning Data Engineering Data Engineer 100

Monte Carlo

OCTOBER 5, 2023

Data is pretty useless if you can’t act on it, but how do you share insights without crossing privacy lines or worrying that your counterparty mishandles the data? Built from the ground-up with a distinctive multi-cluster architecture, Snowflake promotes a paradigm where data sharing doesn’t necessarily mean sharing raw data.

Raw Data 52

Raw Data Medical Manufacturing AWS 52

Data Engineering Podcast

AUGUST 17, 2020

In this episode the creator and founder of TileDB shares how he first started working on the underlying technology and the benefits of using a single engine for efficiently storing and querying any form of data. Go to dataengineeringpodcast.com/97things to add your voice and share your hard-earned expertise.

Data Engineering 100

Data Engineering Data Engineer Engineering Database Design 100

Towards Data Science

APRIL 3, 2023

And so it almost seems unfair that new ideas are already springing up to disrupt the disruptors: Zero-ETL has data ingestion in its sights AI and Large Language Models could transform transformation Data product containers are eyeing the table’s thrown as the core building block of data Are we going to have to rebuild everything (again)?

Data Engineering 98

Data Engineering Data Engineer Engineering Data Warehouse 98

Monte Carlo

DECEMBER 20, 2023

Each year, we chat with one of the data industry’s pioneering leaders about their predictions for the modern data stack – and share a few of our own. And this year, we invited our good friend and famed venture capitalist Tomasz Tunguz to share his data engineering predictions for 2024. And make no mistake, AI is a data product.

Amazon Web Services 52

Amazon Web Services Metadata AWS Data 52

Monte Carlo

DECEMBER 20, 2023

Each year, we chat with one of the data industry’s pioneering leaders about their predictions for the modern data stack – and share a few of our own. And this year, we invited our good friend and famed venture capitalist Tomasz Tunguz to share his data engineering predictions for 2024. And make no mistake, AI is a data product.

Amazon Web Services 52

Amazon Web Services Metadata AWS Data 52

Engineering at Meta

MARCH 20, 2024

People who use our products and services experience a range of network conditions – some have really great networks, while others are using throttled or low-bandwidth networks. Note that this bandwidth is the share for the video encoder. In-product quality measurement To compare the quality between H.264

Algorithm 86

Algorithm Manufacturing Media Coding 86

Data Engineering Podcast

OCTOBER 15, 2021

They also share their ambitions for the near future of adding data observability and data quality management features. How has your experience of building and running DataHub at LinkedIn informed your product direction for Acryl? Interview Introduction How did you get involved in the area of data management?

Metadata 100

Metadata BI Data Warehouse Government 100