Data Engineering Podcast

MARCH 24, 2024

With their launch of Dagster+ as the redesigned commercial companion to the open source project they are investing in that capability with a suite of new features. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

Data Engineering Podcast

APRIL 14, 2024

Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication. Validate consistency between source and target at any scale, and receive alerts about any discrepancies. The Machine Learning Podcast helps you go from idea to production with machine learning.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Christophe Blefari

MARCH 15, 2024

Elon Musk said out loud that xAI will open-source Grok this week. Just-in-time for a reminder about the fact that open-source ≠ open-weights when it comes to AI licensing but differences in weights licensing are not as important as they seem.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Snowflake

APRIL 24, 2024

Members of the Snowflake AI Research team pioneered systems such as ZeRO and DeepSpeed , PagedAttention / vLLM , and LLM360 which significantly reduced the cost of LLM training and inference, and open sourced them to make LLMs more accessible and cost-effective for the community. Arctic is efficiently intelligent and truly open.

Amazon Web Services 145

Amazon Web Services SQL AWS Architecture 145

Data Engineering Podcast

DECEMBER 24, 2023

Elad Eldor has experienced these challenges first-hand, leading to his work writing the book "Kafka: : Troubleshooting in Production" In this episode he highlights the sources of complexity that contribute to Kafka's operational difficulties, and some of the main ways to identify and mitigate potential sources of trouble.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

The Pragmatic Engineer

MARCH 19, 2024

” It seems they’re aiming for something which performs better than human software engineers, today: Image source: Daniel Gross on X It’s a very bold claim whose success would mean a direct replacement of developers! Source: Cognition So far, all we have is video demos, and accounts of those with access to this tool.

Software Engineer 255

Software Engineer Software Engineering Programming Language Media 255

Data Engineering Podcast

MARCH 31, 2024

While there are numerous products available to provide that visibility, they all have different technologies and workflows that they focus on. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Christophe Blefari

AUGUST 4, 2023

A lot of logos and products will be mentioned. A few requirements The source data lies in Postgres database, in flat CSV and in Google Sheets. I want to use open-source tooling. Everything I do should be production-ready and public. In this article we will try to design the fancy data stack for a batch usage.

Google Cloud 225

Google Cloud MongoDB NoSQL Data 225

Christophe Blefari

JANUARY 26, 2024

This is a fun part where students learn how to serve a simple classifier in production. Here a few takeaways in a messy way: Data products and organisational impacts Data engineers are still the limiting human resource. UX for data products — This is a presentation I really enjoyed by Claire Lebarz, VP data at Malt.

Google Cloud 130

Google Cloud Algorithm Data Java 130

Data Engineering Podcast

FEBRUARY 25, 2024

It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability. One of the major benefits of using open source components is the network effect of ecosystem integrations.

Database 162

Database Technology Data Lake High Quality Data 162

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Max Kanat-Alexander and Grant Jenks Today we are open-sourcing the LinkedIn Developer Productivity & Happiness Framework (DPH Framework) - a collection of documents that describe the systems, processes, metrics, and feedback systems we use to understand our developers and their needs internally at LinkedIn.

Data Schemas 121

Data Schemas Software Engineer Software Engineering Data Pipeline 121

Christophe Blefari

MARCH 2, 2024

I'll give a talk—in French—about how to put in production machine learning at a small scale. Final comment, with these 2 announcement Mistral left the open side to go commercial / closed. And let's be honest this is a productivity boost, especially when you want to write docstrings and comments.

Data 162

Data Python Database BI 162

Data Engineering Podcast

MARCH 3, 2024

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Integration 162

Data Integration BI SQL Python 162

Data Engineering Podcast

MARCH 17, 2024

Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Database 147

Database Data Lake High Quality Data Data Workflow 147

The Pragmatic Engineer

SEPTEMBER 26, 2023

Image source: The Atlas story Today, it is compute that’s much cheaper than software engineers’ time. Developer productivity investments.   At companies with large codebases, CDEs are a pragmatic way to increase the productivity of software engineers. Open source VS Code Server. Remote work.

Cloud 238

Cloud Software Engineer Software Engineering Cloud Computing 238

The Pragmatic Engineer

SEPTEMBER 22, 2023

The dilemma is this: a large, market-leading company has some motivation to innovate, but also a strong disincentive as well, because innovation risks undermining its existing products. Bun also reveals the contrast between venture-funded open source and ‘community sweat’-funded open source.

Programming Language 235

Programming Language Coding Project Engineering 235

Data Engineering Podcast

FEBRUARY 4, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. The Machine Learning Podcast helps you go from idea to production with machine learning. Want to see Starburst in action? Don't forget to check out our other shows.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

MARCH 10, 2024

It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability. What are some of the sources of, and challenges in resolving, merge conflicts between table branches? Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Christophe Blefari

APRIL 5, 2024

DBRX, a new state-of-the-art open LLM — Databricks has to be an AI company ( bragging vs. Snowflake ). This week they released a new open model that performs great. How we built Text-to-SQL at Pinterest — Pinterest open-sourced a tool called Querybook that they used to access Pinterest data every day.

SQL 130

SQL Data Python Datasets 130

Data Engineering Podcast

FEBRUARY 11, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. The Machine Learning Podcast helps you go from idea to production with machine learning. Want to see Starburst in action? Don't forget to check out our other shows.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

The Pragmatic Engineer

APRIL 9, 2024

As I shared in November 2021 , Stack Overflow is retiring this product, and the last day of operation will be 30 March 2022. I asked Stack Overflow why the product is being shut down and had a long conversation with someone working on the product. I spent about 2-3 hours per week vetting inbound candidates.

Software Engineer 253

Software Engineer Software Engineering Engineering Recruitment 253

Data Engineering Podcast

JANUARY 28, 2024

The majority of products that are available either require too much effort to structure the logs, or aren't fast enough for interactive use cases. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

NOVEMBER 26, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Jesse Anderson

OCTOBER 5, 2023

Hearing the focus on data products in the various keynotes was good. However, data products aren’t just exposed on Kafka, as the keynotes tried to make it seem. Kafka is a way to move data around, but that is just one part of a data product. For example, is the data quality open source or closed source?

Kafka 195

Kafka Finance Cloud Designing 195

Analytics Vidhya

APRIL 17, 2023

Introduction Kedro is an open-source Python framework for creating reproducible, maintainable, and modular data science code. It uses best practices of software engineering to build production-ready data science pipelines. This article will give you a glimpse of Kedro framework using news classification tasks.

Data Science 221

Data Science Software Engineer Software Engineering Machine Learning 221

Zalando Engineering

AUGUST 2, 2023

In this post, we explore how adopting Backstage impacted the daily life of Software Engineers at Zalando and get insights from Lacey and Arthur who led the efforts on the Product and Engineering side. What features were added on top of Backstage's open-source project? Why did Zalando choose Backstage for its developer portal?

Software Engineer 98

Software Engineer Software Engineering Engineering Data Pipeline 98

Christophe Blefari

MARCH 22, 2024

Exploration, Friday morning I read the last 7 days of 2 Twitter lists ( MDS , Data voices ) and I open interesting stuff in tabs. Then I use Feedly which is connected to ~500 websites, Reddit and Medium and opens interesting articles in tabs. Then I opens the elements saved from LinkedIn. But I could do it better I think.

Electronics 130

Electronics Media Data Python 130

The Pragmatic Engineer

OCTOBER 10, 2023

However, Martin had not written a line of production code for the last four years, as he’s taken on the role of CEO, and heads up observability scaleup Chronosphere – at more than 250 people and growing. I joined the Microsoft Office team but was tasked to work on a new product. I’m not productive at all.

Software Engineer 247

Software Engineer Software Engineering Architecture Media 247

Christophe Blefari

OCTOBER 20, 2023

Still dbt Labs announcements were mainly towards dbt Cloud with great features to drive adoption of the paid product. They announced dbt Mesh a product enabling cross-project dependencies for teams with multiple dbt projects. I'll, as usual, write a takeaway post after binge watching all keynotes, but this is for next week.

Generalist 130

Generalist Entertainment NoSQL Datasets 130

Data Engineering Podcast

DECEMBER 31, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Want to see Starburst in action? With Materialize, you can!

Designing 130

Designing Data Lake High Quality Data SQL 130

Snowflake

SEPTEMBER 18, 2023

Any data practitioner or product owner will attest to how it takes a lot of steps to build a data app. The product owners work with tooling teams, which are often understaffed and have backlog requests that run into months. Enter Streamlit Streamlit is an open-source library that turns Python scripts into shareable web apps in minutes.

Python 117

Python Building Cloud Amazon Web Services 117

Data Engineering Podcast

DECEMBER 10, 2023

Andrew Maguire got tired of solving that problem for each of the different roles he has ended up in, so he created the open source Anomstack project. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can! Want to see Starburst in action?

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

JANUARY 7, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. What are the open questions today in technical scalability of data engines? The Machine Learning Podcast helps you go from idea to production with machine learning.

Data Process 147

Data Process Process Data Lake High Quality Data 147