Confessions of a Data Guy

APRIL 19, 2024

Back in March, I did a writeup and experiment called DuckDB vs Polars, Thunderdom, 16GB on 4GB machine challenge. The idea was to see if the two tools could process “larger than memory” datasets with lazy execution. Polars worked fine, DuckDB failed in spectacular fashion.

IT 140

IT Datasets Process Data 140

Big Data Tools

NOVEMBER 8, 2021

The lockdowns are back again in Moscow, which means that conferences are again out of the question for me for some time. Apache Spark® has been released and there are a load of changes, including ANSI SQL support, Pandas API layer over PySpark, and lots and lots of other things. Apache Ranger 2.2.0 This release is huge!

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

NOVEMBER 8, 2021

The lockdowns are back again in Moscow, which means that conferences are again out of the question for me for some time. Apache Spark® has been released and there are a load of changes, including ANSI SQL support, Pandas API layer over PySpark, and lots and lots of other things. Apache Ranger 2.2.0 This release is huge!

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Christophe Blefari

MARCH 15, 2024

You can subscribe starting today on the page and you'll get emails as soon as I've developed the email sending—expected to be out at the end of the month. AI News 🤖 Mira Murati answers the Wall Street Journal about OpenAI Sora — OpenAI CTO has been asked a few questions about the underlying technology in Sora.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Engineering at Meta

FEBRUARY 20, 2024

Meta’s Data Infrastructure teams have been rethinking how data management systems are designed. This new convergence helps Meta and the larger community build data management systems that are unified, more efficient, and composable. An introduction to Velox Velox is the first project in our composable data management system program.

Data Management 98

Data Management Bytes Management Datasets 98

Towards Data Science

JUNE 6, 2023

Fitbit activity analysis with DuckDB Photo by Jake Hills on Unsplash Wearable fitness trackers have become an integral part of our lives, collecting and tracking data about our daily activities, sleep patterns, location, heart rate, and much more. I’ve been using a Fitbit device for 6 years to monitor my health. Why DuckDB?

Data Warehouse 92

Data Warehouse Data SQL Data Analysis 92