article thumbnail

Building a Fast, Light, and CHEAP Lake House with DuckDB, Delta Lake, and AWS Lambda

Confessions of a Data Guy

Building fun things is a real part of Data Engineering. Using your creative side when building a Lake House is possible, and using tools that are outside the normal box can sometimes be preferable.

AWS 130
article thumbnail

Building cost effective data pipelines with Python & DuckDB

Start Data Engineering

Building efficient data pipelines with DuckDB 4.1. Introduction 2. Project demo 3. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Building Data Platforms (from scratch)

Confessions of a Data Guy

Build new pipeline, update pipeline, new data model, fix bug, etc, etc. It’s a constant stream of data, new and old, spilling into our Data Warehouses and […] The post Building Data Platforms (from scratch) appeared first on Confessions of a Data Guy. It’s never-ending.

Building 184
article thumbnail

Building an an Early Stage Startup: Lessons from Akita Software

The Pragmatic Engineer

In this issue, we cover: How Akita was founded On cofounders Raising funding Pivoting and growing the company On hiring The tech stack The biggest challenges of building a startup For this article, I interviewed Jean directly. So we started to build API specs on top of our API security product. We pivoted to API observability in 2020.

Building 235
article thumbnail

Reimagined: Building Products with Generative AI

“Reimagined: Building Products with Generative AI” is an extensive guide for integrating generative AI into product strategy and careers featuring over 150 real-world examples, 30 case studies, and 20+ frameworks, and endorsed by over 20 leading AI and product executives, inventors, entrepreneurs, and researchers.

article thumbnail

Kafka to MongoDB: Building a Streamlined Data Pipeline

Analytics Vidhya

We know that streaming data is data that is emitted at high volume […] The post Kafka to MongoDB: Building a Streamlined Data Pipeline appeared first on Analytics Vidhya. Handling and processing the streaming data is the hardest work for Data Analysis.

MongoDB 222
article thumbnail

Building End-to-End Data Pipelines with Dask

KDnuggets

Learn how to implement a parallelization process in your data pipeline.

article thumbnail

Improving the Accuracy of Generative AI Systems: A Structured Approach

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases.

article thumbnail

Agent Tooling: Connecting AI to Your Tools, Systems & Data

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data.

article thumbnail

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Speaker: Maher Hanafi, VP of Engineering at Betterworks & Tony Karrer, CTO at Aggregage

He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.

article thumbnail

Why “Build or Buy?” Is the Wrong Question for Analytics

Every time an application team gets caught up in the “build vs buy” debate, it stalls projects and delays time to revenue. Partnering with an analytics development platform gives you the freedom to customize a solution without the risks and long-term costs of building your own. There is a third option.

article thumbnail

How to Build Data Experiences for End Users

Organizational data literacy is regularly addressed, but it’s uncommon for product managers to consider users’ data literacy levels when building products. Product managers need to research and recognize their end users' data literacy when building an application with analytic features.

article thumbnail

The Essential Guide to Building Analytic Applications

Download this eBook to discover insights from 16 top product experts, and learn what it takes to build a successful application with analytics at its core. What should product managers keep in mind when adding an analytics project to their roadmap?

article thumbnail

LLMs in Production: Tooling, Process, and Team Structure

Speaker: Dr. Greg Loughnane and Chris Alexiuk

Register today to save your seat! December 6th, 2023 at 11:00am PST, 2:00pm EST, 7:pm GMT

article thumbnail

3 Challenges of Building Complex Dashboards with Open Source Components

Speaker: Ryan MacCarrigan, Founding Principal, LeanStudio

Many product teams use charting components and open source code libraries to get dashboards and reporting functionality quickly. But what happens when you have a growing user base and additional feature requests?