article thumbnail

Building Data Platforms (from scratch)

Confessions of a Data Guy

Build new pipeline, update pipeline, new data model, fix bug, etc, etc. It’s a constant stream of data, new and old, spilling into our Data Warehouses and […] The post Building Data Platforms (from scratch) appeared first on Confessions of a Data Guy. It’s never-ending.

Building 184
article thumbnail

Building cost effective data pipelines with Python & DuckDB

Start Data Engineering

Building efficient data pipelines with DuckDB 4.1. Introduction 2. Project demo 3. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4.


Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Kafka to MongoDB: Building a Streamlined Data Pipeline

Analytics Vidhya

We know that streaming data is data that is emitted at high volume […] The post Kafka to MongoDB: Building a Streamlined Data Pipeline appeared first on Analytics Vidhya. Handling and processing the streaming data is the hardest work for Data Analysis.

MongoDB 217
article thumbnail

Building Open-Source Python Packages – SparklePop

Confessions of a Data Guy

In a show of solidarity to the open-source community, I thought I would publish a PYPI package that will probably be used by 5 people […] The post Building Open-Source Python Packages – SparklePop appeared first on Confessions of a Data Guy.

Python 100
article thumbnail

Reimagined: Building Products with Generative AI

“Reimagined: Building Products with Generative AI” is an extensive guide for integrating generative AI into product strategy and careers featuring over 150 real-world examples, 30 case studies, and 20+ frameworks, and endorsed by over 20 leading AI and product executives, inventors, entrepreneurs, and researchers.

article thumbnail

Building ETL Pipelines With Generative AI

Data Engineering Podcast

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. How can you get the best results for your use case?

Building 162
article thumbnail

Building Databricks Data Pipelines 101

Confessions of a Data Guy

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? The post Building Databricks Data Pipelines 101 appeared first on Confessions of a Data Guy. What does it look like, what tools do you use?

article thumbnail

How to Build an Experimentation Culture for Data-Driven Product Development

Speaker: Margaret-Ann Seger, Head of Product, Statsig

In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating at the level of a FAANG company!

article thumbnail

Why “Build or Buy?” Is the Wrong Question for Analytics

Every time an application team gets caught up in the “build vs buy” debate, it stalls projects and delays time to revenue. Partnering with an analytics development platform gives you the freedom to customize a solution without the risks and long-term costs of building your own. There is a third option.

article thumbnail

How to Build Data Experiences for End Users

Organizational data literacy is regularly addressed, but it’s uncommon for product managers to consider users’ data literacy levels when building products. Product managers need to research and recognize their end users' data literacy when building an application with analytic features.

article thumbnail

The Essential Guide to Building Analytic Applications

Download this eBook to discover insights from 16 top product experts, and learn what it takes to build a successful application with analytics at its core. What should product managers keep in mind when adding an analytics project to their roadmap?

article thumbnail

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Speaker: Maher Hanafi, VP of Engineering at Betterworks & Tony Karrer, CTO at Aggregage

He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.

article thumbnail

LLMs in Production: Tooling, Process, and Team Structure

Speaker: Dr. Greg Loughnane and Chris Alexiuk

Register today to save your seat! December 6th, 2023 at 11:00am PST, 2:00pm EST, 7:pm GMT

article thumbnail

3 Challenges of Building Complex Dashboards with Open Source Components

Speaker: Ryan MacCarrigan, Founding Principal, LeanStudio

Many product teams use charting components and open source code libraries to get dashboards and reporting functionality quickly. But what happens when you have a growing user base and additional feature requests?

article thumbnail

LLMOps for Your Data: Best Practices to Ensure Safety, Quality, and Cost

Speaker: Shreya Rajpal, Co-Founder and CEO at Guardrails AI & Travis Addair, Co-Founder and CTO at Predibase

Putting the right LLMOps process in place today will pay dividends tomorrow, enabling you to leverage the part of AI that constitutes your IP – your data – to build a defensible AI strategy for the future.