Confessions of a Data Guy

APRIL 21, 2025

Rethinking Object Storage: A First Look at CloudflareR2 and Its BuiltIn ApacheIceberg Catalog Sometimes, we follow tradition because, well, it worksuntil something new comes along and makes us question the status quo. For many of us, AmazonS3 is that welltrodden path: the backbone of our data platforms and pipelines, used countless times each day. If […] The post Cloudflare R2 Storage with Apache Iceberg appeared first on Confessions of a Data Guy.

IT 130

IT Data 130

Seattle Data Guy

APRIL 21, 2025

If you work in data, then youve likely used BigQuery and youve likely used it without really thinking about how it operates under the hood. On the surface BigQuery is Google Clouds fully-managed, serverless data warehouse. Its the Redshift of GCP except we like it a little more. The question becomes, how does it work?… Read more The post What Is BigQuery And How Do You Load Data Into It?

IT 130

IT Google Cloud Data Warehouse Data 130

dbt Developer Hub

APRIL 20, 2025

dbt is the standard for creating governed, trustworthy datasets on top of your structured data. MCP is showing increasing promise as the standard for providing context to LLMs to allow them to function at a high level in real world, operational scenarios. Today, we are open sourcing an experimental version of the dbt MCP server. We expect that over the coming years, structured data is going to become heavily integrated into AI workflows and that dbt will play a key role in building and provision

Structured Data 98

Structured Data SQL BI Project 98

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew. The data warehouse solved for performance and scale but, much like the databases that preceded it, relied on proprietary formats to build vertically integrated systems.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

If AI agents are going to deliver ROI, they need to move beyond chat and actually do things. But, turning a model into a reliable, secure workflow agent isn’t as simple as plugging in an API. In this new webinar, Alex Salazar and Nate Barbettini will break down the emerging AI architecture that makes action possible, and how it differs from traditional integration approaches.

Systems

Netflix Tech

APRIL 8, 2025

By Cheng Xie , Bryan Shultz , and Christine Xu In a previous blog post , we described how Netflix uses eBPF to capture TCP flow logs at scale for enhanced network insights. In this post, we delve deeper into how Netflix solved a core problem: accurately attributing flow IP addresses to workload identities. A BriefRecap FlowExporter is a sidecar that runs alongside all Netflix workloads.

AWS 75

AWS Kafka Cloud Programming 75

ThoughtSpot

APRIL 22, 2025

Loved by Business Leaders, Trusted by Analysts Last year, we introduced Spotter our AI analyst that delivers agentic data experiences with enterprise-grade trust and scale. Today, were delivering several key innovations that will help you streamline insights-to-actions with agentic analytics, crossing a major milestone on our path to enabling an autonomous business.

BI 59

BI Datasets Business Intelligence Raw Data 59

Seattle Data Guy

APRIL 15, 2025

Planning out your data infrastructure in 2025 can feel wildly different than it did even five years ago. The ecosystem is louder, flashier, and more fragmented. Everyone is talking about AI, chatbots, LLMs, vector databases, and whether your data stack is “AI-ready.” Vendors promise magic, just plug in their tool and watch your insights appear.… Read more The post How To Set Up Your Data Infrastructure In 2025 Part 1 appeared first on Seattle Data Guy.

Database 130

Database Data IT Big Data 130

InData Labs

APRIL 10, 2025

The retail sector is among the most competitive markets, making it exceptionally difficult for businesses to not only thrive but even survive. Business intelligence in retail industry can be a colossal game changer for organizations struggling to compete. BI for retail allows companies to leverage Big data analytics and machine learning techniques to extract valuable.

Business Intelligence 82

Business Intelligence Retail BI Big Data 82

Snowflake

APRIL 9, 2025

The traditional five-year anniversary gift is wood. Since snowboards often have a wooden core, and because a snowboard is the traditional trophy for the Snowflake Startup Challenge, were going to go ahead and say that the snowboard trophy qualifies as a present for the fifth anniversary of our Startup Challenge. The only difference is that instead of receiving the gift, well be giving it to one of the 10 semifinalists listed below!

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

Precisely

APRIL 7, 2025

A former colleague recently asked me to explain my role at Precisely. After my (admittedly lengthy) explanation of what I do as the EVP and GM of our Enrich business, she summarized it in a very succinct, but new way: “Oh, you manage the appending datasets.” That got me thinking. We often use different terms when were talking about the same thing in this case, data appending vs. data enrichment.

Retail 75

Retail Datasets Data Portfolio 75

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Edureka

APRIL 22, 2025

Selecting the appropriate data platform becomes crucial as businesses depend more and more on data to inform their decisions. Although they take quite different approaches, Microsoft Fabric and Snowflake, two of the top players in the current data landscape, both provide strong capabilities. Understanding how these platforms compare can assist you in selecting the best option for your company, regardless of your role as a data engineer, business analyst, or decision-maker.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

DataKitchen

APRIL 9, 2025

Why Data Quality Isnt Worth The Effort : Data Quality Coffee With Uncle Chip Data quality has become one of the most discussed challenges in modern data teams, yet it remains one of the most thankless and frustrating responsibilities. In the first of the Data Quality Coffee With Uncle Chip series, he highlights the persistent tension between the need for clean, reliable data and its overwhelming complexity.

Data 67

Data SQL Data Engineering Data Engineer 67

Pinterest Engineering

APRIL 7, 2025

Jia Zhan, Senior Staff Software Engineer, Pinterest Sachin Holla, Principal Solution Architect, AWS Summary Pinterest is a visual search engine and powers over 550 million monthly active users globally. Pinterests infrastructure runs on AWS and leverages Amazon EC2 instances for its compute fleet. In recent years, while managing Pinterests EC2 infrastructure, particularly for our essential online storage systems, we identified a significant challenge: the lack of clear insights into EC2s network

AWS 57

AWS Bytes Database Data Ingestion 57

Data Engineering Weekly

APRIL 24, 2025

In our latest episode of Data Engineering Weekly, co-hosted by Aswin, we explored the practical realities of AI deployment and data readiness with our distinguished guest, Avinash Narasimha, Former AI Solutions Leader at Koch Industries. This discussion shed significant light on the maturity, challenges, and potential that generative AI and data preparedness present in contemporary enterprises.

Government 52

Government Data Governance High Quality Data Machine Learning 52

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Snowflake

APRIL 13, 2025

Managing and utilizing data effectively is crucial for organizational success in today's fast-paced technological landscape. The vast amounts of data generated daily require advanced tools for efficient management and analysis. Enter agentic AI, a type of artificial intelligence set to transform enterprise data management. As the Snowflake CTO at Deloitte, I have seen the powerful impact of these technologies, especially when leveraging the combined experience of the Deloitte and Snowflake allia

Data Management 78

Data Management Management Consulting Unstructured Data 78

WeCloudData

APRIL 22, 2025

PaaS is a fundamental cloud computing model that offers developers and organizations a robust environment for building, deploying, and managing applications efficiently. This blog provides detailed information on data Platform as a Service (PaaS),, how it differs from other cloud computing models, its working principles, and its benefits. Lets get started and explore PaaS with […] The post Platform as a Service (PaaS) appeared first on WeCloudData.

Cloud Computing 52

Cloud Computing Cloud Building Management 52

Edureka

APRIL 22, 2025

Data is more than simply numbers as we approach 2025; it serves as the foundation for business decision-making in all sectors. However, data alone is insufficient. To remain competitive in the current digital environment, businesses must effectively gather, handle, and manage it. Data engineering can help with it. It is the force behind seamless data flow, enabling everything from AI-driven automation to real-time analytics.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

KDnuggets

APRIL 22, 2025

Looking to expand your programming toolkit? This guide aims to help Python developers quickly get going with Go.

Python 125

Python Programming 125

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Pinterest Engineering

APRIL 4, 2025

Han Wang | Machine Learning Engineer II, Relevance & Query Understanding; Mukuntha Narayanan | Machine Learning Engineer II, Relevance & Query Understanding; Onur Gungor | (former) Staff Machine Learning Engineer, Relevance & Query Understanding; Jinfeng Rao | Senior Staff Machine Learning Engineer, Pinner Discovery Figure: Illustration of the search relevance system at Pinterest.

Machine Learning 56

Machine Learning Metadata Architecture Datasets 56

Data Engineering Weekly

APRIL 13, 2025

Introducing Apache Airflow® 3.0 Be among the first to see Airflow 3.0 in action and get your questions answered directly by the Astronomer team. You won't want to miss this live event on April 23rd! Save Your Spot → Stanford HAI: AI Index 2025 - State of AI in 10 Charts Stanford gives an insight into AI adoption in the industry with the AI adoption.

Data Engineering 63

Data Engineering Data Engineer Engineering Datasets 63

Snowflake

APRIL 5, 2025

At Snowflake, we are committed to providing our customers with industry-leading LLMs. Were pleased to bring Metas latest Llama 4 models to Snowflake Cortex AI! Llama 4 models deliver performant inference so customers can build enterprise-grade generative AI applications and deliver personalized experiences. The Llama 4 Maverick and Llama 4 Scout models can be accessed within the secure Snowflake perimeter on Cortex AI.

Architecture 74

Architecture SQL Data Pipeline Accessible 74

Cloudyard

APRIL 22, 2025

Read Time: 2 Minute, 34 Second Introduction In modern data pipelines, especially in cloud data platforms like Snowflake, data ingestion from external systems such as AWS S3 is common. However, one critical question that often arises is: How do we ensure the data we receive from the source matches the data we ingest into Snowflake tables? This is where “Snowpark Magic: Auto-Validate Your S3 to Snowflake Data Loads”comes into play a powerful approach to automate row-level validation b

Data Validation 52

Data Validation Data Ingestion Data Pipeline AWS 52

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Waitingforcode

APRIL 15, 2025

Previously we learned how to control data quality with Delta Live Tables. Now, it's time to see an open source library in action, Spark Expectations.

Data 130

Data IT 130

KDnuggets

APRIL 7, 2025

Doing data science projects can be demanding, but it doesnt mean it has to be boring. Here are four projects to introduce more fun to your learning and stand out from the masses.

Data Science 137

Data Science Project Data IT 137

Striim

APRIL 18, 2025

AI and analytics have the potential to transform decision-making, streamline operations, and drive innovation. But theyre only as good as the data they rely on. If the underlying data is incomplete, inconsistent, or delayed, even the most advanced AI models and business intelligence systems will produce unreliable insights. Many organizations struggle with: Inconsistent data formats : Different systems store data in varied structures, requiring extensive preprocessing before analysis.

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Data Engineering Weekly

APRIL 20, 2025

Exclusive look at Apache Airflow® 3.0 Get a first look at all the new features in Airflow 3.0, such as DAG versioning, backfills, and dark mode, in a live session this Wednesday, April 23. Plus, get your questions answered directly by Airflow experts and contributors. Register now → Thoughtworks: AI on Technology Radar ThoughtWorks' technology radar inspired many enterprises to build their internal tech radars, standardizing and suggesting technology, tools, and framework adoption.

Data Engineering 68

Data Engineering Data Engineer Engineering Kafka 68

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Snowflake

APRIL 16, 2025

Snowflake Cortex AI now features native multimodal AI capabilities, eliminating data silos and the need for separate, expensive tools. Introducing Cortex AI COMPLETE Multimodal , now in public preview. This major enhancement brings the power to analyze images and other unstructured data directly into Snowflakes query engine, using familiar SQL at scale.

Data Analysis 59

Data Analysis Unstructured Data Manufacturing Retail 59

Start Data Engineering

APRIL 14, 2025

1. Introduction 2. APIs are a way to communicate between systems on the Internet 2.1. HTTP is a protocol commonly used for websites 2.1.1. Request: Ask the Internet exactly what you want 2.1.2. Response is what you get from the server 3. API Data extraction = GET-ting data from a server 3.1. GET data 3.1.1. GET data for a specific entity 3.

Data Pipeline 130

Data Pipeline Python Data Systems 130

databricks

APRIL 8, 2025

Summary: LLMs have revolutionized software development by increasing the productivity of programmers.

Coding 132

Coding Data 132

KDnuggets

APRIL 11, 2025

Give your LLMs the extra ability to fetch live stock prices, compare them, and provide historical analysis by implementation tools within the MCP Server.

Building 131

Building 131

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer