Simon Späti

APRIL 15, 2025

Data orchestrators have been essential since the inception of data workloads, because you need something to orchestrate your tasks and your business logic. In the old days that might have been a Makefile or a cron job. But these days, with the challenges and complexity rising exponentially, and the tools still exploding, the orchestrator is the heart of any data engineering project, potentially any data platform.

Data Engineer 162

Data Engineer Data Engineering Engineering Data Pipeline 162

Edureka

APRIL 22, 2025

Data is more than simply numbers as we approach 2025; it serves as the foundation for business decision-making in all sectors. However, data alone is insufficient. To remain competitive in the current digital environment, businesses must effectively gather, handle, and manage it. Data engineering can help with it. It is the force behind seamless data flow, enabling everything from AI-driven automation to real-time analytics.

Data Engineer 52

Data Engineer Data Engineering Engineering Consulting 52

KDnuggets

APRIL 25, 2025

Ready to level up your data engineering game without wasting hours on setup? From ingestion to orchestration, these Docker containers handle it all.

Data Engineer 119

Data Engineer Data Engineering Engineering Data 119

Engineering at Meta

APRIL 28, 2025

Managing and understanding large-scale data ecosystems is a significant challenge for many organizations, requiring innovative solutions to efficiently safeguard user data. Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale. To address these challenges, we made substantial investments in advanced data understanding technologies, as part of our Privacy Aware Infrastructure (PAI).

Metadata 83

Metadata Data Utilities Data Warehouse 83

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Edureka

APRIL 9, 2025

Imagine being in charge of creating an intelligent data universe where collaboration, analytics, and artificial intelligence all work together harmoniously. That’s what a Microsoft Fabric Engineer does. Microsoft Fabric is a strong, cohesive platform that unifies data lakes, warehousing, governance, real-time analytics, and more under one roof as companies scramble to make sense of mountains of data.

Engineering 40

Engineering Data Ingestion Data Lake Programming Language 40

Start Data Engineering

APRIL 14, 2025

1. Introduction 2. APIs are a way to communicate between systems on the Internet 2.1. HTTP is a protocol commonly used for websites 2.1.1. Request: Ask the Internet exactly what you want 2.1.2. Response is what you get from the server 3. API Data extraction = GET-ting data from a server 3.1. GET data 3.1.1. GET data for a specific entity 3.

Data Pipeline 130

Data Pipeline Python Data Systems 130

The Pragmatic Engineer

APRIL 15, 2025

👋 Hi, this is Gergely with a free issue of the Pragmatic Engineer Newsletter. We cover two out of seven topics in today’s subscriber-only deepdive: Tech hiring: is this an inflection point? If you’ve been forwarded this email, you can subscribe here. Before we start: I do one conference talk every year, and this year it will be a keynote at LDX3 in London, on 16 June.

Recruitment 207

Recruitment Coding Engineering Data Engineer 207

DataKitchen

APRIL 9, 2025

Why Data Quality Isnt Worth The Effort : Data Quality Coffee With Uncle Chip Data quality has become one of the most discussed challenges in modern data teams, yet it remains one of the most thankless and frustrating responsibilities. In the first of the Data Quality Coffee With Uncle Chip series, he highlights the persistent tension between the need for clean, reliable data and its overwhelming complexity.

Data 67

Data SQL Data Engineer Data Engineering 67

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew. The data warehouse solved for performance and scale but, much like the databases that preceded it, relied on proprietary formats to build vertically integrated systems.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

Seattle Data Guy

APRIL 15, 2025

Planning out your data infrastructure in 2025 can feel wildly different than it did even five years ago. The ecosystem is louder, flashier, and more fragmented. Everyone is talking about AI, chatbots, LLMs, vector databases, and whether your data stack is “AI-ready.” Vendors promise magic, just plug in their tool and watch your insights appear.… Read more The post How To Set Up Your Data Infrastructure In 2025 Part 1 appeared first on Seattle Data Guy.

Database 182

Database Data IT Big Data 182

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data Workflow

Confessions of a Data Guy

APRIL 21, 2025

Rethinking Object Storage: A First Look at CloudflareR2 and Its BuiltIn ApacheIceberg Catalog Sometimes, we follow tradition because, well, it worksuntil something new comes along and makes us question the status quo. For many of us, AmazonS3 is that welltrodden path: the backbone of our data platforms and pipelines, used countless times each day. If […] The post Cloudflare R2 Storage with Apache Iceberg appeared first on Confessions of a Data Guy.

IT 130

IT Data 130

dbt Developer Hub

APRIL 20, 2025

dbt is the standard for creating governed, trustworthy datasets on top of your structured data. MCP is showing increasing promise as the standard for providing context to LLMs to allow them to function at a high level in real world, operational scenarios. Today, we are open sourcing an experimental version of the dbt MCP server. We expect that over the coming years, structured data is going to become heavily integrated into AI workflows and that dbt will play a key role in building and provision

Structured Data 98

Structured Data SQL BI Metadata 98

Netflix Tech

APRIL 8, 2025

By Cheng Xie , Bryan Shultz , and Christine Xu In a previous blog post , we described how Netflix uses eBPF to capture TCP flow logs at scale for enhanced network insights. In this post, we delve deeper into how Netflix solved a core problem: accurately attributing flow IP addresses to workload identities. A BriefRecap FlowExporter is a sidecar that runs alongside all Netflix workloads.

AWS 77

AWS Kafka Certification Cloud 77

Engineering at Meta

APRIL 2, 2025

Open source has played an essential role in the tech industry and beyond. Whether in the AI/ML, web, or mobile space, our open source community grew and evolved while connecting people worldwide. At Meta Open Source , 2024 was a year of growth and transformation. Our open source initiatives addressed the evolving needs and challenges of developerspowering breakthroughs in AI and enabling the creation of innovative, user-focused applications and experiences.

Portfolio 95

Portfolio Media Project Technology 95

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Snowflake

APRIL 13, 2025

Managing and utilizing data effectively is crucial for organizational success in today's fast-paced technological landscape. The vast amounts of data generated daily require advanced tools for efficient management and analysis. Enter agentic AI, a type of artificial intelligence set to transform enterprise data management. As the Snowflake CTO at Deloitte, I have seen the powerful impact of these technologies, especially when leveraging the combined experience of the Deloitte and Snowflake allia

Data Management 93

Data Management Management Consulting Unstructured Data 93

Seattle Data Guy

APRIL 27, 2025

Over the past three years our teams have noticed a pattern. Many companies looking to migrate to the cloud go from SQL Server to Snowflake. There are many reasons this makes sense. One of the reasons and common benefits was that teams found it far easier to manage that SQL Server and in almost every… Read more The post How To Migrate From SQL Server To Snowflake appeared first on Seattle Data Guy.

SQL 130

SQL Cloud Management IT 130

Pinterest Engineering

APRIL 4, 2025

Han Wang | Machine Learning Engineer II, Relevance & Query Understanding; Mukuntha Narayanan | Machine Learning Engineer II, Relevance & Query Understanding; Onur Gungor | (former) Staff Machine Learning Engineer, Relevance & Query Understanding; Jinfeng Rao | Senior Staff Machine Learning Engineer, Pinner Discovery Figure: Illustration of the search relevance system at Pinterest.

Machine Learning 65

Machine Learning Metadata Architecture Datasets 65

InData Labs

APRIL 10, 2025

The retail sector is among the most competitive markets, making it exceptionally difficult for businesses to not only thrive but even survive. Business intelligence in retail industry can be a colossal game changer for organizations struggling to compete. BI for retail allows companies to leverage Big data analytics and machine learning techniques to extract valuable.

Business Intelligence 82

Business Intelligence Retail BI Big Data 82

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Precisely

APRIL 7, 2025

A former colleague recently asked me to explain my role at Precisely. After my (admittedly lengthy) explanation of what I do as the EVP and GM of our Enrich business, she summarized it in a very succinct, but new way: “Oh, you manage the appending datasets.” That got me thinking. We often use different terms when were talking about the same thing in this case, data appending vs. data enrichment.

Retail 75

Retail Datasets Data Telecommunication 75

Monte Carlo

APRIL 28, 2025

Different teams love using the same data in totally different ways. Eventually, it gets to the point where everyone has their own secret nickname for the same customer fieldlike Sales calling it cust_id, while Marketing goes with user_ref. And yeah… thats kind of a problem. Thats where data dictionary tools come in. A data dictionary tool helps define and organize your data so everyones speaking the same language.

Metadata 64

Metadata Hadoop Data SQL 64

Snowflake

APRIL 23, 2025

In our fast-paced data- and AI-driven world, teaching students the skills they need to succeed in the industry is more critical than ever. If youre an instructor in data science, data engineering or business intelligence at a nonprofit, accredited institution, Snowflakes Academia Program provides a unique opportunity to enhance your teaching experience while equipping students with the in-demand skills they need to stand out in the job market.

Education 97

Education Certification Retail Portfolio 97

Seattle Data Guy

APRIL 21, 2025

If you work in data, then youve likely used BigQuery and youve likely used it without really thinking about how it operates under the hood. On the surface BigQuery is Google Clouds fully-managed, serverless data warehouse. Its the Redshift of GCP except we like it a little more. The question becomes, how does it work?… Read more The post What Is BigQuery And How Do You Load Data Into It?

IT 130

IT Google Cloud Data Warehouse Data 130

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Pinterest Engineering

APRIL 7, 2025

Jia Zhan, Senior Staff Software Engineer, Pinterest Sachin Holla, Principal Solution Architect, AWS Summary Pinterest is a visual search engine and powers over 550 million monthly active users globally. Pinterests infrastructure runs on AWS and leverages Amazon EC2 instances for its compute fleet. In recent years, while managing Pinterests EC2 infrastructure, particularly for our essential online storage systems, we identified a significant challenge: the lack of clear insights into EC2s network

AWS 66

AWS Bytes Data Ingestion Database 66

ThoughtSpot

APRIL 22, 2025

Loved by Business Leaders, Trusted by Analysts Last year, we introduced Spotter our AI analyst that delivers agentic data experiences with enterprise-grade trust and scale. Today, were delivering several key innovations that will help you streamline insights-to-actions with agentic analytics, crossing a major milestone on our path to enabling an autonomous business.

BI 59

BI Business Intelligence Datasets Raw Data 59

Data Engineering Weekly

APRIL 27, 2025

Try Apache Airflow® 3 on Astro Airflow 3 is here and has never been easier to use or more secure. Spin up a new 3.0 deployment on Astro to test DAG versioning, backfills, event-driven scheduling, and more. Get started → Chip Huyen: Exploring three strategies - functional correctness, AI-as-a-judge, and comparative evaluation As AI development becomes mainstream, so does the need to adopt all the best practices in software engineering.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Waitingforcode

APRIL 15, 2025

Previously we learned how to control data quality with Delta Live Tables. Now, it's time to see an open source library in action, Spark Expectations.

Data 147

Data IT 147

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Snowflake

APRIL 9, 2025

The traditional five-year anniversary gift is wood. Since snowboards often have a wooden core, and because a snowboard is the traditional trophy for the Snowflake Startup Challenge, were going to go ahead and say that the snowboard trophy qualifies as a present for the fifth anniversary of our Startup Challenge. The only difference is that instead of receiving the gift, well be giving it to one of the 10 semifinalists listed below!

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

Edureka

APRIL 22, 2025

Selecting the appropriate data platform becomes crucial as businesses depend more and more on data to inform their decisions. Although they take quite different approaches, Microsoft Fabric and Snowflake, two of the top players in the current data landscape, both provide strong capabilities. Understanding how these platforms compare can assist you in selecting the best option for your company, regardless of your role as a data engineer, business analyst, or decision-maker.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

KDnuggets

APRIL 30, 2025

Learn five powerful open-source AI tools to boost your projects, save time, and stay ahead in AI innovation.

Project 139

Project 139

databricks

APRIL 8, 2025

Summary: LLMs have revolutionized software development by increasing the productivity of programmers.

Coding 130

Coding Data 130

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineer