RandomTrees
MAY 6, 2025
The introduction of artificial intelligence technology is revolutionizing how companies do business, increase operations, and compete with one another. This new technology is helping businesses make faster marketing predictions and better manage customer interactions. However, to succeed, AI requires a foundation of reliable and structured data. Modern data engineering can help with this.
Edureka
MAY 8, 2025
When you purchase online, do you ever find yourself pondering how your tastes get changed into suggestions for products that are uniquely suited to you? Or how self-driving cars get through very complicated situations with amazing accuracy? These are the ways that data engineering improves our lives in the real world. The field of data engineering turns unstructured data into ideas that can be used to change businesses and our lives.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
MAY 14, 2025
Add these 4 data analytic-based projects to your resume to land your next job.
DataKitchen
MAY 2, 2025
Is Your Team in Denial of Data Quality? Here’s How to Tell In many organizations, data quality problems fester in the shadowsignored, rationalized, or swept aside with confident-sounding statements that mask a deeper dysfunction. While the cost of poor data quality is well-documentedwasted time, lost trust, and flawed decisionsteams often operate in collective denial.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
ArcGIS
MAY 23, 2025
Learn how to prepare your data to perform time-to-event predictions in ArcGIS Pro 3.
Striim
MAY 9, 2025
Get More Insights In Your Inbox Sol Rashidi has built AI, data, and digital strategies inside some of the worlds biggest companiesand shes seen the same mistakes play out again and again. In this episode, she unpacks why AI initiatives often stall, how executives misread what transformation really requires, and why the future of AI success isnt technicalits cultural.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Monte Carlo
MAY 22, 2025
Alright, so youve got data flying in from all directionsapps, websites, databases, you name itand you need to wrangle it into something clean, useful, and actually understandable. Thats where ETL Orchestration comes in. ETL orchestration is the process of managing and automating the flow of data as its extracted, transformed, and loaded (yep, thats the E.T.L. ) from one place to another.
Simon Späti
MAY 19, 2025
Imagine building enterprise data infrastructure where you write 90% less code but deliver twice the value. This is the promise of declarative data stacks. The open and modern data stack freed us from vendor lock-in, allowing teams to select best-of-breed tools for ingestion, ETL, and orchestration. But this freedom comes at a cost: fragmented governance, security gaps, and potential technical debt when stacking disconnected tools across your organization.
RandomTrees
MAY 27, 2025
Introduction In today’s digitally linked world, intuition is no longer sufficient to drive B2B marketing. Data analytics has developed as a critical component of effective marketing strategies, allowing companies to make educated decisions that improve performance and create quantifiable results. With vast amounts of client data available across digital channels, organizations that use data analytics may acquire a significant competitive edge.
KDnuggets
MAY 27, 2025
You dont need to be a Python pro to write fast, clean code. Just a few smart coding habits can go a long way.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Waitingforcode
MAY 7, 2025
One of the biggest changes for PySpark has been the DataFrame API. It greatly reduces the JVM-to-PVM communication overhead and improves the performance. However, it also complexities the code. Probably, some of you have already seen, written, or worked with the code like this.
Jesse Anderson
MAY 5, 2025
I n this episode of Unapologetically Technical, I interview Shane Murray, Field CTO at Monte Carlo Data. Shane shares his compelling journey from studying math and finance in Sydney, Australia, to leading AI strategy at a major data observability company in New York. We explore his early work in choice modeling and pioneering online multivariate experimentation long before A/B testing became mainstream, including fascinating examples from cruise lines, American Express, and even cultural surpris
databricks
MAY 28, 2025
Apache Spark 4.0 marks a major milestone in the evolution of the Spark analytics engine.
ArcGIS
MAY 27, 2025
In ArcGIS Pro 3.5, you can choose between three options to overwrite existing tool data, including appending and replacing data.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Engineering at Meta
MAY 5, 2025
Meta and Quantsight have improved key libraries in the Python Ecosystem. There is plenty more to do and we invite the community to help with our efforts. Well look at two key efforts in Pythons packaging ecosystem to make packages faster and easier to use: Unlock performance wins for developers through free-threaded Python where we leverage Python 3.13s support for concurrent programming (made possible by removing the Global Interpreter Lock (GIL)).
KDnuggets
MAY 5, 2025
A step-by-step guide to securing a FastAPI machine learning applications' endpoints with native authentication and user management.
Uber Engineering
MAY 5, 2025
Goodbye resource leaks! Learn how the FixrLeak framework leverages GenAI and AST-level analysis to automatically detect and fix resource leaks in large-scale Java applications at Uber.
Snowflake
MAY 8, 2025
Data providers want their data available to their customers, no matter where in the world or on which cloud service provider the customer is located. However, egress costs can contribute up to 70% of total data transfer costs. Providers have historically had to balance the desire to increase the availability of their data to any relevant Snowflake regions with the need to manage egress costs.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
databricks
MAY 5, 2025
Atlassian recently partnered with Databricks to power new data sharing capabilities from Atlassian Analytics, using the Delta Sharing protocol.
ArcGIS
MAY 14, 2025
Simplify CAD and BIM integration in ArcGIS Pro 3.5 for model federation and sharing workflows. Bring content into context for analysis and collaboration.
Engineering at Meta
MAY 22, 2025
Meta is partnering with V.tal to extend the Malbec subsea cable to Porto Alegre, Brazil by 2027. With this new extension, Malbec will become the first subsea cable to land in the state of Rio Grande do Sul, bringing more connectivity to millions of people in Southern Brazil and neighboring countries. Malbec will improve the scale and reliability of digital infrastructure in Porto Alegre, establishing it as a digital hub and improving online experiences across Southern Brazil, Argentina, Chile, P
KDnuggets
MAY 13, 2025
Are you eager to enhance your artificial intelligence skills? We've curated a fantastic selection of free AI books to aid your learning journey!
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Cloudyard
MAY 4, 2025
Read Time: 3 Minute, 21 Second Snowflake and DBT (Data Build Tool) are two of the most powerful players in the modern data stack. Traditionally, DBT is known for transformations and Snowflake for its cloud-native warehousing. When combined, DBT handles your transformations and Snowflake provides the storage and compute power. This combination streamlines ETL processes, increases flexibility, and reduces manual coding.
Data Engineering Weekly
MAY 25, 2025
Dagster Components is now here Components provides a modular architecture that enables data practitioners to self-serve while maintaining engineering quality. Built for the AI era, Components offers compartmentalized code units with proper guardrails that prevent "AI slop" while supporting code generation. See how it works in 4 easy steps Onehouse: ClickHouse vs StarRocks vs Presto vs Trino vs Apache Spark™ — Comparing Analytics Engines As we adopt the Lakehouse architecture more and
databricks
MAY 14, 2025
Today, we are excited to announce that we have agreed to acquire Neon, a developer-first, serverless Postgres company.
ArcGIS
MAY 14, 2025
Learn more about exciting new functionality and improvements made to ArcGIS Utility Network with the 2025 Network Management Release.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
dbt Developer Hub
MAY 2, 2025
The AI revolution is herebut are we ready? Across the world, the excitement around AI is undeniable. Discussions on large language models, agentic workflows, and how AI is set to transform every industry abound, yet real-world use cases of AI in production remain few and far between. A common issue blocking people from moving AI use cases to production is an ability to evaluate the validity of AI responses in a systematic and well governed way.
KDnuggets
MAY 14, 2025
In this article, you'll master 10 essential Linux file system commands. This guide provides helpful examples to make working with files easier.
Precisely
MAY 2, 2025
Earlier this year, Precisely announced Data Link : an ecosystem of pre-linked datasets from leading data providers. And now, I get to say that Dun & Bradstreet , a leading global provider of business decisioning data and a nalytics , has joined this groundbreaking program. Ive previously shared how Preciselys Data Link program will streamline the biggest challenges that businesses face today when it comes to onboarding and reconciling third-party datasets across multiple providers think com
Data Engineering Weekly
MAY 4, 2025
Try Apache Airflow® 3 on Astro Airflow 3 is here and has never been easier or more secure. Spin up a new 3.0 deployment on Astro to test DAG versioning, backfills, event-driven scheduling, and more. Get started → Editor’s Note: OpenXData Conference - 2025 - A Free Virtual Event A free virtual event on open data architectures - Iceberg, Hudi, lakehouses, query engines, and more.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Expert insights. Personalized for you.
52 
Let's personalize your content