Edureka

MAY 8, 2025

When you purchase online, do you ever find yourself pondering how your tastes get changed into suggestions for products that are uniquely suited to you? Or how self-driving cars get through very complicated situations with amazing accuracy? These are the ways that data engineering improves our lives in the real world. The field of data engineering turns unstructured data into ideas that can be used to change businesses and our lives.

Data Engineer 40

Data Engineer Data Engineering Engineering Telecommunication 40

databricks

MAY 5, 2025

We are experiencing an unprecedented pace of technological innovation driven by AI and data.

Data 63

Data Technology 63

KDnuggets

MAY 5, 2025

Learn how to implement a parallelization process in your data pipeline.

Data Pipeline 81

Data Pipeline Building Data Process 81

Cloudyard

MAY 4, 2025

Read Time: 3 Minute, 21 Second Snowflake and DBT (Data Build Tool) are two of the most powerful players in the modern data stack. Traditionally, DBT is known for transformations and Snowflake for its cloud-native warehousing. When combined, DBT handles your transformations and Snowflake provides the storage and compute power. This combination streamlines ETL processes, increases flexibility, and reduces manual coding.

ETL Tools 59

ETL Tools Raw Data SQL Python 59

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

InData Labs

MAY 6, 2025

Artificial intelligence services have been a hot topic for the last decade. It is hard to find an area or industry nowadays that hasnt at least tried to use this relatively new tool in its work. However, there is one thing that makes it possible for AI to exist. This thing is DATA. Without high-quality. AI data collection guide InData Labs.

Data Collection 60

Data Collection Data IT BI 60

KDnuggets

MAY 8, 2025

STAR isnt suitable for technical jobs, so how do you answer behavioral interview questions while still showing youre a data scientist?

Data 73

Data 73

databricks

MAY 5, 2025

Why migrate from Netezza to Databricks? The limitations of traditional enterprise data warehouse (EDW) appliances like Netezza are becoming increasingly apparent.

Data Warehouse 69

Data Warehouse Systems Data 69

Uber Engineering

MAY 5, 2025

Migrating tech stacks at Ubers scale isnt easy. Learn how we migrated our stateless container orchestration platform to Kubernetes and operate it at a scale of 3 million cores with 1.5 million pod launches daily.

IT 56

IT 56

Precisely

MAY 8, 2025

Where do you need your data most and how easily can you access it? I ask because getting trusted third-party data into your preferred environment has always been harder than it should be. It often involves downloading files, managing cumbersome ingestion processes, troubleshooting formatting issues, and manually stitching everything together only to repeat the process with every new update.

Cloud 64

Cloud Google Cloud IT Datasets 64

KDnuggets

MAY 5, 2025

A step-by-step guide to securing a FastAPI machine learning applications' endpoints with native authentication and user management.

Machine Learning 122

Machine Learning Management 122

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data Workflow

Jesse Anderson

MAY 5, 2025

I n this episode of Unapologetically Technical, I interview Shane Murray, Field CTO at Monte Carlo Data. Shane shares his compelling journey from studying math and finance in Sydney, Australia, to leading AI strategy at a major data observability company in New York. We explore his early work in choice modeling and pioneering online multivariate experimentation long before A/B testing became mainstream, including fascinating examples from cruise lines, American Express, and even cultural surpris

Unstructured Data 130

Unstructured Data Finance Metadata Architecture 130

databricks

MAY 5, 2025

Atlassian recently partnered with Databricks to power new data sharing capabilities from Atlassian Analytics, using the Delta Sharing protocol.

Data 107

Data 107

Engineering at Meta

MAY 5, 2025

Meta and Quantsight have improved key libraries in the Python Ecosystem. There is plenty more to do and we invite the community to help with our efforts. Well look at two key efforts in Pythons packaging ecosystem to make packages faster and easier to use: Unlock performance wins for developers through free-threaded Python where we leverage Python 3.13s support for concurrent programming (made possible by removing the Global Interpreter Lock (GIL)).

Python 74

Python Coding Programming Project 74

KDnuggets

MAY 7, 2025

Learn to build AI agents, fine-tune reasoning models, and master practical AI skills with these courses.

Building 140

Building 140

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Uber Engineering

MAY 5, 2025

Goodbye resource leaks! Learn how the FixrLeak framework leverages GenAI and AST-level analysis to automatically detect and fix resource leaks in large-scale Java applications at Uber.

Java 72

Java 72

Snowflake

MAY 8, 2025

Data providers want their data available to their customers, no matter where in the world or on which cloud service provider the customer is located. However, egress costs can contribute up to 70% of total data transfer costs. Providers have historically had to balance the desire to increase the availability of their data to any relevant Snowflake regions with the need to manage egress costs.

AWS 70

AWS Cloud Algorithm Data Security 70

Data Engineering Weekly

MAY 4, 2025

Try Apache Airflow® 3 on Astro Airflow 3 is here and has never been easier or more secure. Spin up a new 3.0 deployment on Astro to test DAG versioning, backfills, event-driven scheduling, and more. Get started → Editor’s Note: OpenXData Conference - 2025 - A Free Virtual Event A free virtual event on open data architectures - Iceberg, Hudi, lakehouses, query engines, and more.

Data Engineer 67

Data Engineer Data Engineering Engineering Java 67

KDnuggets

MAY 6, 2025

Write better Python by mastering the built-in and standard library data structures for clean, efficient, and elegant code.

Python 90

Python Data Coding 90

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Engineering at Meta

MAY 8, 2025

Meta and NVIDIA collaborated to accelerate vector search on GPUs by integrating NVIDIA cuVS into Faiss v1.10 , Metas open source library for similarity search. This new implementation of cuVS will be more performant than classic GPU-accelerated search in some areas. For inverted file (IVF) indexing, NVIDIA cuVS outperforms classical GPU-accelerated IVF build times by up to 4.7x; and search latency is reduced by as much as 8.1x.

Algorithm 89

Algorithm Datasets Machine Learning Database 89

databricks

MAY 7, 2025

As organizations consolidate analytics workloads to Databricks, they often need to adapt traditional data warehouse techniques.

Data Warehouse 68

Data Warehouse SQL Data 68

Netflix Tech

MAY 7, 2025

Enhancing Member Experience Through Strategic Collaboration Ozzie Sutherland , Iroro Orife , Chih-Wei Wu , BhanuSrikanth At Netflix, delivering the best possible experience for our members is at the heart of everything we do, and we know we cant do it alone. Thats why we work closely with a diverse ecosystem of technology partners, combining their deep expertise with our creative and operational insights.

Digital Media 89

Digital Media Entertainment Media Machine Learning 89

KDnuggets

MAY 5, 2025

Master element-wise operations, comparisons, logic, aggregation, and broadcasting using NumPy ufuncs for high-performance array processing.

Process 71

Process 71

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineer

Precisely

MAY 5, 2025

Delivering seamless, personalized experiences for customers across channels continues to be a priority for organizations across industries. To make this goal a reality, they seek out powerful customer communications management (CCM) solutions. However, theres often a debate on whether to build a custom in-house solution or purchase an enterprise-grade platform.

Building 59

Building Telecommunication Certification Data Security 59

databricks

MAY 5, 2025

The imperative for modernization Traditional database solutions like SQL Server have struggled to keep up with the demands of modern data workloads due to a

SQL 69

SQL Database Data 69

Start Data Engineering

MAY 5, 2025

1. Introduction 2. Step-by-step process to solve any SQL interview question 2.1. Define what the input data is and how they are related 2.2. Understand the input table’s grain, foreign keys, and how they relate to each other 2.3. Define the dimensions and metrics required for the output 2.4. Filter/Join/Group by input columns to get the output dimension and metrics 3.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

KDnuggets

MAY 3, 2025

Is vibe coding the future? Or a way for non-coder to dabble? Or a tool for programmers to make their lives a little easier? Or none of these things?

Coding 109

Coding IT 109

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Edureka

MAY 8, 2025

When you purchase online, do you ever find yourself pondering how your tastes get changed into suggestions for products that are uniquely suited to you? Or how self-driving cars get through very complicated situations with amazing accuracy? These are the ways that data engineering improves our lives in the real world. The field of data engineering turns unstructured data into ideas that can be used to change businesses and our lives.

Data Engineer 52

Data Engineer Data Engineering Engineering Telecommunication 52

Netflix Tech

MAY 9, 2025

Kinesh Satiya Introduction In a digital advertising platform, a robust feedback system is essential for the lifecycle and success of an ad campaign. This system comprises of diverse sub-systems designed to monitor, measure, and optimize ad campaigns. At Netflix, we embarked on a journey to build a robust event processing platform that not only meets the current demands but also scales for future needs.

Process 73

Process Building Metadata Kafka 73

databricks

MAY 5, 2025

Before making architectural decisions, its worth revisiting the broader migration strategy.

Professional Services 59

Professional Services Data Warehouse Architecture IT 59

KDnuggets

MAY 8, 2025

Breaking into data science has never been easy. In this tutorial, well make your life easier by providing you with a step-by-step roadmap for data science beginners.

Data Science 94

Data Science Data 94

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer