KDnuggets
JUNE 4, 2025
The direct integration of AI-powered SQL operators and support for references to arbitrary files in object stores with mechanisms like ObjectRef represent a fundamental shift in how we interact with data.
ProjectPro
JUNE 6, 2025
Data engineering is the foundation for data science and analytics by integrating in-depth knowledge of data technology, reliable data governance and security, and a solid grasp of data processing. Data engineers create data pipelines, which are the infrastructural designs for modern data analytics, to facilitate smooth data analysis. Data engineers need to meet various requirements to build data pipelines.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
ProjectPro
JUNE 6, 2025
Amazon Web Services (AWS) provides a wide range of tools and services for handling enormous amounts of data. The two most popular AWS data engineering services for processing data at scale for analytics operations are Amazon EMR and AWS Glue. Executing ETL tasks in the cloud is fast and simple with AWS Glue. EMR is a more powerful big data processing solution to provide real-time data streaming for machine learning applications.
databricks
JUNE 2, 2025
Apache Iceberg v3, now approved by the Apache Iceberg community, introduces advanced new features and data types.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Start Data Engineering
JUNE 3, 2025
1. Introduction 2. SQL techniques 3. Query optimization 4. Data modeling & data flow 5. Conclusion 6. Further reading 1. Introduction Most data engineering job descriptions these days expect “knowledge of advanced SQL,” but ask any data engineer that question, and you will get a different answer every time. Are you Frustrated that “advanced SQL” ebooks or Udemy courses aren’t all that advanced!
KDnuggets
JUNE 3, 2025
Machine learning pipelines help turn data into predictions. Apache Spark makes it easy to build these pipelines for big data.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
databricks
JUNE 2, 2025
Apache Iceberg v3, now approved by the Apache Iceberg community, introduces advanced new features and data types.
Snowflake
JUNE 4, 2025
Each year, Im genuinely inspired as I reflect on the incredible impact of our Snowflake Partner Network (SPN). Its an honor to celebrate the dedication and innovation our partners demonstrate through the Snowflake Partner Awards. Together, were not just driving customer successwere building a vibrant, ever-growing connected ecosystem around the AI Data Cloud.
KDnuggets
JUNE 4, 2025
In this article we explore 10 generative AI concepts that are key to understanding, whether you are an engineer, user, or consumer of generative AI.
ProjectPro
JUNE 6, 2025
‘Man and machine together can be better than the human’ All thanks to deep learning frameworks like PyTorch, Tensorflow, Keras, Caffe, and DeepLearning4j for making machines learn like humans with special brain-like architectures known as Neural Networks. The war of deep learning frameworks has two prominent competitors- PyTorch vs Tensorflow because the other frameworks have not yet been adopted widely.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Pinterest Engineering
JUNE 6, 2025
Xue Xia | Machine Learning Engineer, Home Feed Ranking; Saurabh Vishwas Joshi | Principal Engineer, ML Platform; Kousik Rajesh | Machine Learning Engineer, Applied Science; Kangnan Li | Machine Learning Engineer, Core ML Infrastructure; Yangyi Lu | Machine Learning Engineer, Home Feed Ranking; Nikil Pancha | (formerly) Machine Learning Engineer, Applied Science; Dhruvil Deven Badani | Engineering Manager, Home Feed Ranking; Jiajing Xu | Engineering Manager, Applied Science; Pong Eksombatchai | P
Snowflake
JUNE 1, 2025
Today, Snowflake advances our vision to be the ultimate platform for data-driven innovation with our announcement that we have agreed to acquire Crunchy Data, a leading provider of trusted, open source PostgreSQL technology. This move will allow us to deliver Snowflake Postgres, a new kind of Postgres designed to power the most demanding, mission-critical AI and transactional systems at enterprise scale and with enterprise confidence.
KDnuggets
JUNE 3, 2025
What are the most important cognitive biases, and how do you overcome them to make your data analysis as objective as possible?
ProjectPro
JUNE 6, 2025
As a big data architect or a big data developer, when working with Microservices-based systems, you might often end up in a dilemma whether to use Apache Kafka or RabbitMQ for messaging. Rabbit MQ vs. Kafka - Which one is a better message broker? You might find some articles across the web that conclude that Apache Kafka is better than RabbitMQ and few others that mention RabbitMQ to be more reliable than Kafka.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
ThoughtSpot
JUNE 2, 2025
For data analysts and engineers, the journey from raw data to actionable business insights for business users is never as simple as it sounds. The semantic layer is a critical component in this process, serving as the bridge between complex data sources and the business logic required for informed decision-making. However, not all semantic layers are created equal, and the evolving landscape of AI-powered analytics demands a new approach.
databricks
JUNE 6, 2025
Most enterprises sit on a massive amount of unstructured data—documents, images, audio, video—yet only a fraction ever turns into actionable insight.
KDnuggets
JUNE 4, 2025
Explore this unified API for file uploading, document parsing, embedding models, vector store, and a retrieval pipeline.
ProjectPro
JUNE 6, 2025
With over 54 repositories and 20k stars, Streamlit is an open-source Python framework for developing and distributing web apps for data science and machine learning projects. Engineers can easily create highly dynamic online applications based on their data, machine learning models, etc., using Streamlit. Let us explore a few exciting Streamlit python project ideas for data scientists and data engineers. 5 Streamlit Python Project Ideas to Try Your Hands-On Here are five streamlit projects in Py
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
DataKitchen
JUNE 6, 2025
Data Quality Testing: A Shared Resource for Modern Data Teams In today’s AI-driven landscape, where data is king, every role in the modern data and analytics ecosystem shares one fundamental responsibility: ensuring that incorrect data never reaches business customers. Whether you’re a Data Engineer building ETL pipelines, a Data Scientist developing predictive models, or a Data Steward ensuring compliance, we all want the same outcome: data that is trustworthy, accurate, and underst
RudderStack
JUNE 6, 2025
Launch high-ROI personalization projects that drive engagement and conversions without complex engineering.
KDnuggets
JUNE 6, 2025
Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter 5 Error Handling Patterns in Python (Beyond Try-Except) Stop letting errors crash your app. Master these 5 Python patterns that handle failures like a pro!
ProjectPro
JUNE 6, 2025
Early in 2020, when Myntra launched its visual product search for the first time, it created waves in e-commerce. With this new feature, the customers no longer had to spend hours searching for a dress similar to the one they came across randomly in an advertisement. All they had to do was take a picture/screenshot and upload it on Myntra; the app would automatically fetch outfits similar to the picture.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Data Engineering Weekly
JUNE 1, 2025
Dagster for MLOps: Deep Dive into AI Orchestration Learn what it really takes to run production-grade ML systems—without breaking your architecture or compliance efforts. Join Dagster and Neurospace to learn: - How to build AI pipelines with orchestration baked in - How to track data lineage for audits and traceability - Tips for designing compliant workflows under the EU AI Act Register for the technical session DuckDB: DuckLake - SQL as a Lakehouse Format DuckDB announced a new open tabl
Snowflake
JUNE 6, 2025
Eight months. Over one thousand submissions from more than one hundred countries. Ten semi-finalists. Three finalists. Seven heart-pounding minutes as the judges deliberated. And finally, one winner: we are thrilled to announce that Lumilinks is the 2025 Snowflake Startup Challenge Winner! The judges zeroed in on the potential of several aspects of Lumilinks’ product and business strategy, including its focus on solving business users’ problems and finding impact with more conventional businesse
KDnuggets
JUNE 5, 2025
How about some alternative options for a data career? Learn about five non-standard career paths, required skills, and how to learn them for free.
ProjectPro
JUNE 6, 2025
The demand for data-related roles has increased massively in the past few years. Companies are actively seeking talent in these areas, and there is a huge market for individuals who can manipulate data, work with large databases and build machine learning algorithms. While data science is the most hyped-up career path in the data industry, it certainly isn't the only one.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
databricks
JUNE 2, 2025
Iceberg v3, now approved by the Apache Iceberg community, introduces advanced new features and data types.
Uber Engineering
JUNE 4, 2025
What if chatbots could deliver near-human precision? Learn how we adapted Genie, Ubers on-call copilot, to use enhanced agentic RAG for improved accuracy.
KDnuggets
JUNE 6, 2025
Stay ahead in 2025 with the latest OCR models optimized for speed, accuracy, and versatility in handling everything from scanned documents to complex layouts.
ProjectPro
JUNE 6, 2025
With the global data volume projected to surge from 120 zettabytes in 2023 to 181 zettabytes by 2025, PySpark's popularity is soaring as it is an essential tool for efficient large scale data processing and analyzing vast datasets. This clearly indicates that the need for Big Data Engineers and Specialists would surge in the future years. Source: ExplodingTopics Originally built in Scala , Spark now supports Python through PySpark , enabling seamless work with Resilient Distributed Datasets (RDD
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
127 
Let's personalize your content