KDnuggets

MARCH 25, 2025

Want to make data cleaning more enjoyable? These pandas one-liners for data cleaning will help you get more done with less!

Data 113

Data 113

Edureka

MARCH 26, 2025

Imputation in statistics means replacing missing data with different numbers. “Unit imputation” means replacing a whole data point, while “item imputation” means replacing part of a data point. Missing information can cause bias, make data analysis harder, and lower efficiency. These are the three main problems it creates. Imputation is a way to handle missing data instead of simply removing cases with missing values, as missing information can make data analysis more dif

Medical 40

Medical Datasets Data Analysis Machine Learning 40

WeCloudData

MARCH 27, 2025

Artificial intelligence (AI) is no longer a futuristic concept, but a powerful tool reshaping education, offering personalized learning and improved outcomes. This aligns perfectly with the Education 4.0 Framework, which aims to transform education for the Fourth Industrial Revolution. By strategically implementing AI, we can directly contribute to realizing this vision, ensuring learners are prepared […] The post AI in Education: Transforming Learning in the Digital Age appeared first on

Education 52

Education Machine Learning Data 52

Confluent

MARCH 24, 2025

Learn how Confluent Champion Syed solves complex problems for customersand how Confluent's collaborative culture keeps him motivated.

Engineering 93

Engineering 93

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Pinterest Engineering

MARCH 26, 2025

Megan Blake, Usha Amrutha Nookala, Jeremy Browning, Sarah Tao, AJ Oxendine, SiddarthMalreddy Overview &Context The holiday shopping season presents a unique challenge: helping millions of Pinners discover and save perfect gifts across a vast sea of possibilities. While Pinterest has always been a destination for gift inspiration, our data showed that users were facing two key friction points: discovery overwhelm and fragmented wishlists.

Building 59

Building Engineering Algorithm Systems 59

Edureka

MARCH 26, 2025

With the advent of generative AI, the creative and innovative capabilities of machines have been greatly enhanced. It all comes down to sophisticated neural network architectures that try to imitate human intellect in order to make realistic films, images, and text. Transformers power conversational agents and GANs generate photorealistic art; these models are altering businesses.

Raw Data 40

Raw Data Architecture Deep Learning Finance 40

DareData

MARCH 25, 2025

Youve seen it before: a company starts deploying AI but struggles to move beyond the most high-level use cases. Yet, current AI technology has the potential to power near-fully automated business workflowsso whats missing? The challenge lies in how you manage the error. In critical business processes, AI errors can be extremely costly and deploying these systems becomes far more complex.

Process 52

Process Pharmaceutical Banking Systems 52

Edureka

MARCH 26, 2025

Imagine a future where connecting with technology is as natural as conversing with a friend. That is the idea behind DeepBrain AI, a groundbreaking platform that is altering how people engage with AI. DeepBrain AI enables organizations and individuals to effortlessly create, communicate, and develop, with lifelike virtual avatars and intelligent automation.

Education 45

Education Media Deep Learning Machine Learning 45

KDnuggets

MARCH 27, 2025

This guide provides a detailed, step-by-step explanation of how to connect ChatGPT with Google Sheets, along with practical examples and advanced features to make the most of this integration.

111

111

Simon Späti

MARCH 28, 2025

The database landscape has reached 394 ranked systems across multiple categoriesrelational, document, key-value, graph, search engine, time series, and the rapidly emerging vector databases. As AI applications multiply quickly, vector technologies have become a frontier that data engineers must explore. The essential questions to be answered are: When should you choose specialized vector solutions like Pinecone, Weaviate, or Qdrant over adding vector extensions to established databases like Post

Technology 130

Technology PostgreSQL MySQL Database 130

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Pipeline-centric

Netflix Tech

MARCH 28, 2025

By Ko-Jen Hsiao , Yesu Feng and Sudarshan Lamkhede Motivation Netflixs personalized recommender system is a complex system, boasting a variety of specialized machine learned models each catering to distinct needs including Continue Watching and Todays Top Picks for You. (Refer to our recent overview for more details). However, as we expanded our set of personalization algorithms to meet increasing business needs, maintenance of the recommender system became quite costly.

Metadata 85

Metadata Bytes Entertainment Data Mining 85

Snowflake

MARCH 24, 2025

Todays organizations recognize the importance of data-driven decision-making, but the process of setting up a data pipeline thats easy to use, easy to track and easy to trust continues to be a complex challenge. Reducing time to success allows organizations to see immediate value from their data investments and scale up productivity. Our investment in DataOps.live , a SaaS platform for data engineering and operations, will help Snowflake users accelerate that timeline.

Cloud 68

Cloud Data Pipeline Data Workflow Data Engineering 68

Precisely

MARCH 28, 2025

In today’s fast-paced digital world, maintaining high standards and addressing contemporary requirements is crucial for any company. One of our customers, a leading automotive manufacturer, relies on the IBM Z for its computing power and rock-solid reliability. However, they faced a growing challenge: integrating and accessing data across a complex environment.

Data Integration 68

Data Integration Pipeline-centric Database-centric Kafka 68

databricks

MARCH 26, 2025

Were excited to announce that Anthropic Claude 3.7 Sonnet is now natively available in Databricks across AWS, Azure, and GCP. For the first time, you.

AWS 143

AWS 143

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

KDnuggets

MARCH 28, 2025

Learn model serving, CI/CD, ML orchestration, model deployment, local AI, and Docker to streamline ML workflows, automate pipelines, and deploy scalable, portable AI solutions effectively.

Machine Learning 136

Machine Learning Project 136

Data Engineering Weekly

MARCH 23, 2025

Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. Data Council 2025 is set for April 22-24 in Oakland, CA. As a special perk for Data Engineering Weekly subscribers, you can use the code dataeng20 for an exclusive 20% discount on tickets!

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Waitingforcode

MARCH 25, 2025

Data contracts was a hot topic in the data space before LLMs and GenAI came out. They promised a better world with less communication issues between teams, leading to more reliable and trustworthy data. Unfortunately, the promise has been too hard to put into practice. Has been, or should I write "was"?

Project 130

Project Data 130

databricks

MARCH 25, 2025

Large language models are challenging to adapt to new enterprise tasks. Prompting is error-prone and achieves limited quality gains, while fine-tuning requires large amounts of.

Data 135

Data 135

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Wayne Yaddow

MARCH 28, 2025

Unleashing GenAIEnsuring Data Quality at Scale (Part2) Transitioning from individual repository source systems to consolidated AI LLM pipelines, the importance of automated checks, end-to-end observability, and compliance with enterprise businessrules. T Introduction There are several opportunities (and needs!) to improve operational effectiveness and analytical capacity when integrating data repository systems for AI Large Language Model (LLM) pipelines.

Data Integration 52

Data Integration Data Governance Datasets Government 52

KDnuggets

MARCH 24, 2025

Check out the story of a Reddit user who has achieved success by following 7 simple rules.

125

125

Monte Carlo

MARCH 28, 2025

AI can do a lot these days. At this very moment, an army of SaaS companies are hard at work infusing AI assistants and copilots into every horizontal B2B workflow currently known to humankind. ChatGPT can summarize the web to help sales prospects. Gemini can polish Google documents for research teams. GitHub copilot can even code alongside you like your own pocket-sized Steve Wozniak.

Coding 52

Coding Systems Data Pipeline ETL Tools 52

WeCloudData

MARCH 24, 2025

Retrieval-augmented generation (RAG) is an AI cutting-edge approach that combines the power of traditional retrieval-based techniques with the capabilities of a generative large language model (LLM) to enhance the accuracy and relevance of AI-generated content. Instead of depending entirely on pre-trained knowledge, RAG incorporates external knowledge sources, such as documents or databases, to enhance the […] The post What is Retrieval-Augmented Generation (RAG)?

Database 52

Database Data Science Data Engineer Data Engineering 52

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Wayne Yaddow

MARCH 28, 2025

Unleashing GenAIEnsuring Data Quality at Scale (Part1) Transitioning from isolated repository systems to consolidated AI LLM pipelines Photo by Joshua Sortino on Unsplash Introduction This blog is based on insights from articles in Database Trends and Applications, Feb/Mar 2025 ( DBTA Journal ). Across these informative articles, one message rings loud and clear: Artificial intelligence (AI)and large language models (LLMs) in particularrequires relentless attention to dataquality.

Government 52

Government Data Governance Data Data Integration 52

KDnuggets

MARCH 26, 2025

Check out this step-by-step guide to building a speech-to-text system with PyTorch & Hugging Face.

Systems 118

Systems Building 118

Edureka

MARCH 27, 2025

CycleGAN is a powerful Generative Adversarial Network (GAN) optimized for unpaired image-to-image translation. CycleGAN, unlike traditional GANs, does not require paired datasets, in which each image in one domain corresponds to an image in another. This makes it extremely useful for tasks that require collecting paired data, which can be difficult or impossible.

Datasets 40

Datasets Medical Architecture Algorithm 40

WeCloudData

MARCH 26, 2025

Natural Language Processing (NLP) is transforming the manufacturing industry by enhancing decision-making, enabling intelligent automation, and improving quality control. As Industry 4.0 continues to evolve, NLP is becoming an essential tool for gaining insights from unstructured data, increasing productivity, and reducing human error. Lets learn more about the use cases of NLP in manufacturing and […] The post Natural Language Processing(NLP) in Manufacturing appeared first on WeCloudData

Manufacturing 52

Manufacturing Process Unstructured Data Data 52

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

DataKitchen

MARCH 26, 2025

Announcing Actionable, Automated, & Agile Data Quality Scorecards Are you ready to unlock the power of influence to transform your organizations data qualityand become the hero your data deserves? Watch the previously recorded webinar unveiling our latest innovation: Data Quality Scorecards, powered by our AI-driven DataOps Data Quality TestGen software.

Data 52

Data Programming Technology IT 52

KDnuggets

MARCH 25, 2025

In this article, I will go through 5 pointers on how to help you secure your dream job.

Machine Learning 115

Machine Learning 115

Confessions of a Data Guy

MARCH 28, 2025

Running dbt on Databricks has never been easier. The integration between dbtcore and Databricks could not be more simple to set up and run. Wondering how to approach running dbt models on Databricks with SparkSQL? Watch the tutorial below. The post dbt on Databricks appeared first on Confessions of a Data Guy.

Data 100

Data Big Data Data Engineer Data Engineering 100

Picnic Engineering

MARCH 28, 2025

TL;DR Over the past decade, Picnic transformed its approach to dataevolving from a single, all-purpose data team into multiple specialized teams using a lean, scalable tech stack. We empowered analysts by giving them access and training to the same tools as engineers, dramatically increasing speed and impact. Our investments in a lakeless data warehouse, modern analytics platform, and strong master data practices have made data a core strategic capability.

Data Warehouse 40

Data Warehouse PostgreSQL Python Machine Learning 40

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer