Confessions of a Data Guy

APRIL 6, 2024

Some time ago I wrote a very simple comparison of switching from Pandas to Polars, I didn’t put much real effort into it, yet it was popular, so this is my attempt at trying to expand on that topic a […] The post How to JOIN datasets in Polars … compared to Pandas. appeared first on Confessions of a Data Guy.

Datasets 113

Datasets IT Data Data Engineer 113

Analytics Vidhya

JULY 6, 2023

Building an accurate machine learning and AI model requires a high-quality dataset. Introduction In this era of Generative Al, data generation is at its peak.

Datasets 243

Datasets Machine Learning Building Algorithm 243

Towards Data Science

JANUARY 30, 2025

Best runs for furthest-from-centroid selection compared to full dataset. In my recent experiments with the MNIST dataset, thats exactly what happened. Data PruningResults The plot above shows the models accuracy compared to the training dataset size when using the most effective pruning method Itested. Image byauthor.

Database-centric 106

Database-centric Datasets Data Architecture 106

Cloudera

NOVEMBER 13, 2024

Fine Tuning Studio enables users to track the location of all datasets, models, and model adapters for training and evaluation. We can import this dataset on the Import Datasets page. The goal is to train an adapter for this base model that gives it better predictive capabilities for our specific dataset. Model Selection.

Datasets 87

Datasets Machine Learning Coding Data Preparation 87

KDnuggets

AUGUST 27, 2024

Dealing with outliers is crucial in data preprocessing. This guide covers multiple ways to handle outliers along with their pros and cons.

Datasets 113

Datasets Data Python 113

Start Data Engineering

SEPTEMBER 18, 2024

Understand input datasets available 3.1.2. Define what the output dataset will look like 3.1.3. Define checks to ensure the output dataset is usable 3.2. Introduction 2. Parts of data engineering 3.1. Requirements 3.1.1. Define SLAs so stakeholders know what to expect 3.1.4. Identify what tool to use to process data 3.3.

Project 240

Project Building Datasets Architecture 240

KDnuggets

FEBRUARY 3, 2025

Fine-tune the DeepSeek model step by step. even if you're new to LLMs!

Datasets 131

Datasets 131

databricks

OCTOBER 17, 2023

Introduction Large Language Models (LLMs) have given us a way to generate text, extract information, and identify patterns in industries from healthcare to.

Datasets 123

Datasets Healthcare Data 123

Netflix Tech

NOVEMBER 12, 2024

The Counter Abstraction API resembles Java’s AtomicInteger interface: AddCount/AddAndGetCount : Adjusts the count for the specified counter by the given delta value within a dataset. We create one such Rollup table per dataset and use Cassandra as our persistent store. The delta value can be positive or negative.

Datasets 101

Datasets Computer Science Systems Kafka 101

Start Data Engineering

JULY 26, 2024

Running checks on one dataset 5.2. Checks involving the current dataset and its historical data 5.3. Checks involving comparing datasets 5. TL;DR: How the greatexpectations library works 4.1. greatexpectations quick setup 5. From an implementation perspective, there are four types of tests 5.1.

Datasets 208

Datasets Data Project IT 208

Yelp Engineering

JANUARY 21, 2025

These models handle large tabular datasets with small parameter spaces, requiring innovative data solutions. At Yelp, we encountered challenges that prompted us to enhance the training time of our ad-revenue generating models, which use a Wide and Deep Neural Network architecture for predicting ad click-through rates (pCTR).

Datasets 104

Datasets Architecture Data Solutions Data 104

Analytics Vidhya

FEBRUARY 23, 2023

Introduction Big Data is a large and complex dataset generated by various sources and grows exponentially. It is so extensive and diverse that traditional data processing methods cannot handle it. The volume, velocity, and variety of Big Data can make it difficult to process and analyze.

Big Data Tools 242

Big Data Tools Big Data Datasets Data 242

Christophe Blefari

MARCH 15, 2024

Matthaus gives the dlt vision about creating the foundation for developers to be able to create sources in a wink creating a large ecosystem of APIs datasets easily maintainable. This is Croissant. Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML.

Metadata 272

Metadata Data Software Engineering Software Engineer 272

Analytics Vidhya

JUNE 12, 2023

Tajinder’s passion for unraveling hidden patterns in complex datasets has driven impactful outcomes, transforming raw data into actionable intelligence. Introduction Meet Tajinder, a seasoned Senior Data Scientist and ML Engineer who has excelled in the rapidly evolving field of data science.

Machine Learning 235

Machine Learning Engineering Raw Data Data Science 235

databricks

NOVEMBER 21, 2024

While large language models (LLMs) are increasingly adept at solving general tasks, they can often fall short on specific domains that are dissimilar.

Datasets 88

Datasets Building 88

Data Engineering Weekly

JANUARY 15, 2025

Images and Videos: Computer vision algorithms must analyze visual content and deal with noisy, blurry, or mislabeled datasets. They are responsible for designing, implementing, and maintaining robust, scalable data pipelines that transform raw unstructured data—text, images, videos, and more—into high-quality, AI-ready datasets.

Data Engineering 62

Data Engineering Data Engineer Unstructured Data Engineering 62

Precisely

APRIL 7, 2025

After my (admittedly lengthy) explanation of what I do as the EVP and GM of our Enrich business, she summarized it in a very succinct, but new way: “Oh, you manage the appending datasets.” Matching accuracy: Matching records between datasets is complex. ” That got me thinking.

Retail 74

Retail Datasets Data Portfolio 74

Precisely

MAY 2, 2025

Earlier this year, Precisely announced Data Link : an ecosystem of pre-linked datasets from leading data providers. Historically, seamless integration across these datasets has been extremely difficult they’re not standardized across providers, which leaves your team with tedious manual mapping and troubleshooting.

Datasets 68

Datasets Insurance Portfolio Retail 68

Analytics Vidhya

FEBRUARY 5, 2023

Big data is nothing but the vast volume of datasets measured in terabytes or petabytes or even more. Introduction In this technical era, Big Data is proven as revolutionary as it is growing unexpectedly. According to the survey reports, around 90% of the present data was generated only in the past two years.

Hadoop 210

Hadoop Big Data Datasets Data 210

DataKitchen

NOVEMBER 5, 2024

The Medallion architecture is a framework that allows data engineers to build organized and analysis-ready datasets in a lakehouse environment. For instance, suppose a new dataset from an IoT device is meant to be ingested daily into the Bronze layer. How do you ensure data quality in every layer?

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

ThoughtSpot

APRIL 22, 2025

Level 2: Understanding your dataset To find connected insights in your business data, you need to first understand what data is contained in the dataset. Spotter quickly translates your datasets into business-friendly terminology so business users can confidently explore their data through natural language conversations.

BI 59

BI Datasets Business Intelligence Raw Data 59

Data Engineering Weekly

JANUARY 8, 2025

This hybrid approach empowers enterprises to efficiently handle massive datasets while maintaining flexibility and reducing operational overhead. These advancements address enterprises' real-world challenges, such as maintaining fresh, up-to-date datasets and optimizing for high-throughput scenarios. Exploring Apache Hudi 1.0:

Data Lake 62

Data Lake Datasets Retail Data Ingestion 62

Precisely

MARCH 20, 2025

Data enrichment is the process of augmenting your organizations internal data with trusted, curated third-party datasets. The Multiple Data Provider Challenge If you rely on data from multiple vendors, you’ve probably run into a major challenge: the datasets are not standardized across providers. What is data enrichment?

Insurance 68

Insurance Datasets Data Programming 68

Analytics Vidhya

FEBRUARY 6, 2023

It is a core component of the Apache Hadoop ecosystem and allows for storing and processing large datasets across multiple commodity servers. Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data.

Data Storage 187

Data Storage Big Data Hadoop Datasets 187

Christophe Blefari

JANUARY 11, 2025

A large international scientist collaboration released The Well : 2 massive datasets from physics simulation (15TB) to astronomical scientific data (100TB). It's easily readable (mildly large ~10 pages) and gives metrics about the performance plateau that we start to see at scale.

Data 130

Data Data Warehouse Coding Programming Language 130

Analytics Vidhya

JUNE 12, 2023

Tajinder’s passion for unraveling hidden patterns in complex datasets has driven impactful outcomes, transforming raw data into actionable intelligence. Introduction Meet Tajinder, a seasoned Senior Data Scientist and ML Engineer who has excelled in the rapidly evolving field of data science.

Machine Learning 185

Machine Learning Engineering Raw Data Data Science 185

DataKitchen

FEBRUARY 20, 2025

Now With Actionable, Automatic, Data Quality Dashboards Imagine a tool that can point at any dataset, learn from your data, screen for typical data quality issues, and then automatically generate and perform powerful tests, analyzing and scoring your data to pinpoint issues before they snowball. Announcing DataOps Data Quality TestGen 3.0:

Datasets 73

Datasets Metadata Data Government 73

The Pragmatic Engineer

APRIL 6, 2023

Most jobs vendors have a ton of ‘junk jobs,’ so we spent a fair bit of time culling the dataset to jobs that are unique. During processing, we match companies, titles and more, with our dataset. We put the jobs data into Amazon S3. We have a network of Lamdas that fire any time new data is added.

Software Engineering 236

Software Engineering Software Engineer Datasets Database 236

Engineering at Meta

JANUARY 22, 2025

The following diagram illustrates how the lineage graph has expanded: Collecting data flow signals for the AI system For our AI systems, we collect lineage signals by tracking relationships between various assets, such as input datasets, features, models, workflows, and inferences.

Data Warehouse 115

Data Warehouse SQL Programming Language Data 115

Netflix Tech

FEBRUARY 14, 2025

Architecture Overview The first pivotal step in managing impressions begins with the creation of a Source-of-Truth (SOT) dataset. This foundational dataset is essential, as it supports various downstream workflows and enables a multitude of usecases.

Kafka 66

Kafka Datasets Metadata Utilities 66

KDnuggets

OCTOBER 30, 2024

Each project, from beginner tasks like Image Classification to advanced ones like Anomaly Detection, includes a link to the dataset and source code for easy access and implementation.

Project 153

Project Datasets Coding Accessible 153

Engineering at Meta

APRIL 28, 2025

Machine learning models : trained on labeled datasets using supervised learning and improved through unsupervised learning to identify patterns and anomalies in unlabeled data. For example, in the data warehouse, it’s represented as a Dataset – an in-code Python class capturing the asset’s schema and metadata.

Metadata 79

Metadata Data Utilities Data Warehouse 79

KDnuggets

SEPTEMBER 12, 2024

The GitHub repository includes up-to-date learning resources, research papers, guides, popular tools, tutorials, projects, and datasets.

Datasets 150

Datasets Project 150

Data Engineering Weekly

MARCH 2, 2025

I found the blog to be a fresh take on the skill in demand by layoff datasets. Our internal benchmark of the NYC dataset shows a 48% performance gain of smallpond over Spark!! Whether you use Datasets already or want to get started, we've got you covered! link] Mehdio: DuckDB goes distributed?

Data Engineering 62

Data Engineering Data Engineer Engineering Datasets 62

Edureka

JANUARY 21, 2025

First, navigate to Table View The dataset labeled “sample dataset” is shown below. Data Measure = CALCULATE ( SUM ( 'sample dataset'[Revenue] )) 6. Data Measure = CALCULATE ( SUM ( 'sample dataset'[Revenue] ), FILTER ( 7. Data Measure = CALCULATE ( SUM ( 'sample dataset'[Revenue] ), FILTER ( ALL ( 8.

BI 52

BI Datasets Business Intelligence Certification 52

KDnuggets

JULY 18, 2024

Tips and tricks on improving machine learning model performance on diverse and unseen datasets.

Machine Learning 146

Machine Learning Datasets 146

Monte Carlo

NOVEMBER 26, 2024

Synthetic data works by leveraging models to create artificial datasets that reflect what someone might find organically (in some alternate reality where more data actually exists), and then using that new data to train their own models. But is synthetic data a long-term solution? Probably not.

Data Engineer 105

Data Engineer Data Engineering Engineering Unstructured Data 105