How to detect duplicate data in Machine Learning: step by step

João Barros 21 de September de 2026 4 min read

This tutorial shows how to detect and handle duplicate data in Machine Learning to improve training quality and reduce bias. I will explain why duplications are problematic and present a practical step-by-step method with Python, pandas and scikit-learn.

Prerequisites

Python 3.8+ with pandas and scikit-learn installed
A CSV file or DataFrame with records that may repeat
Basic knowledge of DataFrame manipulation (pandas)

Step 1: Understand what types of duplicates exist

Duplicates can be exactly identical rows, records with repeated keys (ID) or nearly identical records (small differences in values). Identifying the type guides the strategy: remove, aggregate or flag.

Step 2: Load the data

Load the CSV into a DataFrame and view the first rows. This helps identify key columns and the presence of NaNs.

import pandas as pd

df = pd.read_csv('dados.csv')
print(df.head())
print(df.shape)

Step 3: Exact duplicates (identical rows)

Use pandas to count exactly identical rows and, if it makes sense, remove them. Before removing, save the count for auditing.

# contar duplicados exactos (mantém a primeira ocorrência)
num_dup = df.duplicated(keep='first').sum()
print(f'Duplicados exactos: {num_dup}')

# remover duplicados exactos
df_no_exact_dup = df.drop_duplicates(keep='first')
print(df_no_exact_dup.shape)

Step 4: Duplicates by key (ID) — locate and decide

If there is an ID column that should be unique, identify repeated IDs. Then decide: keep the first, the last, or aggregate (for example mean).

# encontrar IDs duplicados
id_col = 'customer_id'  # ajusta para a tua coluna
dups = df[df.duplicated(subset=[id_col], keep=False)].sort_values(id_col)
print(dups.head())

# exemplo: manter o registo mais recente por data
df[id_col] = df[id_col].astype(str)
df['date'] = pd.to_datetime(df['date'])
# ordena por date e remove duplicados mantendo o mais recente
df_latest = df.sort_values('date').drop_duplicates(subset=[id_col], keep='last')
print(df_latest.shape)

Step 5: Nearly identical duplicates (fuzzy duplicates)

For records with small differences (typos or variations) use fuzzy matching techniques. A simple approach: create hashes of normalized columns or use Levenshtein distance. Here I show a version with normalization and comparison by similarity using scikit-learn and fingerprints.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# exemplo para colunas nome + morada
df['name_addr'] = (df['name'].fillna('').str.lower().str.replace(r'\W+', ' ', regex=True)
                   + ' ' + df['address'].fillna('').str.lower().str.replace(r'\W+', ' ', regex=True))

vec = TfidfVectorizer().fit_transform(df['name_addr'])
sim = cosine_similarity(vec[:1000], vec[:1000])  # limita para evitar OOM

# encontra pares com similaridade alta
threshold = 0.85
pairs = []
for i in range(sim.shape[0]):
    for j in range(i+1, sim.shape[1]):
        if sim[i, j] > threshold:
            pairs.append((i, j, sim[i, j]))
print('pares fuzzy semelhantes:', len(pairs))

Step 6: Aggregate or flag duplicates for training

Decide whether you will remove, aggregate (for example mean of metrics) or flag with a column. For models, sometimes it is preferable to keep and add an is_duplicate column so the model can learn patterns.

# exemplo: adicionar flag de duplicado por chave
df['is_duplicate_by_id'] = df.duplicated(subset=[id_col], keep='first')

# exemplo: agregar métricas por ID
agg = df.groupby(id_col).agg({'value': 'mean', 'is_active': 'max'}).reset_index()
print(agg.head())

Step 7: Integrate into the Machine Learning pipeline

Include duplicate cleaning before the train/test split. If you remove duplicates after the split, you risk having identical records in both sets (data leakage).

# limpeza antes do split
from sklearn.model_selection import train_test_split

df_clean = df_no_exact_dup.copy()
X = df_clean.drop('target', axis=1)
y = df_clean['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Verify the result

Confirm that there are no longer problematic duplicates: count exact and key-based duplicates, check suspicious fuzzy pairs and ensure there is no leakage between train and test (IDs in common).

# verificar duplicados exactos e por ID
print('exact dup after:', X_train.duplicated().sum() + X_test.duplicated().sum())
print('IDs comuns entre treino e teste:', set(X_train[id_col]).intersection(set(X_test[id_col])))

Conclusion

Handling duplicates improves the quality of the training set and reduces bias and leakage. Next steps: automate detection with reusable functions, experiment with thresholds for fuzzy matching and log changes in an ETL pipeline. Tip: always back up the original data before removing or aggregating — do you prefer to flag or delete?