(+351) 21 24 10006  ·  info@bconcepts.pt
Carnaxide, Lisbon

How to build an automatic missing-value imputation detector in Machine Learning

João Barros 16 de November de 2026 5 min read

Learning to automatically identify where to impute values in datasets is useful to keep pipelines robust and avoid inappropriate imputations. This tutorial shows how to build a detector that suggests the imputation method per column (mean/median/mode/model) using Python and scikit-learn.

Prerequisites

  • Python 3.8+ installed
  • Libraries: pandas, scikit-learn, numpy
  • Basic knowledge of pandas and scikit-learn

Step 1: Understand the problem and the strategy

The goal is to build a classifier that, for each column with missing values, recommends an imputation method: mean, median, mode or model (use a predictive model). This is useful when working with many datasets/columns and wanting to automate imputation choice based on column characteristics.

Step 2: Create a synthetic training set

We will generate synthetic columns with different properties (symmetric numeric, skewed numeric, categorical, with outliers) and define the best imputation by rule. This serves as the label to train the detector.

import numpy as np
import pandas as pd
from sklearn.utils import shuffle

def gen_numeric(size, dist='normal'):
    if dist=='normal':
        return np.random.normal(loc=0, scale=1, size=size)
    if dist=='skew':
        return np.random.exponential(scale=1, size=size)
    if dist=='outlier':
        x = np.random.normal(size=size)
        x[:int(size*0.02)] = x[:int(size*0.02)] * 20
        return x

def gen_categorical(size, n_cat=3):
    return np.random.choice([f'c{i}' for i in range(n_cat)], size=size)

ROWS=500
X=[]
y=[]
for _ in range(400):
    typ = np.random.choice(['normal','skew','outlier','cat'])
    if typ=='cat':
        col = gen_categorical(ROWS, n_cat=np.random.randint(2,6))
        # best imput: mode
        label='mode'
        s = pd.Series(col)
        features={'dtype':'cat','n_unique':s.nunique(),'freq_top':s.value_counts(normalize=True).iloc[0]}
    else:
        col = gen_numeric(ROWS, dist=typ)
        s = pd.Series(col)
        # decide best imput: mean if near-normal, median if skewed/outliers
        skew = s.skew()
        if abs(skew) < 0.5:
            label='mean'
        elif abs(skew) < 2:
            label='median'
        else:
            label='median'
        features={'dtype':'num','mean':s.mean(),'median':s.median(),'std':s.std(),'skew':skew}
    # introduce missingness pattern
    mask = np.random.rand(ROWS) < 0.1
    s[mask]=np.nan
    # compute summary features for detector
    feat = {
        'pct_missing': s.isna().mean(),
        'n_unique': s.nunique(dropna=True),
    }
    if features['dtype']=='num':
        feat.update({
            'mean': features['mean'], 'median': features['median'], 'std': features['std'], 'skew': features['skew']
        })
    else:
        feat.update({'freq_top': features['freq_top']})
    X.append(feat)
    y.append(label)

df = pd.DataFrame(X).fillna(0)
labels = pd.Series(y)
# shuffle
df, labels = shuffle(df, labels, random_state=42)

Step 3: Train a simple classifier to recommend imputation

We use a RandomForestClassifier to predict the label (mean/median/mode/model). In this version we do not include the 'model' option to simplify; that can be added with more advanced criteria.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

X_train, X_test, y_train, y_test = train_test_split(df, labels, test_size=0.2, random_state=42)
clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
print(classification_report(y_test, pred))

Step 4: Create a practical function that applies the recommendation

This function receives a real DataFrame, computes the same features per column and applies the classifier to suggest imputation. It also shows an example of applying SimpleImputer when the recommendation is mean/median/mode.

from sklearn.impute import SimpleImputer

def column_features(series):
    s = series
    feat = {
        'pct_missing': s.isna().mean(),
        'n_unique': s.nunique(dropna=True)
    }
    if pd.api.types.is_numeric_dtype(s):
        feat.update({'mean': s.mean(), 'median': s.median(), 'std': s.std(), 'skew': s.skew()})
    else:
        top = s.value_counts(normalize=True).iloc[0] if s.dropna().size>0 else 0
        feat.update({'mean':0,'median':0,'std':0,'skew':0,'freq_top':top})
    return feat

def recommend_imputation(df, clf):
    recs = {}
    for col in df.columns:
        feat = column_features(df[col])
        # align columns
        f_df = pd.DataFrame([feat]).fillna(0)
        pred = clf.predict(f_df)[0]
        recs[col]=pred
    return recs

# Example of application
sample = pd.DataFrame({
    'A': np.append(np.random.normal(size=95), [np.nan]*5),
    'B': np.append(np.random.exponential(size=95), [np.nan]*5),
    'C': np.append(np.random.choice(['x','y','z'], size=95), [np.nan]*5)
})
recs = recommend_imputation(sample, clf)
print(recs)

# Apply simple imputation according to recommendation (mean/median/mode)
imputed = sample.copy()
for col, method in recs.items():
    if method in ('mean','median'):
        strategy = method
        imp = SimpleImputer(strategy=strategy)
        imputed[col] = imp.fit_transform(imputed[[col]])
    elif method=='mode':
        imp = SimpleImputer(strategy='most_frequent')
        imputed[col] = imp.fit_transform(imputed[[col]])
    else:
        # fallback: median
        imp = SimpleImputer(strategy='median')
        imputed[col] = imp.fit_transform(imputed[[col]])

Check the result

Confirm that the classifier has good accuracy with classification_report and inspect the printed recommendations. In example data, symmetric numeric columns should recommend mean, skewed ones median and categorical ones mode. Also verify that the imputed DataFrame has no NaN values.

Conclusion

This automatic imputation detector helps standardize decisions in ETL pipelines and reduce manual choices. Next steps: add a 'model' label when a column requires regression/classification to impute, use cross-validation to tune the classifier and integrate into scikit-learn Pipeline. Tip: test the detector on real datasets and compare the impact of imputation choices on final model performance — which technique changes the metrics the most?