How to build an automatic missing-value imputation detector in Machine Learning
Learning to automatically identify where to impute values in datasets is useful to keep pipelines robust and avoid inappropriate imputations. This tutorial shows how to build a detector that suggests the imputation method per column (mean/median/mode/model) using Python and scikit-learn.
Prerequisites
- Python 3.8+ installed
- Libraries: pandas, scikit-learn, numpy
- Basic knowledge of pandas and scikit-learn
Step 1: Understand the problem and the strategy
The goal is to build a classifier that, for each column with missing values, recommends an imputation method: mean, median, mode or model (use a predictive model). This is useful when working with many datasets/columns and wanting to automate imputation choice based on column characteristics.
Step 2: Create a synthetic training set
We will generate synthetic columns with different properties (symmetric numeric, skewed numeric, categorical, with outliers) and define the best imputation by rule. This serves as the label to train the detector.
import numpy as np
import pandas as pd
from sklearn.utils import shuffle
def gen_numeric(size, dist='normal'):
if dist=='normal':
return np.random.normal(loc=0, scale=1, size=size)
if dist=='skew':
return np.random.exponential(scale=1, size=size)
if dist=='outlier':
x = np.random.normal(size=size)
x[:int(size*0.02)] = x[:int(size*0.02)] * 20
return x
def gen_categorical(size, n_cat=3):
return np.random.choice([f'c{i}' for i in range(n_cat)], size=size)
ROWS=500
X=[]
y=[]
for _ in range(400):
typ = np.random.choice(['normal','skew','outlier','cat'])
if typ=='cat':
col = gen_categorical(ROWS, n_cat=np.random.randint(2,6))
# best imput: mode
label='mode'
s = pd.Series(col)
features={'dtype':'cat','n_unique':s.nunique(),'freq_top':s.value_counts(normalize=True).iloc[0]}
else:
col = gen_numeric(ROWS, dist=typ)
s = pd.Series(col)
# decide best imput: mean if near-normal, median if skewed/outliers
skew = s.skew()
if abs(skew) < 0.5:
label='mean'
elif abs(skew) < 2:
label='median'
else:
label='median'
features={'dtype':'num','mean':s.mean(),'median':s.median(),'std':s.std(),'skew':skew}
# introduce missingness pattern
mask = np.random.rand(ROWS) < 0.1
s[mask]=np.nan
# compute summary features for detector
feat = {
'pct_missing': s.isna().mean(),
'n_unique': s.nunique(dropna=True),
}
if features['dtype']=='num':
feat.update({
'mean': features['mean'], 'median': features['median'], 'std': features['std'], 'skew': features['skew']
})
else:
feat.update({'freq_top': features['freq_top']})
X.append(feat)
y.append(label)
df = pd.DataFrame(X).fillna(0)
labels = pd.Series(y)
# shuffle
df, labels = shuffle(df, labels, random_state=42)
Step 3: Train a simple classifier to recommend imputation
We use a RandomForestClassifier to predict the label (mean/median/mode/model). In this version we do not include the 'model' option to simplify; that can be added with more advanced criteria.
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
X_train, X_test, y_train, y_test = train_test_split(df, labels, test_size=0.2, random_state=42)
clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
print(classification_report(y_test, pred))
Step 4: Create a practical function that applies the recommendation
This function receives a real DataFrame, computes the same features per column and applies the classifier to suggest imputation. It also shows an example of applying SimpleImputer when the recommendation is mean/median/mode.
from sklearn.impute import SimpleImputer
def column_features(series):
s = series
feat = {
'pct_missing': s.isna().mean(),
'n_unique': s.nunique(dropna=True)
}
if pd.api.types.is_numeric_dtype(s):
feat.update({'mean': s.mean(), 'median': s.median(), 'std': s.std(), 'skew': s.skew()})
else:
top = s.value_counts(normalize=True).iloc[0] if s.dropna().size>0 else 0
feat.update({'mean':0,'median':0,'std':0,'skew':0,'freq_top':top})
return feat
def recommend_imputation(df, clf):
recs = {}
for col in df.columns:
feat = column_features(df[col])
# align columns
f_df = pd.DataFrame([feat]).fillna(0)
pred = clf.predict(f_df)[0]
recs[col]=pred
return recs
# Example of application
sample = pd.DataFrame({
'A': np.append(np.random.normal(size=95), [np.nan]*5),
'B': np.append(np.random.exponential(size=95), [np.nan]*5),
'C': np.append(np.random.choice(['x','y','z'], size=95), [np.nan]*5)
})
recs = recommend_imputation(sample, clf)
print(recs)
# Apply simple imputation according to recommendation (mean/median/mode)
imputed = sample.copy()
for col, method in recs.items():
if method in ('mean','median'):
strategy = method
imp = SimpleImputer(strategy=strategy)
imputed[col] = imp.fit_transform(imputed[[col]])
elif method=='mode':
imp = SimpleImputer(strategy='most_frequent')
imputed[col] = imp.fit_transform(imputed[[col]])
else:
# fallback: median
imp = SimpleImputer(strategy='median')
imputed[col] = imp.fit_transform(imputed[[col]])
Check the result
Confirm that the classifier has good accuracy with classification_report and inspect the printed recommendations. In example data, symmetric numeric columns should recommend mean, skewed ones median and categorical ones mode. Also verify that the imputed DataFrame has no NaN values.
Conclusion
This automatic imputation detector helps standardize decisions in ETL pipelines and reduce manual choices. Next steps: add a 'model' label when a column requires regression/classification to impute, use cross-validation to tune the classifier and integrate into scikit-learn Pipeline. Tip: test the detector on real datasets and compare the impact of imputation choices on final model performance — which technique changes the metrics the most?