How to build a false positive detector in Machine Learning

João Barros 31 de October de 2026 5 min read

This tutorial shows how to build a false positive detector in Machine Learning to reduce excessive alarms in a classification system. Knowing how to identify false positives is useful to improve operational efficiency and prioritize real alerts.

Prerequisites

Python 3.8+ and pip installed
Libraries: pandas, scikit-learn, numpy, matplotlib (installation shown)
Basic knowledge of classification and evaluation (precision, recall, ROC)

Step 1: Understand the problem and prepare the data

A false positive detector learns to predict when a positive class prediction from the primary model is likely incorrect. We need a dataset where each example has: original features, the primary model's prediction (pred_label) and the true label (true_label). We'll create a synthetic example to follow the flow.

# instalar dependências (executar no terminal, se necessário)  # pip install pandas scikit-learn numpy matplotlib    import numpy as np  import pandas as pd  from sklearn.model_selection import train_test_split    # exemplo sintético  rng = np.random.RandomState(42)  n = 2000  X = pd.DataFrame({'f1': rng.normal(size=n), 'f2': rng.uniform(size=n)})  # rótulo verdadeiro: evento raro dependente de f1  y = (X['f1'] + 0.5 * X['f2'] + rng.normal(scale=0.5, size=n) > 0.5).astype(int)    # simulamos um modelo primário com algumas previsões (com erros)  from sklearn.ensemble import RandomForestClassifier  m_primary = RandomForestClassifier(n_estimators=50, random_state=1)  X_train, X_rest, y_train, y_rest = train_test_split(X, y, test_size=0.5, random_state=1)  m_primary.fit(X_train, y_train)  pred_prob = m_primary.predict_proba(X_rest)[:, 1]  pred_label = (pred_prob > 0.4).astype(int)  # threshold do modelo primário

Step 2: Build the false positive detector dataset

The detector is a second model that takes as input: original features, probabilities or decisions from the primary model and aims to predict whether a positive prediction is a false positive (pred_label=1 & true_label=0). We filter for cases where the primary predicted positive.

# montar dataset para o detector  X_det = X_rest.copy().reset_index(drop=True)  y_det = ((pred_label == 1) & (y_rest.reset_index(drop=True) == 0)).astype(int)  # 1 = falso positivo  X_det['pred_prob'] = pred_prob  X_det['pred_label'] = pred_label    # consideramos só os exemplos em que o primário previu positivo (casos a decidir)  mask = X_det['pred_label'] == 1  X_det_pos = X_det[mask].reset_index(drop=True)  y_det_pos = y_det[mask].reset_index(drop=True)    X_det_pos.shape, y_det_pos.value_counts().to_dict()

Step 3: Train a simple detector model

We use a lightweight classifier (e.g., LogisticRegression) to predict false positives. This allows estimating the probability of being a false positive and applying a threshold to filter alarms.

from sklearn.linear_model import LogisticRegression  from sklearn.metrics import precision_score, recall_score, roc_auc_score    X_train_d, X_test_d, y_train_d, y_test_d = train_test_split(X_det_pos, y_det_pos, test_size=0.3, random_state=2)    features = ['f1', 'f2', 'pred_prob']  detector = LogisticRegression(max_iter=1000)  detector.fit(X_train_d[features], y_train_d)    y_pred_prob = detector.predict_proba(X_test_d[features])[:, 1]  y_pred = (y_pred_prob > 0.5).astype(int)    print('AUC detector:', round(roc_auc_score(y_test_d, y_pred_prob), 3))  print('Precision:', round(precision_score(y_test_d, y_pred), 3))  print('Recall:', round(recall_score(y_test_d, y_pred), 3))

Step 4: Integrate the detector into the inference pipeline

In production, when the primary model makes a positive prediction, we call the detector. If the detector indicates a high risk of false positive, we mark the alert as low priority or suppress it. Example of batch application over X_rest.

# aplicar detector a todo o conjunto onde primário previu positivo  all_df = X_rest.reset_index(drop=True).copy()  all_df['pred_prob'] = pred_prob  all_df['pred_label'] = pred_label    mask_pos = all_df['pred_label'] == 1  probs_fp = np.zeros(len(all_df))  probs_fp[mask_pos] = detector.predict_proba(all_df[mask_pos][features])[:, 1]    # decidir suprimir se probabilidade de falso positivo > 0.6  suppress = probs_fp > 0.6  all_df['suppress_alert'] = suppress.astype(int)    all_df[['f1','f2','pred_prob','pred_label','suppress_alert']].head()

Verify the result

Confirm that the detector reduces false positives without removing many true positives. We measure metrics before and after: false positive rate (FPR) and the primary model's recall on real positives.

from sklearn.metrics import confusion_matrix    # métricas sem detector (considerando threshold primário)  tp = ((all_df['pred_label']==1) & (y_rest.reset_index(drop=True)==1)).sum()  fp = ((all_df['pred_label']==1) & (y_rest.reset_index(drop=True)==0)).sum()  fn = ((all_df['pred_label']==0) & (y_rest.reset_index(drop=True)==1)).sum()  print('Antes - TP, FP, FN:', tp, fp, fn)    # aplicar supressão  pred_after = all_df['pred_label'].copy()  pred_after[all_df['suppress_alert']==1] = 0  # suprimido vira negativo    tp_a = ((pred_after==1) & (y_rest.reset_index(drop=True)==1)).sum()  fp_a = ((pred_after==1) & (y_rest.reset_index(drop=True)==0)).sum()  fn_a = ((pred_after==0) & (y_rest.reset_index(drop=True)==1)).sum()  print('Depois - TP, FP, FN:', tp_a, fp_a, fn_a)    print('Redução de FP:', fp - fp_a)

Conclusion

A false positive detector is a complementary model that reduces unnecessary alarms. Next steps: calibrate thresholds, use contextual features (time, user), try more robust models (XGBoost) and validate the impact in production. Tip: start by measuring the cost of each false positive before defining the threshold.