How to build a Stacking ensemble in Machine Learning: step by step

João Barros 23 de October de 2026 6 min read

This tutorial shows how to build a Stacking ensemble in Machine Learning to combine multiple models and improve classification performance. Stacking is particularly useful when individual models have different strengths (for example, one captures linear relationships well, another captures non-linear interactions) and you want to obtain a more robust overall performance often with lower error than each model alone.

Prerequisites

Python 3.8+
scikit-learn installed (pip install scikit-learn)
pandas and numpy for data handling
basic knowledge of classification and cross-validation

Having an environment with a reasonable CPU helps (Stacking with cv=5 and multiple estimators can take time). For larger datasets consider using n_jobs=-1 to parallelize. Examples below use the Iris dataset, small and didactic.

Step 1: Understand what Stacking is and when to use it

Stacking is an ensemble that trains multiple base models and then a final model (meta-model) that learns how to combine the predictions of those base models. Instead of simply voting or averaging, the meta-model observes the predictions of the base models (usually out-of-fold predictions) and learns weights/combinations, reducing bias and/or variance.

When to use:

If you have models with complementary behaviors (e.g., RandomForest captures interactions, LogisticRegression is stable for linear relationships).
If you want to improve generalization without relying only on heavy hyperparameter tuning.
When you have enough data to create meta-features without overfitting — for example, for Iris (150 samples) it works well as a demonstration; in real problems you typically have hundreds or thousands of samples.

Key technique: use cross-validation (cv) to generate out-of-fold predictions from the base models, avoiding information leakage from training directly into the meta-model.

Step 2: Prepare the data

Load a simple dataset (e.g., Iris) and split into train/test. Ensure cleaning and encoding of categorical variables if present. Important: for classifiers sensitive to scale (KNN, SVM, LogisticRegression) apply StandardScaler inside a pipeline.

import pandas as pd
import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

data = load_iris()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = pd.Series(data.target)

# With test_size=0.2 and 150 samples -> 120 train / 30 test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

Stratify keeps class proportions in train/test. For larger datasets consider additional validation with cross_val_score to measure robustness.

Step 3: Choose base models and meta-model

Choose diverse models to capture different types of patterns. Practical example: LogisticRegression (linear), RandomForestClassifier (tree ensemble) and KNeighborsClassifier (distance-based method). The meta-model is usually simple — LogisticRegression is a good initial choice to avoid overfitting.

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import StackingClassifier

estimators = [
    ('rf', RandomForestClassifier(n_estimators=50, random_state=42)),
    ('knn', KNeighborsClassifier(n_neighbors=5)),
    ('lr', LogisticRegression(max_iter=200))
]

meta_clf = LogisticRegression()

Notes: n_estimators=50 is chosen for speed; in production use 100-500 depending on budget. KNN with k=5 is typical; try k between 3 and 15. A simple meta_clf prevents the ensemble from becoming too complex.

Step 4: Build and train the Stacking

Use StackingClassifier from scikit-learn: define estimators and final_estimator. The cv parameter determines how the meta-features are generated (by default, each base model makes out-of-fold predictions in a 5-fold cv, producing a set of predictions used to train the meta-model).

stack = StackingClassifier(estimators=estimators, final_estimator=meta_clf, cv=5, n_jobs=-1)
stack.fit(X_train, y_train)

Practical explanation: with cv=5, each base model is trained 5 times on 80% of the data and predicts the remaining 20%; the concatenated predictions form the features for the meta-model. For multi-class classification you can use predict_proba (it generates more columns) or predict (one column per model), depending on the StackingClassifier parameter.

Step 5: Evaluate the ensemble and compare with individual models

Evaluate with metrics: accuracy, precision, recall, f1 and confusion matrix. Compare Stacking with each base model to confirm improvement. Typical examples with Iris: Stacking accuracy can rise to 0.96-0.98 while individual models range between 0.90 and 0.96 — modest but consistent gains.

from sklearn.metrics import accuracy_score, classification_report

# Predictions
y_pred_stack = stack.predict(X_test)
print('Stacking accuracy:', accuracy_score(y_test, y_pred_stack))
print(classification_report(y_test, y_pred_stack))

# Evaluate individual models
for name, est in estimators:
    est.fit(X_train, y_train)
    y_pred = est.predict(X_test)
    print(f"{name} accuracy:", accuracy_score(y_test, y_pred))

Interpret results: if Stacking does not improve, check if base models are too similar or if the meta-model is overfitting.

Step 6: Avoid overfitting and common mistakes

Common mistakes: using the same training fold to generate meta-features without proper validation, not normalizing features when needed, and using a meta-model that is too complex (e.g., a very deep tree) that learns noise. Best practices:

Use pipelines with StandardScaler for KNN and LogisticRegression.
Validate the ensemble with cross_val_score to estimate robustness (e.g., mean cross-val accuracy with cv=5).
Experiment with predict_proba in StackingClassifier to provide more information to the meta-model.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score

# Example with pipeline for KNN
estimators_pipe = [
    ('rf', RandomForestClassifier(n_estimators=50, random_state=42)),
    ('knn', make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))),
    ('lr', make_pipeline(StandardScaler(), LogisticRegression(max_iter=200)))
]
stack_pipe = StackingClassifier(estimators=estimators_pipe, final_estimator=LogisticRegression(), cv=5, n_jobs=-1)
print('Cross-val accuracy (stack):', np.mean(cross_val_score(stack_pipe, X_train, y_train, cv=5)))

Check the outcome

Confirm that Stacking has accuracy equal to or higher than the individual models and that the confusion matrix distributes errors acceptably. Use cross_val_score to validate robustness and check if the meta-model adds value — for example, by looking at the coefficients if it is a LogisticRegression (larger values correspond to greater weight on certain base models' predictions).

Conclusion

Stacking allows combining strengths of multiple models to improve predictions; the next step is to test with larger datasets or different models (e.g., XGBoost, LightGBM) and try alternative meta-models. Practical tip: start with simple models and cv=5, measure percentage gains (e.g., +1-3% in accuracy) and only then increase complexity to avoid introducing overfitting. Which combination will you try next?