How to prune trees in Machine Learning: step by step

João Barros 07 de October de 2026 6 min read

This explains how to reduce overfitting in tree models through pruning and why this improves generalization. We will look at pre-pruning and post-pruning techniques with a practical Python example using DecisionTreeClassifier. The central idea is simple: very deep trees overfit to training noise; by pruning irrelevant branches you reduce variance and obtain a simpler, more interpretable model.

Prerequisites

Python 3.8+ installed
Libraries: scikit-learn, pandas, numpy, matplotlib (e.g.: pip install scikit-learn pandas numpy matplotlib)
Basic knowledge of classification and train/test (train_test_split, metrics such as accuracy)
Machine with basic resources: Iris has 150 rows, modern CPUs are sufficient

Step 1: Why pruning in Machine Learning

Decision trees often overfit to training data. Pruning removes unimportant branches to simplify the tree and improve performance on new data. There are two common approaches: pre-pruning (by limiting max_depth, min_samples_leaf, etc.) and post-pruning (cost-complexity pruning). Pre-pruning acts during training (prevents growth), while post-pruning grows the full tree and then cuts branches with low benefit. In practice, when the difference between training and test accuracy is large (for example >10 percentage points), it is a signal to apply pruning.

Step 2: Prepare an example dataset

Use a standard dataset to demonstrate: the Iris dataset or a simulated dataset. Here we use Iris to simplify reproduction. Iris has 150 samples (50 per class), so with test_size=0.3 you get 105 training samples and 45 test samples — enough to see overfitting trends without complicating things.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import pandas as pd

data = load_iris()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
# X_train.shape => (105, 4), X_test.shape => (45, 4)

Step 3: Train a tree without pruning (baseline)

Train a DecisionTreeClassifier without constraints to observe overfitting. Evaluate train and test. Typically an unconstrained tree reaches 100% on the training set (perfect fit) and may have 95–99% on the test set for Iris; the difference reveals the degree of overfitting.

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

clf = DecisionTreeClassifier(random_state=42)
clf.fit(X_train, y_train)

train_acc = accuracy_score(y_train, clf.predict(X_train))
test_acc = accuracy_score(y_test, clf.predict(X_test))
print(f"Baseline train acc: {train_acc:.3f}, test acc: {test_acc:.3f}")

Step 4: Pre-pruning — use parameters to control complexity

Try limiting max_depth, min_samples_leaf and min_samples_split. These parameters prevent the tree from growing arbitrarily. For example, with max_depth=3 and min_samples_leaf=5 you typically reduce depth and number of nodes drastically; on Iris this tends to keep test accuracy close to the baseline (e.g. 0.93–0.97) but with a much smaller train/test gap.

clf_pre = DecisionTreeClassifier(random_state=42, max_depth=3, min_samples_leaf=5)
clf_pre.fit(X_train, y_train)

print("Pre-pruning train:", accuracy_score(y_train, clf_pre.predict(X_train)))
print("Pre-pruning test:", accuracy_score(y_test, clf_pre.predict(X_test)))

Step 5: Post-pruning — cost-complexity pruning (ccp_alpha)

scikit-learn implements cost-complexity pruning. First obtain a sequence of possible ccp_alpha values and then choose the best one by validation performance. The algorithm typically returns 5–30 alpha values, from very small (almost no pruning) to large (tree reduced to the root node). Evaluate each alpha and choose the one that maximizes the metric on the validation or test set. Warning: avoid choosing alpha based only on the final test set; use cross-validation when possible.

# obter caminho de poda
path = DecisionTreeClassifier(random_state=42).cost_complexity_pruning_path(X_train, y_train)
ccp_alphas = path.ccp_alphas

clfs = []
for ccp in ccp_alphas:
    clf_tmp = DecisionTreeClassifier(random_state=42, ccp_alpha=ccp)
    clf_tmp.fit(X_train, y_train)
    clfs.append((ccp, clf_tmp))

# avaliar cada alpha
import numpy as np
train_scores = [accuracy_score(y_train, m.predict(X_train)) for _, m in clfs]
test_scores = [accuracy_score(y_test, m.predict(X_test)) for _, m in clfs]

best_idx = int(np.argmax(test_scores))
best_alpha, best_model = clfs[best_idx]
print(f"Melhor ccp_alpha: {best_alpha}, test acc: {test_scores[best_idx]:.3f}")

Step 6: Choose the final model and validate with cross-validation

Confirm the robustness of the chosen alpha using cross-validation to avoid selecting by chance on the test set. For example, do 5-fold CV and compare the mean and standard deviation of the accuracies. Consistent values (e.g. mean=0.95, std<0.03) indicate stability.

from sklearn.model_selection import cross_val_score

final_clf = DecisionTreeClassifier(random_state=42, ccp_alpha=best_alpha)
scores = cross_val_score(final_clf, X, y, cv=5)
print("CV accuracy:", scores.mean(), scores)

Check the result

Compare the metrics: if the pruned model has similar test performance but a smaller train-test gap, pruning worked. Also check tree simplicity (depth and number of nodes) to confirm reduced complexity. E.g.: Baseline depth: 5, nodes: 35 vs Pruned depth: 3, nodes: 9 — this translates to a more interpretable model less sensitive to noise.

print("Baseline depth:", clf.get_depth(), "nodes:", clf.tree_.node_count)
print("Pruned depth:", best_model.get_depth(), "nodes:", best_model.tree_.node_count)

Conclusion

Pruning (pre-pruning and post-pruning) helps reduce overfitting and obtain more interpretable and robust tree models. Next steps: test with more complex datasets (e.g. hundreds of thousands of records), use k-fold validation to select parameters and compare with ensembles (RandomForest, GradientBoosting) that reduce variance without as much manual tuning. Practical tip: when you see a large difference between train and test, try max_depth and min_samples_leaf first; if you need more precision use cost-complexity pruning (ccp_alpha) with cross-validation.