How to do feature selection with Recursive Feature Elimination (RFE) in Machine Learning

João Barros 08 de November de 2026 4 min read

This tutorial shows how to do feature selection with Recursive Feature Elimination (RFE) in Machine Learning using scikit-learn. Feature selection reduces noise, speeds up training and helps interpret models — useful when you have many variables.

Prerequisites

Python 3.8+ with scikit-learn, pandas and numpy installed
Basic knowledge of Python and Machine Learning concepts (features, labels, overfitting)
An environment to run code (Jupyter, VS Code, or terminal)

Step 1: Understand why to use RFE

RFE (Recursive Feature Elimination) iteratively removes the least important features according to an estimator (for example, LogisticRegression or RandomForest). It's useful when you want a compact subset of features that maintains or improves performance. Common mistakes: choosing a weak estimator, not validating selection with cross-validation, or using RFE on a small dataset without repetition.

Step 2: Prepare example data

We create a synthetic dataset to demonstrate. I used make_classification to control feature relevance.

from sklearn.datasets import make_classification
import pandas as pd

X, y = make_classification(n_samples=1000, n_features=20, n_informative=5,
                           n_redundant=2, n_repeated=0, random_state=42)
cols = [f'feat_{i}' for i in range(X.shape[1])]
df_X = pd.DataFrame(X, columns=cols)
df_y = pd.Series(y, name='target')

Step 3: Do a baseline model evaluation

Evaluate a model without selection to have a reference. Use cross_val_score to obtain the mean and standard deviation.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

clf = LogisticRegression(max_iter=1000, solver='liblinear')
scores = cross_val_score(clf, df_X, df_y, cv=5, scoring='accuracy')
print('Acc base:', scores.mean(), '+/-', scores.std())

Step 4: Apply simple RFE

Use RFE to select a fixed number k of features. Choose an estimator that provides coefficients or feature_importances_.

from sklearn.feature_selection import RFE

rfe = RFE(estimator=LogisticRegression(max_iter=1000, solver='liblinear'), n_features_to_select=5)
rfe.fit(df_X, df_y)
selected = df_X.columns[rfe.support_].tolist()
print('Selected features:', selected)

Step 5: Evaluate model with selected features

Compare the performance of the same estimator using only the selected features. This confirms whether selection helped or not.

X_sel = df_X[selected]
scores_sel = cross_val_score(clf, X_sel, df_y, cv=5, scoring='accuracy')
print('Acc with RFE:', scores_sel.mean(), '+/-', scores_sel.std())

Step 6: Use RFECV to choose k automatically

RFECV performs RFE with cross-validation to choose the optimal number of features. This avoids picking k arbitrarily.

from sklearn.feature_selection import RFECV
from sklearn.model_selection import StratifiedKFold

cv = StratifiedKFold(5)
rfecv = RFECV(estimator=LogisticRegression(max_iter=1000, solver='liblinear'), cv=cv, scoring='accuracy')
rfecv.fit(df_X, df_y)
print('Opt. no. features:', rfecv.n_features_)
opt_sel = df_X.columns[rfecv.support_].tolist()
print('Optimal features:', opt_sel)

Step 7: Best practices and variations

Best practices: scale features if the estimator is sensitive (e.g., LogisticRegression), try different estimators (RandomForest for non-linear relationships), and include selection inside a pipeline to avoid data leakage. Example with Pipeline:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([('scaler', StandardScaler()),
                 ('rfe', RFE(LogisticRegression(max_iter=1000, solver='liblinear'), n_features_to_select=5)),
                 ('clf', LogisticRegression(max_iter=1000, solver='liblinear'))])
scores_pipe = cross_val_score(pipe, df_X, df_y, cv=5)
print('Acc pipeline with RFE:', scores_pipe.mean())

Check the result

Confirm that the metrics (accuracy, precision/recall, AUC) with selected features are similar to or better than the baseline. Also check stability: run RFECV multiple times with different seeds and compare the selected features. If the list varies a lot, consider increasing data or using alternative methods (L1, tree-based).

Conclusion

RFE and RFECV are practical ways to reduce dimensionality while preserving model performance. Next steps: try RandomForest as the estimator, evaluate other metrics (AUC) and combine with techniques like PCA. Tip: when selected features are not stable across runs, ask whether you really need selection or more data.