How to Tune Hyperparameters with GridSearchCV in Python

A hyperparameter is a setting we choose before training a model — for example, how many trees a Random Forest has. Picking the best values by hand is slow and unreliable: you test one, change another, and quickly lose track of what you already tried. scikit-learn's GridSearchCV automates this: it tests several combinations with cross-validation and tells you which one performs best, making your model more accurate with little effort. Below is a practical example of hyperparameter tuning in Python.

Prerequisites

• Python 3 with scikit-learn and pandas installed (pip install scikit-learn pandas).
• Knowing how to train a simple model and split data into train and test sets.
• A notebook or code editor (Jupyter, VS Code or equivalent).

Step 1: Prepare the data

To avoid depending on external files, we use the Wine dataset bundled with scikit-learn — 178 wine samples with 13 features and three classes. We load the data and set aside 20% for testing; that set will only be used at the end, for an honest evaluation of the model.

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

Step 2: Define the model and the grid

The grid is a dictionary: each key is the name of a hyperparameter and each value is the list of options to try. GridSearchCV will try every possible combination. Here we use a Random Forest and vary three important settings: the number of trees (n_estimators), the maximum depth of each tree (max_depth) and the minimum number of samples required to split a node (min_samples_split).

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(random_state=42)

param_grid = {
    "n_estimators": [100, 200, 300],
    "max_depth": [None, 5, 10],
    "min_samples_split": [2, 5],
}

Tip: start with a small grid. These three lists produce 18 combinations (3 × 3 × 2) and, with 5-fold cross-validation, add up to 90 model fits. The larger the grid, the longer it takes.

Step 3: Run the search with cross-validation

We pass the model and the grid to GridSearchCV. The cv=5 parameter splits the training data into five parts and tests each combination on all of them, giving a far more reliable estimate than a single split. With scoring="accuracy" we state that the metric to maximise is accuracy, and n_jobs=-1 uses every CPU core to speed up the search.

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(
    estimator=model,
    param_grid=param_grid,
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
)

grid.fit(X_train, y_train)

Step 4: See the best combination

Once training is done, GridSearchCV stores the best combination in best_params_ and the mean cross-validation score in best_score_. The best model, already refitted on all the training data, is available in best_estimator_ and can be used directly to predict.

print("Best hyperparameters:", grid.best_params_)
print("Best CV score:", round(grid.best_score_, 3))

Step 5: Evaluate on the test set

Finally, we measure the best model's performance on the test data we set aside. It is this value — not the cross-validation one — that shows how the model behaves on data it has never seen.

accuracy = grid.score(X_test, y_test)
print("Test accuracy:", round(accuracy, 3))

Verify the result

If everything went well, you will see the best hyperparameter combination printed along with two scores — cross-validation and test — both close to each other and high, typically above 0.9 on this dataset. If the test accuracy is much lower than the cross-validation one, it may signal overfitting: try lowering max_depth or using more data. If both are low, widen the grid with other values.

Conclusion

With just a few lines, you have stopped guessing hyperparameters and started choosing them based on data. From here, use RandomizedSearchCV when the grid is too large to test in full, or combine GridSearchCV with a Pipeline to tune preprocessing too without risking data leakage. Which hyperparameter do you think will make the biggest difference in your next model?