How do I tune hyperparameters with GridSearchCV in scikit-learn?

The short version

GridSearchCV tries every combination of the parameter values you give it, evaluates each with cross-validation, and keeps the best. Give it a pipeline and address parameters with the step__parameter syntax. Keep a separate test set that the search never sees, so your final reported score is honest.

A worked example

from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

pipe = Pipeline([("scaler", StandardScaler()), ("clf", LogisticRegression())])
grid = {"clf__C": [0.1, 1, 10]}

search = GridSearchCV(pipe, grid, cv=5)
search.fit(X_train, y_train)

print(search.best_params_)
print(search.score(X_test, y_test))

The grid tries three values of C, each scored by five-fold cross-validation, and best_params_ reports the winner. The final score on the untouched test set is your honest performance estimate.

The step__parameter syntax

Because the model is inside a pipeline, you address its parameters by step name, two underscores, then the parameter. So clf__C means the C of the step named clf. This lets you tune the model and the preprocessing at once, for example searching scaler__with_mean alongside clf__C.

Why the held-out test set matters

The search itself uses cross-validation on the training data to choose parameters. But best_score_ is optimistic, because you picked the best of many attempts, and the winner benefits partly from chance. Scoring on a test set the search never touched corrects for that, giving a number you can trust. Never report best_score_ as your final result.

The combinatorial cost

Grid search tries every combination, so the cost multiplies. Three values of one parameter and four of another is twelve fits, each repeated across the folds. Adding parameters explodes this quickly, so keep the grid focused on the values that matter. When the space is large, RandomizedSearchCV samples a fixed number of combinations instead, which explores a big space at a controlled cost.

Reading the full results

Every combination's scores are stored in cv_results_, which you can load into a DataFrame to compare candidates rather than only seeing the winner.

import pandas as pd
pd.DataFrame(search.cv_results_)[["params", "mean_test_score", "std_test_score"]]

This is worth checking, because a slightly worse but much simpler or more stable setting is sometimes the better choice than the raw top scorer.

Choosing the scoring metric

As with cross-validation, set scoring to match your problem rather than accepting the default. Tuning to accuracy on imbalanced classes optimises the wrong thing, so pick a suitable metric (F1, ROC AUC) before running the search.