How do I evaluate a model with cross-validation in scikit-learn?

The short version

cross_val_score splits the data into several folds, trains on all but one and tests on the held-out fold, rotating through them, and returns a score per fold. Averaging these gives a more reliable estimate than a single train-test split. Pass a whole pipeline so preprocessing is refitted on each fold, and use stratified folds for classification so class balance is preserved.

The basic call

from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(StandardScaler(), LogisticRegression())
scores = cross_val_score(model, X, y, cv=5)

print(scores.mean(), scores.std())

Five folds give five scores. The mean is your headline estimate, and the standard deviation tells you how much it varies between folds, which is a rough guide to how stable the model is.

Why a single split is not enough

One train-test split gives one number, and that number depends on which rows happened to land in the test set. A lucky or unlucky split can mislead you. Cross-validation averages over several splits, so the estimate is less sensitive to any one partition, and you also get a sense of the spread.

Pass the pipeline, not preprocessed data

This is the crucial correctness point. Give cross_val_score the pipeline and the raw data, so it refits the preprocessing on each fold's training portion.

cross_val_score(model, X, y, cv=5)     # correct: no leakage

If you scaled X first and then cross-validated, every fold's scaler would have seen the whole dataset, leaking test information into training and inflating the scores. Passing the pipeline prevents this automatically.

Stratify for classification

For classification, especially with imbalanced classes, plain folds can end up with very different class proportions. StratifiedKFold keeps each fold's class balance close to the whole set's.

from sklearn.model_selection import StratifiedKFold
cross_val_score(model, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=0))

In fact cross_val_score already uses stratified folds by default for classifiers, but passing an explicit splitter lets you control shuffling and the random state for reproducibility.

Choosing the scoring metric

The default score is the estimator's own (accuracy for classifiers, R squared for regressors), which is often not what you want. Set scoring to the metric that matches your problem.

cross_val_score(model, X, y, cv=5, scoring="f1")

For imbalanced classes accuracy is misleading, so a metric like F1 or ROC AUC usually tells you more. Pick the metric before you look at the numbers, so you are not tempted to choose whichever looks best.

When you need predictions, not just scores

If you want the cross-validated predictions themselves (for a confusion matrix, say), use cross_val_predict instead, which returns a prediction for every row made by the model that did not train on it.