Which metric should I use to evaluate a classifier in Python?

The short version

Accuracy is the wrong default for imbalanced data, because always predicting the majority class scores well. Use precision when false positives are expensive, recall when missing a positive is expensive, and F1 when you need a single balanced figure. Always look at the confusion matrix, which shows what the model actually does.

The four numbers that matter

from sklearn.metrics import (
    precision_score, recall_score, f1_score, confusion_matrix,
)

y_true = [0, 0, 1, 1]
y_pred = [0, 1, 1, 1]

confusion_matrix(y_true, y_pred)   # [[1, 1], [0, 2]]
precision_score(y_true, y_pred)    # 0.667
recall_score(y_true, y_pred)       # 1.0
f1_score(y_true, y_pred)           # 0.8

The confusion matrix reads as [[true negatives, false positives], [false negatives, true positives]]. Here there is one false positive and no false negatives.

What each metric means, in words

Precision is: of the cases the model flagged, how many were really positive? Low precision means many false alarms. This matters when acting on a positive is expensive, for example sending an engineer to inspect a machine.

Recall is: of the cases that really were positive, how many did the model find? Low recall means it misses things. This matters when a miss is expensive, for example failing to flag a fault before it breaks.

F1 is the harmonic mean of the two, giving one number when you need a single figure and both errors matter roughly equally.

The trade-off is real

Precision and recall pull against each other. Flag everything and recall reaches 1.0 while precision collapses. Flag only the certain cases and precision rises while recall falls. There is no setting that maximises both, so you must decide which error your problem can better afford. That decision belongs to the business context, not to the model.

Why accuracy misleads

With 95 per cent of cases negative, a model that always says "negative" is 95 per cent accurate and completely useless. Accuracy hides this, and the confusion matrix reveals it instantly: an entire row of zeros where the positives should have been detected.

ROC AUC and its caveat

from sklearn.metrics import roc_auc_score
roc_auc_score(y_true, y_scores)    # uses predicted probabilities, not labels

ROC AUC measures how well the model ranks positives above negatives, across every threshold. It is useful because it does not depend on where you set the cut-off. But on severely imbalanced data it can look flatteringly high, because the huge number of true negatives dominates. For rare positives, the precision-recall curve (and its average precision) is more informative.

Choose before you look

Decide which metric matters, and why, before you see the results. Choosing the metric afterwards, on the basis of which one makes your model look best, is a way of fooling yourself and everyone reading the report.