Classification Metrics Explained: Precision, Recall, F1, ROC-AUC and More

"The model is 95% accurate" sounds great until you learn that 95% of the data belongs to one class and the model predicts that class every time. Choosing the right evaluation metric is as important as choosing the right model. It defines what "good" means, and a model optimized for the wrong metric will confidently do the wrong thing.

#Key takeaways

• Start with the confusion matrix; every classification metric derives from it.
• Accuracy misleads on imbalanced data.
• Precision: of the items you flagged, how many were right? Recall: of the real positives, how many did you catch?
• ROC-AUC measures ranking quality across thresholds; PR-AUC is more informative when positives are rare.
• The decision threshold is a business choice. Pick it using the costs of errors.

#The confusion matrix

For a binary classifier (say, fraud detection):

• False positive: a legitimate transaction blocked, so an annoyed customer.
• False negative: fraud gets through, so money is lost.

Every metric below is a different way to summarize these four numbers.

#Accuracy

textAccuracy = (TP + TN) / (TP + TN + FP + FN)

Fine when classes are balanced and errors cost the same. Misleading otherwise: with 1% fraud, predicting "legit" for everything gives 99% accuracy and catches zero fraud.

#Precision and recall

textPrecision = TP / (TP + FP)     "When the model says fraud, how often is it right?"
Recall    = TP / (TP + FN)     "Of all actual fraud, how much did we catch?"

They pull in opposite directions. Flag more transactions and recall rises while precision usually falls.

#F1 score

The harmonic mean of precision and recall:

textF1 = 2 × (Precision × Recall) / (Precision + Recall)

Useful as a single number when you care about both roughly equally. The F-beta score weights recall β times as much as precision (F2 favors recall; F0.5 favors precision). But F1 ignores true negatives and hides the trade-off, so report precision and recall alongside it.

#Thresholds: the hidden decision

Most classifiers output a score or probability, and a threshold (often 0.5 by default) turns it into a decision. The default is rarely optimal.

Choose the threshold using business costs:

pythonimport numpy as np

def expected_cost(y_true, scores, threshold, cost_fp=5, cost_fn=200):
    pred = scores >= threshold
    fp = np.sum(pred & (y_true == 0))
    fn = np.sum(~pred & (y_true == 1))
    return fp * cost_fp + fn * cost_fn

thresholds = np.linspace(0.01, 0.99, 99)
best = min(thresholds, key=lambda t: expected_cost(y_val, val_scores, t))

Other practical approaches: pick the threshold that achieves a required recall (catch 90% of fraud), or a capacity limit (the team can review 200 cases per day).

#ROC curve and ROC-AUC

The ROC curve plots the true positive rate (recall) against the false positive rate (FP / (FP + TN)) across all thresholds. ROC-AUC is the area under it:

• 1.0: perfect ranking.
• 0.5: random guessing.
• Interpretation: the probability that a random positive is scored higher than a random negative.

ROC-AUC is threshold-independent and good for comparing models' ranking ability. But with heavy class imbalance it can look optimistic, because the false positive rate stays small even when the absolute number of false positives is large.

#Precision-recall curve and PR-AUC

The PR curve plots precision against recall across thresholds. PR-AUC (or average precision) focuses on the positive class and is more informative for rare events such as fraud, defects or churn in low-churn businesses. Its baseline equals the positive rate: with 1% positives, a random model scores about 0.01.

#Ranking metrics: precision@k and lift

When you'll act on the top k items (the top 100 leads, the top 50 at-risk accounts), measure exactly that:

• Precision@k: fraction of true positives in the top k.
• Lift: how much better the top segment performs than random selection.

These map directly to business workflows and are easy to explain to stakeholders.

#Calibration

If a model says "70% probability of churn", do about 70% of those customers actually churn? That's calibration. It matters when you use probabilities directly: expected value calculations, pricing, risk scoring.

• Check with calibration curves (reliability diagrams) and the Brier score.
• Fix with Platt scaling or isotonic regression on validation data.

Tree ensembles and some other models can produce poorly calibrated scores even when their ranking (AUC) is excellent.

#Multi-class metrics

For more than two classes:

• Per-class precision and recall, which show where the model struggles.
• Macro average: an unweighted mean across classes, so every class counts equally (good when minority classes matter).
• Weighted average: weighted by class frequency.
• Confusion matrix: shows which classes get confused with which.

#Regression metrics, briefly

• MAE (mean absolute error): average error in original units. Robust and interpretable.
• RMSE: penalizes large errors more heavily.
• MAPE: percentage error. It breaks down when actuals are near zero.
• R²: proportion of variance explained.

For forecasting specifics, see time series forecasting.

#Choosing metrics: a checklist

1. What decision will the model drive, and what does each error type cost?
2. Is the data imbalanced? If so, avoid accuracy and prefer PR-AUC, precision and recall.
3. Will you act on top-k items? Use precision@k or lift.
4. Do you need probabilities? Check calibration.
5. Evaluate on a proper held-out set, with cross-validation for stability (overfitting and bias-variance).
6. Break metrics down by important segments (region, customer type) to catch unfair or uneven performance.
7. Confirm offline metrics translate to online impact with an experiment (A/B testing done right).

#FAQ

#What is a good ROC-AUC score?

It depends on the problem and baseline. In many business applications, 0.7–0.8 is useful and above 0.8 is strong, but domain matters. Always compare against a simple baseline, and check PR-AUC when positives are rare.

#Should I optimize precision or recall?

Decide based on the costs of false positives versus false negatives. When missing a positive is expensive (safety, fraud losses), favor recall; when false alarms are expensive (customer friction, wasted review time), favor precision.

#Why is my model's accuracy high but it's useless?

Probably class imbalance: the model predicts the majority class and gets high accuracy without detecting the minority class you care about. Look at the confusion matrix, recall and PR-AUC.

#How do these metrics apply to LLM classifiers?

The same way. If you use an LLM to classify tickets or detect intent, evaluate it on a labeled set with a confusion matrix, per-class precision and recall. See how to evaluate LLM applications.