ROC and AUC Explained

This StatQuest video by Josh Starmer provides a clear explanation of ROC (Receiver Operating Characteristic) curves and AUC (Area Under the Curve), which are tools used to evaluate the performance of classification models (like Logistic Regression).

See: https://www.youtube.com/watch?v=4jRBRDbJemM

Explanation in Words

1. The Problem: Choosing a Threshold

When a machine learning model makes a prediction (e.g., "Is this mouse obese?"), it usually outputs a probability (e.g., "There is a 0.8 chance this mouse is obese").

• To make a final decision, you must choose a threshold.

• Standard Threshold (0.5): If probability > 0.5, classify as Obese.

• Low Threshold (e.g., 0.1): You classify almost everyone as Obese. You catch all the actual cases (High Sensitivity), but you also falsely accuse many healthy mice (High False Positives). This is useful for dangerous diseases like Ebola where you can't afford to miss a case.

• High Threshold (e.g., 0.9): You are very strict. You rarely make a false accusation (Low False Positives), but you might miss some actual cases.

2. The ROC Graph

Instead of staring at dozens of "Confusion Matrices" for every possible threshold, we plot them all on a single graph called the ROC Curve.

• Y-Axis (True Positive Rate / Sensitivity): What percentage of the actual positive cases did we catch? (We want this to be 1.0).

• X-Axis (False Positive Rate / 1 - Specificity): What percentage of the actual negative cases did we incorrectly flag as positive? (We want this to be 0.0).

3. The AUC (Area Under the Curve)

• The ROC curve shows the trade-off. To judge if a model is "good" overall, we measure the Area Under the Curve (AUC).

• AUC = 1.0: A perfect model. It covers the entire square.

• AUC = 0.5: A random guess (flipping a coin). It covers half the square.

• Comparison: If Model A has an AUC of 0.9 and Model B has an AUC of 0.7, Model A is generally better.

---

The ROC Graph

The graph below represents the trade-off described in the video. The goal is to be as close to the Top Left Corner (Perfect) as possible.

Plaintext

       (True Positive Rate / Sensitivity)
       ^
  1.0  |        X  <-- BEST THRESHOLD (Perfect Classification)
       |      _/|
       |    _/  |
       |  _/    |      The "Curve" (Model Performance)
       |_/      |
       |        |
  0.5  |  (Bad) /      (Random Guess Line)
       |       /|
       |      / |
       |     /  |
       |    /   |
  0.0  |___/____|____________________>
      0.0      0.5      1.0            (False Positive Rate)
                                       (1 - Specificity)

Key Points on the Graph:

• X (Top Left): This point represents a threshold where you have 100% True Positives and 0% False Positives. It is the ideal scenario.

• The Curve: Represents the model's performance across all possible thresholds (from 0 to 1).

• The Diagonal Line: Represents a model that guesses randomly (e.g., flipping a coin). If your curve is on this line, the model is useless.

• AUC: The shaded area under that curve. The more area it covers (closer to 1.0), the better the model is at separating the two categories.

Video Reference:

• [07:05] - Introduction to the ROC graph axes.

• [10:07] - Explanation of the diagonal line (random chance).

• [13:36] - Explanation of AUC (Area Under the Curve).

Receiver-operating characteristic curve (ROC)

Also see: https://www.youtube.com/watch?v=4jRBRDbJemM

Another Blog: https://milindai.blogspot.com/2025/12/roc-and-auc-explained.html

The ROC curve is a visual representation of model performance across all thresholds. The long version of the name, receiver operating characteristic, is a holdover from WWII radar detection.

The ROC curve is drawn by calculating the true positive rate (TPR) and false positive rate (FPR) at every possible threshold (in practice, at selected intervals), then graphing TPR over FPR. A perfect model, which at some threshold has a TPR of 1.0 and a FPR of 0.0, can be represented by either a point at (0, 1) if all other thresholds are ignored, or by the following:

Figure 1. ROC and AUC of a hypothetical perfect model.

Area under the curve (AUC)

The area under the ROC curve (AUC) represents the probability that the model, if given a randomly chosen positive and negative example, will rank the positive higher than the negative.

The perfect model above, containing a square with sides of length 1, has an area under the curve (AUC) of 1.0. This means there is a 100% probability that the model will correctly rank a randomly chosen positive example higher than a randomly chosen negative example. In other words, looking at the spread of data points below, AUC gives the probability that the model will place a randomly chosen square to the right of a randomly chosen circle, independent of where the threshold is set.

Figure 2. A spread of predictions for a binary classification model. AUC is the chance a randomly chosen square is positioned to the right of a randomly chosen circle.

In more concrete terms, a spam classifier with AUC of 1.0 always assigns a random spam email a higher probability of being spam than a random legitimate email. The actual classification of each email depends on the threshold that you choose.

For a binary classifier, a model that does exactly as well as random guesses or coin flips has a ROC that is a diagonal line from (0,0) to (1,1). The AUC is 0.5, representing a 50% probability of correctly ranking a random positive and negative example.

In the spam classifier example, a spam classifier with AUC of 0.5 assigns a random spam email a higher probability of being spam than a random legitimate email only half the time.

Figure 3. ROC and AUC of completely random guesses.

(Optional, advanced) Precision-recall curve

AUC and ROC for choosing model and threshold

AUC is a useful measure for comparing the performance of two different models, as long as the dataset is roughly balanced. The model with greater area under the curve is generally the better one.

 

Figure 4. ROC and AUC of two hypothetical models. The curve on the right, with a greater AUC, represents the better of the two models.

The points on a ROC curve closest to (0,1) represent a range of the best-performing thresholds for the given model. As discussed in the Thresholds, Confusion matrix and Choice of metric and tradeoffs sections, the threshold you choose depends on which metric is most important to the specific use case. Consider the points A, B, and C in the following diagram, each representing a threshold:

Figure 5. Three labeled points representing thresholds.

If false positives (false alarms) are highly costly, it may make sense to choose a threshold that gives a lower FPR, like the one at point A, even if TPR is reduced. Conversely, if false positives are cheap and false negatives (missed true positives) highly costly, the threshold for point C, which maximizes TPR, may be preferable. If the costs are roughly equivalent, point B may offer the best balance between TPR and FPR.

Here is the ROC curve for the data we have seen before:

See:

https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc

Receiver-operating characteristic curve (ROC)

Area under the ROC curve (AUC) 

Exercise: Check your understanding

In practice, ROC curves are much less regular than the illustrations given above. Which of the following models, represented by their ROC curve and AUC, has the best performance?

Which of the following models performs worse than chance?

(Optional, advanced) Bonus question

Imagine a situation where it's better to allow some spam to reach the inbox than to send a business-critical email to the spam folder. You've trained a spam classifier for this situation where the positive class is spam and the negative class is not-spam. Which of the following points on the ROC curve for your classifier is preferable?

Point A

Point B

Point C

See: https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc#exercise_check_your_understanding