What AUC misses about predicted probabilities

Machine Learning
Tutorial
Python
Halving every predicted probability of a crime leaves the AUC unchanged, while the probabilities then add up to about half the crimes that occur.
Published

September 25, 2026

Updated

November 15, 2026

Say a model gives each company announcement a probability of a crime, here, illegal insider trading, trading on information that is not yet public and material, so it is price sensitive. The area under the receiver operating characteristic curve (AUC) only concerns the ranking. Take one announcement with a crime and one without. If the model gives the one with the crime a higher probability, the pair counts as correct, even if the two probabilities are extremely close, or if I halve both. The ranking is still correct, but the predicted probabilities are not, and that is what calibration measures.

To show the difference, I take the logit model from Logit and probit: what the coefficients mean and test it on 10,000 simulated announcements. Then I halve every predicted probability. The AUC stays at 0.8975, while the expected number of crimes (the sum of the predicted probabilities) falls from 3,724 to 1,862, compared with 3,703 observed crimes. So ranking is one check and calibration another.

Simulating the data

With real data, I never know the truth, because some people commit a crime and do not get caught. So I simulate the data to know the truth: how many commit a crime and how many do not. I simulate the same 30,000 announcements as in the logit post, each with one variable, information_value, which measures how much the price moves on the announcement date, in standard deviations from its average. A crime happens at random, with a probability that rises with information_value through two coefficients that I set. I fit the logit model to the first 20,000 announcements and test it on the last 10,000.

The true score of announcement \(i\) uses the two coefficients I set,

\[\eta_i = -1.0 + 2.5\,x_i,\]

where \(x_i\) is information_value, a standard normal draw. The logit model turns each score into a probability of a crime, and then I draw whether a crime happens.

import numpy as np
import pandas as pd

N_ANNOUNCEMENTS = 30000
N_FITTING = 20000
TRUE_INTERCEPT = -1.0
TRUE_COEFFICIENT = 2.5

rng = np.random.default_rng(11)
information_value = rng.normal(0, 1, N_ANNOUNCEMENTS)
true_score = TRUE_INTERCEPT + TRUE_COEFFICIENT * information_value
true_probability = 1.0 / (1.0 + np.exp(-true_score))
crime_draw = rng.uniform(0, 1, N_ANNOUNCEMENTS)
is_crime = pd.Series((crime_draw < true_probability).astype(int))
variables = pd.DataFrame({"information_value": information_value})

fitting_sample = np.arange(N_ANNOUNCEMENTS) < N_FITTING
test_sample = ~fitting_sample

print(f"announcements {N_ANNOUNCEMENTS:,}, crimes {is_crime.sum():,} ({is_crime.mean():.2%})")
print(f"  fitting sample {fitting_sample.sum():,}, crimes {is_crime.loc[fitting_sample].sum():,} "
      f"({is_crime.loc[fitting_sample].mean():.2%})")
print(f"  test sample    {test_sample.sum():,}, crimes {is_crime.loc[test_sample].sum():,} "
      f"({is_crime.loc[test_sample].mean():.2%})")
announcements 30,000, crimes 11,112 (37.04%)
  fitting sample 20,000, crimes 7,409 (37.05%)
  test sample    10,000, crimes 3,703 (37.03%)

Of the 30,000 announcements, 11,112 are crimes (37.0%), with 7,409 in the first 20,000 and 3,703 in the last 10,000.

The block below fits the logit model with the statsmodels library, which gives each announcement a predicted probability of a crime.

import statsmodels.api as sm

design = sm.add_constant(variables)
logit = sm.Logit(is_crime.loc[fitting_sample], design.loc[fitting_sample]).fit(disp=0)
print(logit.params.round(4).to_string())
const               -1.0001
information_value    2.4498

The estimates, −1.0001 and 2.4498, are the same as in the logit post.

Measuring the ranking with the ROC curve

To act on the predicted probabilities, I pick a threshold and flag every announcement with a predicted probability above it. At a threshold of 50%, the model flags 3,452 of the 10,000 test announcements. They include 2,693 of the 3,703 crimes, so the sensitivity (the share of crimes that are flagged) is 72.7%. They also include 759 of the 6,297 announcements without a crime, so the false positive rate, the share of announcements without a crime that are flagged, is 12.1%. And 78.0% of the flagged announcements are crimes, the precision.

A lower threshold flags more announcements, so both the sensitivity and the false positive rate rise. The receiver operating characteristic (ROC) curve shows the two for every threshold from 100% down to 0%, and the AUC is the area under it. The animation below shows how the curve and its area come about.

The predicted probabilities of crimes and other announcements, a falling threshold, the sensitivity and the false positive rate it gives, and the ROC curve it traces, before and after halving every probability sensitivity = flagged crimes all crimes = 3,703 = false positive rate = flagged without a crime all without a crime = 6,297 = random ranking AUC 0.8975 crimes without a crime Predicted probability of a crime False positive rate Sensitivity

from sklearn.metrics import roc_auc_score

test_outcome = is_crime.loc[test_sample]
test_probability = logit.predict(design.loc[test_sample])
halved_probability = test_probability / 2

crimes = test_outcome == 1
flagged = test_probability > 0.5
caught = (flagged & crimes).sum()
false_alarms = (flagged & ~crimes).sum()
print(f"threshold 50%: {flagged.sum():,} flagged, {caught:,} of {crimes.sum():,} crimes, "
      f"{false_alarms:,} of {(~crimes).sum():,} without a crime")
print(f"sensitivity {caught / crimes.sum():.1%}, "
      f"false positive rate {false_alarms / (~crimes).sum():.1%}, "
      f"precision {caught / flagged.sum():.1%}")
print(f"AUC {roc_auc_score(test_outcome, test_probability):.4f}, "
      f"with halved probabilities {roc_auc_score(test_outcome, halved_probability):.4f}")
threshold 50%: 3,452 flagged, 2,693 of 3,703 crimes, 759 of 6,297 without a crime
sensitivity 72.7%, false positive rate 12.1%, precision 78.0%
AUC 0.8975, with halved probabilities 0.8975

The AUC is 0.8975. It is also the share of pairs, one crime and one announcement without a crime, in which the crime has the higher predicted probability, so the model ranks 89.75% of such pairs correctly. Halving every probability keeps the order of the announcements, so the AUC stays at 0.8975.

Checking calibration band by band

Calibration asks whether the predicted probabilities match how often crimes happen. If they do, they add up to about the number of crimes. Some probabilities may be too low and others too high. They cancel out in the total, so I also sort the 10,000 test announcements by predicted probability into ten bands of 1,000 and compare each band’s average predicted probability with its share of crimes.

print(f"expected crimes {test_probability.sum():,.0f}, with halved probabilities "
      f"{halved_probability.sum():,.0f}, observed crimes {test_outcome.sum():,}")

bands = pd.DataFrame({"band": pd.qcut(test_probability, 10, labels=range(1, 11)),
                      "observed": test_outcome,
                      "predicted": test_probability,
                      "predicted, halved": halved_probability})
calibration = 100 * bands.groupby("band", observed=True).mean()
print(calibration.round(1).to_string())
expected crimes 3,724, with halved probabilities 1,862, observed crimes 3,703
      observed  predicted  predicted, halved
band                                        
1          0.8        0.7                0.3
2          2.2        3.0                1.5
3          6.3        7.1                3.5
4         12.6       13.0                6.5
5         21.1       21.8               10.9
6         35.0       33.9               17.0
7         47.0       49.3               24.6
8         66.6       66.6               33.3
9         83.5       82.2               41.1
10        95.2       94.9               47.4

The original probabilities add up to 3,724 expected crimes, close to the 3,703 observed, while the halved probabilities add up to 1,862, about half. In band 5, the average predicted probability is 21.8%, and 21.1% of the announcements are crimes. Halving keeps the order, so band 5 holds the same 1,000 announcements, but its average predicted probability falls to 10.9%.

The animation below shows all ten bands.

Ten bands of announcements: average predicted probability against share of crimes, before and after halving every probability expected crimes 3,724 → 1,862observed 3,703 original probabilities halved probabilities Average predicted probability in the band Share of crimes in the band

Limitations

With real data, I can never know the truth, because I do not have the true label for every announcement. Some crimes are never caught, so it is much more difficult to judge both how a model ranks and how well its probabilities are calibrated, here for illegal insider trading.

The simulated announcements are not time-dependent, but real financial data is.

Conclusion

So the AUC only tells me whether the model ranks the announcements in the right order. If the predicted probabilities matter, I also need to check their calibration, band by band. And once I pick a threshold, the sensitivity and the precision tell me how many crimes the flags catch and how many of the flags are crimes.