Halving every predicted probability of a crime leaves the AUC unchanged, while the probabilities then add up to about half the crimes that occur.
Published
September 25, 2026
Updated
November 15, 2026
Say a model gives each company announcement a probability of a crime, here, illegal insider trading, trading on information that is not yet public and material, so it is price sensitive. The area under the receiver operating characteristic curve (AUC) only concerns the ranking. Take one announcement with a crime and one without. If the model gives the one with the crime a higher probability, the pair counts as correct, even if the two probabilities are extremely close, or if I halve both. The ranking is still correct, but the predicted probabilities are not, and that is what calibration measures.
To show the difference, I take the logit model from Logit and probit: what the coefficients mean and test it on 10,000 simulated announcements. Then I halve every predicted probability. The AUC stays at 0.8975, while the expected number of crimes (the sum of the predicted probabilities) falls from 3,724 to 1,862, compared with 3,703 observed crimes. So ranking is one check and calibration another.
Simulating the data
With real data, I never know the truth, because some people commit a crime and do not get caught. So I simulate the data to know the truth: how many commit a crime and how many do not. I simulate the same 30,000 announcements as in the logit post, each with one variable, information_value, which measures how much the price moves on the announcement date, in standard deviations from its average. A crime happens at random, with a probability that rises with information_value through two coefficients that I set. I fit the logit model to the first 20,000 announcements and test it on the last 10,000.
NoteHow the announcements are simulated
The true score of announcement \(i\) uses the two coefficients I set,
\[\eta_i = -1.0 + 2.5\,x_i,\]
where \(x_i\) is information_value, a standard normal draw. The logit model turns each score into a probability of a crime, and then I draw whether a crime happens.
Of the 30,000 announcements, 11,112 are crimes (37.0%), with 7,409 in the first 20,000 and 3,703 in the last 10,000.
The block below fits the logit model with the statsmodels library, which gives each announcement a predicted probability of a crime.
NoteCode: the logit model
import statsmodels.api as smdesign = sm.add_constant(variables)logit = sm.Logit(is_crime.loc[fitting_sample], design.loc[fitting_sample]).fit(disp=0)print(logit.params.round(4).to_string())
const -1.0001
information_value 2.4498
The estimates, −1.0001 and 2.4498, are the same as in the logit post.
Measuring the ranking with the ROC curve
To act on the predicted probabilities, I pick a threshold and flag every announcement with a predicted probability above it. At a threshold of 50%, the model flags 3,452 of the 10,000 test announcements. They include 2,693 of the 3,703 crimes, so the sensitivity (the share of crimes that are flagged) is 72.7%. They also include 759 of the 6,297 announcements without a crime, so the false positive rate, the share of announcements without a crime that are flagged, is 12.1%. And 78.0% of the flagged announcements are crimes, the precision.
A lower threshold flags more announcements, so both the sensitivity and the false positive rate rise. The receiver operating characteristic (ROC) curve shows the two for every threshold from 100% down to 0%, and the AUC is the area under it. The animation below shows how the curve and its area come about.
threshold 50%: 3,452 flagged, 2,693 of 3,703 crimes, 759 of 6,297 without a crime
sensitivity 72.7%, false positive rate 12.1%, precision 78.0%
AUC 0.8975, with halved probabilities 0.8975
The AUC is 0.8975. It is also the share of pairs, one crime and one announcement without a crime, in which the crime has the higher predicted probability, so the model ranks 89.75% of such pairs correctly. Halving every probability keeps the order of the announcements, so the AUC stays at 0.8975.
Checking calibration band by band
Calibration asks whether the predicted probabilities match how often crimes happen. If they do, they add up to about the number of crimes. Some probabilities may be too low and others too high. They cancel out in the total, so I also sort the 10,000 test announcements by predicted probability into ten bands of 1,000 and compare each band’s average predicted probability with its share of crimes.
The original probabilities add up to 3,724 expected crimes, close to the 3,703 observed, while the halved probabilities add up to 1,862, about half. In band 5, the average predicted probability is 21.8%, and 21.1% of the announcements are crimes. Halving keeps the order, so band 5 holds the same 1,000 announcements, but its average predicted probability falls to 10.9%.
The animation below shows all ten bands.
Limitations
With real data, I can never know the truth, because I do not have the true label for every announcement. Some crimes are never caught, so it is much more difficult to judge both how a model ranks and how well its probabilities are calibrated, here for illegal insider trading.
The simulated announcements are not time-dependent, but real financial data is.
Conclusion
So the AUC only tells me whether the model ranks the announcements in the right order. If the predicted probabilities matter, I also need to check their calibration, band by band. And once I pick a threshold, the sensitivity and the precision tell me how many crimes the flags catch and how many of the flags are crimes.