Estimating the frequency of market crimes

Python

Tutorial

Code

From a prosecution rate of 5%, Detection-Controlled Estimation estimates a crime rate of 23%, where the simulated true rate is 25%.

Published

May 27, 2026

To check a model, we usually compare its predictions with the truth: a predicted return with the actual return. Market crime is harder to check. We observe prosecutions, but not every crime, so we do not know how many crimes were missed or how accurate the surveillance system is.

A low number of prosecutions can mean two very different things: few crimes, or many crimes of which few were detected.

Detection-Controlled Estimation (DCE), introduced by Feinstein (1990), separates the number of crimes from the share detected. Patel and Putniņš (2020) use it to estimate how much illegal insider trading happens around announcements of mergers and acquisitions (M&A) and of earnings, and Comerton-Forde and Putniņš (2014) to estimate how often the closing price is manipulated. Both estimate many more market crimes than prosecutions.

I code DCE in Python to estimate the frequency of illegal insider trading. I test it on a synthetic market in which I decide how many crimes happen and how many are detected, so I can compare its estimates with the truth. Prosecutions put the crime rate at about 5% of the announcements, and DCE estimates 22.9%, against a true 25.3%.

Why prosecutions are not enough

The unit of observation is an announcement, a single dated, price-sensitive event such as a merger bid or an earnings release, where someone could trade on inside information. Every announcement has two unobserved steps. First, \(P(\text{crime})\), the probability that insider trading happened. Second, \(P(\text{detection} \mid \text{crime})\), the probability that a crime, if it happened, was detected and prosecuted; the vertical bar reads “given”.

We observe only the joint outcome:

\[P(\text{prosecution}) = P(\text{crime}) \times P(\text{detection} \mid \text{crime})\]

A 5% prosecution rate can come from many combinations of crime rate and detection rate:

Crime rate	Detection rate	Prosecution rate
5%	100%	5%
10%	50%	5%
25%	20%	5%
50%	10%	5%

The observed data are identical in all four rows. Prosecutions tell us only the product of the crime rate and the detection rate. So, prosecution counts alone cannot answer “how much market crime is there?”

Announcements with no crime and announcements with an undetected crime are both in the non-prosecuted group, so we cannot compute accuracy the usual way.

What DCE adds

DCE estimates two equations at the same time. One equation explains the probability that a crime happened. The other equation explains the probability that a crime was detected, given that it happened.

The model needs variables that separate the two probabilities. Some affect only the crime probability, such as the value of the information and how many people had access to it. Others affect only the detection probability, such as surveillance signals and regulatory resources. These are the exclusion restrictions: once the other features are known, a crime-side variable adds no information about the detection probability, and a detection-side variable none about the crime probability. A variable that affects both, such as liquidity, how easily a stock trades without its price moving much, can enter both equations, but it cannot help separate crime from detection.

Step 1. Simulating the data

I build 10,000 announcements for which I decide the crimes and detections. DCE receives only the features and the prosecution indicator, 1 for a prosecuted announcement and 0 otherwise, and I compare its estimates with the truth. Each announcement has five features, weighted by coefficients I choose: two affect only the probability of a crime, two only the probability that a crime is detected, and one both. So the exclusion restrictions are true by construction, and the simulation tests whether DCE’s calculations work, not whether the restrictions hold in real data. The box below gives the formulas and the code.

How the announcements are simulated

I build the features from two unobserved factors, so they are correlated as real features would be: bigger firms have bigger deals and more liquidity.

import numpy as np                      # arrays and random number generation
import pandas as pd                     # tabular containers for clean reporting
from scipy.optimize import minimize     # numerical optimiser for maximum likelihood

# The steps below draw from the generator numpy keeps by default, so fixing its seed here
# gives the same numbers on every render.
np.random.seed(42)

The block below builds five observable features: two on the crime side, two on the detection side and one on both. Each is a weighted sum of an unobserved factor \(F_i\) and the feature’s own noise \(u_i\), rescaled to a mean of 0 and a standard deviation of 1, a measure of how far values typically are from their mean,

\[x_i = \operatorname{standardise}(a\,F_i + b\,u_i),\]

where \(F_i\) is firm size for info_value, leakage and liquidity, with weights \((a, b)\) of (0.7, 0.7), (0.6, 0.8) and (0.5, 0.9), and the enforcement era for surveillance_signal and reg_resources, with (0.7, 0.7) and (0.8, 0.6). The two factors and every \(u_i\) are independent draws from the standard normal distribution, which is bell-shaped with a mean of 0 and a standard deviation of 1, one per announcement. So two features built on the same factor are correlated, info_value and leakage at \(0.7 \times 0.6/\sqrt{(0.7^2 + 0.7^2)(0.6^2 + 0.8^2)} \approx 0.42\), while a feature built on firm size and one built on the enforcement era are uncorrelated.

N = 10000  # number of announcements


def standardise(feature):
    """Rescale one feature to mean 0 and standard deviation 1.

    An array with mean 4 and standard deviation 2 comes back with mean 0 and standard
    deviation 1. Putting every feature on the same scale makes the coefficients chosen below
    comparable with each other. Without it, each coefficient would depend on the units
    of its raw feature.
    """
    centred = feature - feature.mean()
    return centred / feature.std()


# Two latent factors that nobody observes. Building every feature partly out of one of them
# gives the five features realistic correlations.
firm_size_factor     = np.random.normal(0, 1, N)  # bigger firms: bigger deals, more liquidity
detection_era_factor = np.random.normal(0, 1, N)  # periods of stronger enforcement

# Each feature is a share of its latent factor plus its own private noise, and the two
# weights set how much of each. Numbers leave the generator in the order these lines run,
# so reordering these lines changes which draws the features receive.

# Crime side, correlated with firm size
raw_info_value          = 0.7 * firm_size_factor     + 0.7 * np.random.normal(0, 1, N)
raw_leakage             = 0.6 * firm_size_factor     + 0.8 * np.random.normal(0, 1, N)
# Detection side, correlated with the enforcement era
raw_surveillance_signal = 0.7 * detection_era_factor + 0.7 * np.random.normal(0, 1, N)
raw_reg_resources       = 0.8 * detection_era_factor + 0.6 * np.random.normal(0, 1, N)
# Both sides, correlated with firm size
raw_liquidity           = 0.5 * firm_size_factor     + 0.9 * np.random.normal(0, 1, N)

info_value          = standardise(raw_info_value)
leakage             = standardise(raw_leakage)
surveillance_signal = standardise(raw_surveillance_signal)
reg_resources       = standardise(raw_reg_resources)
liquidity           = standardise(raw_liquidity)

Two equations turn the features into the unobserved probabilities. For announcement \(i\), the crime score and the detection score are weighted sums of the features,

\[s_{Ci} = -1.5 + 0.9\,\text{info}_i + 0.6\,\text{leak}_i + 0.4\,\text{liq}_i, \qquad s_{Di} = -1.8 + 1.2\,\text{surv}_i + 0.7\,\text{reg}_i - 0.3\,\text{liq}_i,\]

where info, leak, liq, surv and reg are info_value, leakage, liquidity, surveillance_signal and reg_resources. The logistic function, sigmoid() in the code, turns each score into a probability between 0 and 1:

\[\sigma(s) = \frac{1}{1 + e^{-s}}, \qquad p_i = \sigma(s_{Ci}), \qquad d_i = \sigma(s_{Di}),\]

where \(p_i\) is the probability of a crime and \(d_i\) the probability that a crime, if it happened, is detected and prosecuted. A score of 0 gives a probability of 0.5, and the crime score of an announcement with average features, −1.5, gives 0.18. Two independent random draws then turn the probabilities into outcomes: a crime happens, \(C_i = 1\), with probability \(p_i\), a crime would be detected, \(D_i = 1\), with probability \(d_i\), and the announcement is prosecuted when both happen, \(A_i = C_i D_i\).

# The true coefficients, which DCE is never given. Changing one of them changes
# the crime rate, the detection rate, or both.
crime_intercept                = -1.5   # the crime score at average features
crime_weight_info_value        =  0.9
crime_weight_leakage           =  0.6
crime_weight_liquidity         =  0.4

detection_intercept            = -1.8   # the detection score at average features
detection_weight_surveillance  =  1.2
detection_weight_reg_resources =  0.7
detection_weight_liquidity     = -0.3   # negative: harder to spot in liquid markets


def sigmoid(score):
    """Turn any number into a probability between 0 and 1.

    sigmoid(0) is 0.5, sigmoid(-4) is about 0.02, and sigmoid(4) is about 0.98. The sigmoid
    makes a weighted sum of features readable as a probability.
    """
    return 1.0 / (1.0 + np.exp(-score))


# A weighted sum of the features, one number per announcement, before the sigmoid runs.
crime_score = (crime_intercept
               + crime_weight_info_value * info_value
               + crime_weight_leakage    * leakage
               + crime_weight_liquidity  * liquidity)
detection_score = (detection_intercept
                   + detection_weight_surveillance  * surveillance_signal
                   + detection_weight_reg_resources * reg_resources
                   + detection_weight_liquidity     * liquidity)

true_crime_probability     = sigmoid(crime_score)
true_detection_probability = sigmoid(detection_score)

# A Bernoulli draw: take a random number between 0 and 1 and call the event a 1 whenever
# the number is below the probability, which happens exactly that often in the long run.
crime_random_number     = np.random.uniform(0, 1, N)
detection_random_number = np.random.uniform(0, 1, N)

# 1 = a crime happened here
C = (crime_random_number < true_crime_probability).astype(int)
# 1 = it would be detected, if a crime happened here
D_if_crime = (detection_random_number < true_detection_probability).astype(int)
# 1 = prosecution. Of the three columns, A is the only one DCE is allowed to read.
A = C * D_if_crime

print("TRUTH (not given to DCE):")
print(f"  mean P(crime)             = {true_crime_probability.mean():.2%}")
print(f"  mean P(detection | crime) = {true_detection_probability.mean():.2%}")
print()
print("OBSERVED (the only input to DCE, alongside the features):")
print(f"  P(prosecution)            = {A.mean():.2%}")
print(f"  P(no prosecution)         = {1 - A.mean():.2%}")

TRUTH (not given to DCE):
  mean P(crime)             = 25.30%
  mean P(detection | crime) = 22.93%

OBSERVED (the only input to DCE, alongside the features):
  P(prosecution)            = 4.97%
  P(no prosecution)         = 95.03%

The true crime rate is about 25% of announcements and the prosecution rate about 5%, so most crimes in this simulated market are never detected. The two rates come from assumed coefficients and are close to Patel and Putniņš’s estimates: insider trading in 1 in 5 M&A announcements, and at least four cases of insider trading for every prosecution. DCE receives only the prosecution indicator, A in the code, and not the crime label C or the detection label D_if_crime.

Step 2. Writing the likelihood

DCE writes the crime and detection probabilities with coefficients it has to estimate, \(p_i = \sigma(X_{1i}\beta_1)\) and \(d_i = \sigma(X_{2i}\beta_2)\). The logistic function \(\sigma(s) = 1/(1 + e^{-s})\) turns any score into a probability between 0 and 1. The crime-side features of announcement \(i\) are \(X_{1i}\), its detection-side features are \(X_{2i}\), and \(\beta_1\) and \(\beta_2\) are the coefficients.

Every announcement is in exactly one of two groups, prosecuted (\(A = 1\)) or not prosecuted (\(A = 0\)), so the probabilities of the two groups sum to one.

A prosecution needs both a crime and its detection, so its probability is the product:

\[P(A = 1) = p \cdot d.\]

Everything else is in the not-prosecuted group, either no crime, or a crime that went undetected, which has the leftover probability:

\[P(A = 0) = 1 - p \cdot d.\]

So each announcement contributes \(p \cdot d\) if it was prosecuted and \(1 - p \cdot d\) if it was not: with \(p = 25\%\) and \(d = 20\%\), 0.05 if prosecuted and 0.95 if not. The product of those contributions across all announcements is the likelihood, the probability of the observed prosecutions. Its logarithm, the log-likelihood, is

\[\log L = \sum_{i:\,A_i = 1} \log(p_i d_i) \;+\; \sum_{i:\,A_i = 0} \log(1 - p_i d_i).\]

Each feature goes into the equation it bears on. The five synthetic features split like this:

info_value, crime side. The more valuable the information, the larger the payoff from trading on it, and the more tempting the crime.
leakage, crime side. The more people who hold the information, the more potential sources of illegal trading.
surveillance_signal, detection side. Abnormal pre-announcement trading, which makes a crime, if it happened, easier to notice. The model assumes that, once the other features are known, it adds no information about the crime probability.
reg_resources, detection side. More enforcement resources make detection and prosecution easier.
liquidity, both. Higher liquidity raises the crime probability and lowers the detection probability, through coefficients I set, \(0.4\) and \(-0.3\). It enters both equations, so it cannot separate the two probabilities.

These assignments of features to equations are the model’s economic assumptions: the crime equation excludes surveillance_signal and reg_resources, and the detection equation excludes info_value and leakage.

Code: the likelihood

# One row per announcement, one column per coefficient. crime_features is X1 in the formula
# above and detection_features is X2. The leading column of ones is the intercept, so one
# matrix multiplication covers the intercept and the slopes together.
#
# The next two lines impose the exclusion restrictions: surveillance_signal and
# reg_resources are absent from the crime side, info_value and leakage are absent from the
# detection side.
crime_features     = np.column_stack([np.ones(N), info_value,          leakage,       liquidity])
detection_features = np.column_stack([np.ones(N), surveillance_signal, reg_resources, liquidity])

n_crime_coefficients     = crime_features.shape[1]     # shape is (rows, columns)
n_detection_coefficients = detection_features.shape[1]


def dce_negative_log_likelihood(params):
    """How badly one candidate set of coefficients explains the prosecutions we observe.

    minimize takes a single flat array, so params holds both coefficient vectors end
    to end: the first n_crime_coefficients entries belong to the crime equation and the
    rest to the detection equation. Lower is better, because this is the log-likelihood
    with its sign flipped, and minimize searches for the lowest value.
    """
    crime_coefficients     = params[:n_crime_coefficients]
    detection_coefficients = params[n_crime_coefficients:]

    # @ is matrix multiplication, so every row of features meets the coefficients once.
    crime_probability     = sigmoid(crime_features @ crime_coefficients)
    detection_probability = sigmoid(detection_features @ detection_coefficients)

    # A prosecution needs the crime and its detection, so its probability is the product.
    prosecution_probability = crime_probability * detection_probability

    # np.log(0) is minus infinity and would stall the optimiser, so hold the probability a
    # hair inside 0 and 1.
    guarded_probability = np.clip(prosecution_probability, 1e-12, 1 - 1e-12)

    # A is 1 or 0, so each announcement contributes exactly one of these two terms.
    prosecuted_terms     = A * np.log(guarded_probability)
    not_prosecuted_terms = (1 - A) * np.log(1 - guarded_probability)
    return -np.sum(prosecuted_terms + not_prosecuted_terms)

I fit both equations at once, because the data have no column for the crime or for its detection alone. Maximum likelihood finds the two sets of coefficients that make the observed prosecutions most likely. Many \((p, d)\) pairs multiply to the same prosecution rate, and the exclusion restrictions separate them: a feature that enters only one equation moves only that probability, so how the prosecution rate varies with that feature informs that probability alone.

Step 3. Fitting the model and comparing with the truth

Code: the fit and the comparison with the truth

# The optimiser has to start somewhere. Every slope starts at zero, and both intercepts
# start at -1.0, which is a low crime rate and a low detection rate. Starting the two
# intercepts below zero starts the search at a prosecution rate closer to the one in the data.
starting_values = np.zeros(n_crime_coefficients + n_detection_coefficients)
starting_values[0] = -1.0                     # position 0 is the crime intercept
starting_values[n_crime_coefficients] = -1.0  # the detection intercept opens the second half

# L-BFGS-B walks downhill on the negative log-likelihood until it stops improving.
fit = minimize(dce_negative_log_likelihood, starting_values, method='L-BFGS-B',
               options={'maxiter': 2000, 'ftol': 1e-10})

# fit.x is the flat array the optimiser settled on, split exactly as params was.
estimated_crime_coefficients     = fit.x[:n_crime_coefficients]
estimated_detection_coefficients = fit.x[n_crime_coefficients:]

estimated_crime_probability     = sigmoid(crime_features @ estimated_crime_coefficients)
estimated_detection_probability = sigmoid(detection_features @ estimated_detection_coefficients)

# A prosecution needs both, on the true side and on the estimated side alike.
true_prosecution_probability      = true_crime_probability * true_detection_probability
estimated_prosecution_probability = estimated_crime_probability * estimated_detection_probability

not_observed = "n/a"   # n/a marks a quantity real data never reveals

result = pd.DataFrame({
    "Quantity":         ["mean P(crime)", "mean P(detection | crime)", "mean P(prosecution)"],
    "TRUE (unobserved)": [f"{true_crime_probability.mean():.2%}",
                         f"{true_detection_probability.mean():.2%}",
                         f"{true_prosecution_probability.mean():.2%}"],
    "OBSERVED (data)":  [not_observed, not_observed, f"{A.mean():.2%}"],
    "DCE estimate":     [f"{estimated_crime_probability.mean():.2%}",
                         f"{estimated_detection_probability.mean():.2%}",
                         f"{estimated_prosecution_probability.mean():.2%}"],
})
print(result.to_string(index=False))

                 Quantity TRUE (unobserved) OBSERVED (data) DCE estimate
            mean P(crime)            25.30%             n/a       22.86%
mean P(detection | crime)            22.93%             n/a       24.04%
      mean P(prosecution)             5.35%           4.97%        5.00%

TRUE is the mean, across the 10,000 announcements, of the probabilities I generated and withheld. OBSERVED is what the data show: only the prosecution rate, so the other two cells are n/a. DCE matches the observed prosecution rate, 5.00% against 4.97%, and its crime and detection rates can be compared with the truth. The true mean prosecution probability, 5.35%, differs from the observed rate, 4.97%, only by chance, because each prosecution is a random draw.

Results

Counting prosecutions would put market crimes at about 5% of announcements, because it misses the crimes never detected. DCE estimates that crimes happened at 22.9% of the announcements, against a true 25.3%, 2.4 percentage points below, and the mean detection probability at 24.0%, against a true 22.9%. The chart below draws the three crime rates side by side.

Code: the chart

import matplotlib.pyplot as plt

RED, TEAL, GREY = "#C0392B", "#17868A", "#888888"
# The site's chart colours, in order: the cream background, the text, the light grid lines,
# the pale frame and the axis labels, the same on every chart of the site.
BG, INK, GRID, SPINE, AXIS_TEXT = "#FCEFE3", "#1f1f1f", "#EADCCC", "#D5C6B4", "#4a4a4a"


def style_chart(figure, axis):
    """Give a finished chart the site's cream background, light grid and pale frame."""
    figure.patch.set_facecolor(BG)
    axis.set_facecolor(BG)
    axis.grid(True, axis="y", color=GRID, lw=1.0)
    axis.set_axisbelow(True)
    for side in ("top", "right"):
        axis.spines[side].set_visible(False)
    for side in ("left", "bottom"):
        axis.spines[side].set_color(SPINE)
    axis.tick_params(axis="both", length=0, colors=INK, pad=6)
    axis.xaxis.label.set_color(AXIS_TEXT)
    axis.yaxis.label.set_color(AXIS_TEXT)
    axis.title.set_color(INK)
    # Write each legend entry in the colour of what it labels: a line's colour or a bar's fill
    # colour. get_legend() returns None when the chart has no legend.
    legend = axis.get_legend()
    if legend is not None:
        for text, handle in zip(legend.get_texts(), legend.legend_handles):
            if hasattr(handle, "get_color"):
                text.set_color(handle.get_color())
            else:
                text.set_color(handle.get_facecolor())


# The three crime rates, in percent of the 10,000 announcements: the share that was prosecuted,
# the average crime probability that DCE estimates, and the average true crime probability,
# which DCE never sees. \n in a label starts a second line.
bar_labels = ["counted from\nprosecutions", "DCE estimate", "true crime rate"]
bar_heights = [100 * A.mean(),
               100 * estimated_crime_probability.mean(),
               100 * true_crime_probability.mean()]

# plt.subplots creates the figure and its one chart, axis; figsize is the figure's width and
# height in inches.
figure, axis = plt.subplots(figsize=(7, 4.5))
bars = axis.bar(bar_labels, bar_heights, color=[RED, TEAL, GREY])
# bar_label writes each bar's height above it, 3 points up: %.1f writes one decimal and %% a
# percent sign
axis.bar_label(bars, fmt="%.1f%%", padding=3, color=INK)
axis.set_ylim(0, 30)
axis.set_ylabel("Share of announcements (%)")
axis.set_title("The crime rate from prosecutions, from DCE and in truth")
style_chart(figure, axis)
# tight_layout() fits the labels inside the figure. savefig writes the chart to a PNG file:
# dpi=140 sets its resolution, bbox_inches="tight" trims the empty margin, and facecolor=BG
# keeps the cream background. show() displays the chart.
plt.tight_layout()
plt.savefig("dce_crime_rates.png", dpi=140, bbox_inches="tight", facecolor=BG)
plt.show()

The red bar is the prosecution rate, the teal bar the crime rate that DCE estimates from the prosecutions and the five features, and the grey bar the true crime rate, which I know only because I generated the crimes. The prosecution rate is about a fifth of the true crime rate, and the DCE estimate nine tenths of it.

Limitations

The synthetic test succeeds because the exclusion restrictions are true by construction. I generated info_value so that it does not enter the detection equation, and surveillance_signal so that it does not enter the crime equation.

In real data we have to argue that our crime-only variable does not affect detection, and vice versa. If we are wrong:

The estimated prosecution rate may still match the observed one, because the likelihood depends only on the product \(p \cdot d\).
The split between the crime probability and the detection probability can be biased.

Three things this synthetic test deliberately does not do, which the real papers do:

Confidence intervals. One sample gives one estimate, with no measure of its uncertainty. Patel and Putniņš resample their data 1,000 times, a bootstrap, to get standard errors, measures of how precise each estimate is, and intervals.

External validation. Patel and Putniņš go further with validation checks against cases where part of the truth is known: informal inquiries by the US Securities and Exchange Commission (SEC), and 724 earnings announcements traded by a group that hacked newswire services.

Argue the exclusion restrictions. The estimates depend on them, and they can fail in real data. A trader who expects strong enforcement may decide not to trade, and Patel and Putniņš report that insiders are less likely to trade illegally when the probability of detection is higher. Abnormal pre-announcement trading can also be a result of the crime. Whether our variables satisfy the restrictions is an economic argument that the code cannot settle.

The takeaway is that DCE can estimate a crime rate we never observe directly, and the estimate depends on how well the variables separate crime from detection. The two papers below give the fuller treatment: the variables they use, bootstrap inference and the validation tests.

References

Comerton-Forde, C., and Putniņš, T. J. (2014). Stock Price Manipulation: Prevalence and Determinants. Review of Finance 18(1): 23-66.

Feinstein, J. S. (1990). Detection Controlled Estimation. Journal of Law and Economics 33(1): 233-276.

Patel, V., and Putniņš, T. J. (2020). How Much Insider Trading Happens in Stock Markets? Working paper, SSRN 3764192.

Disclaimer: a synthetic illustration of an estimation method, for discussion purposes only. The probabilities recovered here describe data I generated. They do not describe any real market or any real party. Not investment advice.