import numpy as np
import matplotlib.pyplot as plt
RED, BLUE, GREY = "#C44E52", "#1F77B4", "#CCCCCC"
D = 252 # trading days in a year
def sharpe(x):
x = np.asarray(x)
return x.mean() / x.std(ddof=1) * np.sqrt(D)The art of thinking clearly as a quant
Python
Statistics
Code
Six biases that distort quantitative research, each one measured on a simulated market where the truth is known in advance.
I recently read Rolf Dobelli’s The Art of Thinking Clearly. It made me think about how the same biases enter quantitative research.
Code produces the output. A person still chooses the hypothesis, the data, the tests, and how the evidence is read. So, the quality of a quantitative result rests on judgments that the code cannot check.
This post is about biases, and writing about those has one difficulty: in real data we never learn what the truth was, so we cannot measure the magnitude of the bias. So, in this case, I simulate six small markets instead so that we know the truth. Then I create each bias.
The first bias leads to a report of an average return of 2.2% for stocks a year, where the truth is zero. The second leads to a correlation of 0.86 between two series that, in truth, share nothing. The third leads to a Sharpe ratio of 3.79 in a market with no trading edge. Six biases follow, with one safeguard for each.
1. Survivorship bias
Say we have a thousand stocks and, in truth, the average return is zero. Over ten years, two hundred of them fall so far that they stop trading. If we now measure the average return of the stocks that are still listed, we get 2.2% a year.
So, the issue is that, not accounting for the delisted firms (survivorship bias), we falsely believe that the average return of the stocks is 2.2%. This deviation from the truth can, of course, be much larger in magnitude.
rng = np.random.default_rng(1)
n, T, sd = 1000, 10 * D, 0.02
path = np.cumprod(1 + rng.normal(0, sd, (T, n)), axis=0) # average return of zero
hit = path < 0.20 # the level at which a stock stops trading
dead = hit.any(axis=0)
for j in np.where(dead)[0]: # a stock that stops trading stays at that level
path[hit[:, j].argmax():, j] = 0.20
survivors = path[:, ~dead].mean(axis=1) # the file we are left with
everyone = path.mean(axis=1) # what actually happened
print(f"survivors {(survivors[-1]**0.1 - 1)*100:+.1f}% a year, "
f"everyone {(everyone[-1]**0.1 - 1)*100:+.1f}%, {dead.sum()} of {n} stopped trading")
yrs = np.arange(T) / D
fig, ax = plt.subplots(figsize=(9, 5))
for j in np.where(dead)[0][:80]:
ax.plot(yrs, path[:, j], color=GREY, lw=0.5)
ax.plot(yrs, survivors, color=RED, lw=2.4, label="survivors")
ax.plot(yrs, everyone, color=BLUE, lw=2.4, label="everyone")
ax.axhline(1.0, color="#888", lw=0.8)
ax.set_xlabel("years"); ax.set_ylabel("value of 1")
ax.set_title("Survivors vs everyone", fontsize=13); ax.legend(frameon=False)
plt.show()survivors +2.2% a year, everyone +0.3%, 199 of 1000 stopped trading
The blue line is the average of every stock, including the ones that stopped trading, and it comes out at 0.3% a year, close to the truth of zero. The grey lines are eighty of the two hundred stocks that stopped trading along the way. The red line is the average of the stocks that are still listed at the end, and it comes out at 2.2% a year. So, the gap between the red line and the blue line is the bias.
Safeguard. Some data vendors mark the stocks that stopped trading, so first check whether the data has such a mark, and whether stocks with the mark are in the sample. If there is no mark, look at the cumulative return of each stock from the earliest date in the sample. Over ten years, a thousand stocks should hold some that lost close to everything, so if the worst cumulative return in the data is, for example, around -20%, the stocks that stopped trading are probably missing. This check is not as good as the mark, and it still shows whether the stocks that stopped trading are in the data.
2. Spurious correlation
So the bias came from data that went missing. The next one comes from data where everything is present.
Say we have two price series and, in truth, nothing connects them. I build each one separately, and I give each one an upward drift, so both of them rise over the period. If we now correlate the two price levels, we get 0.86.
So, the issue is that, not accounting for the upward trend in each series (spurious correlation), we falsely believe that the two series are related. The 0.86 only measures that both series rise over the period. The correlation can, of course, come out even closer to one, and it would still not mean that the series are related.
rng = np.random.default_rng(2)
a = 50 + np.cumsum(0.10 + rng.normal(0, 1.0, 600)) # its own drift and noise
b = 100 + np.cumsum(0.30 + rng.normal(0, 2.0, 600)) # drawn separately from a
r_lvl = np.corrcoef(a, b)[0, 1] # what the levels report
r_chg = np.corrcoef(np.diff(a), np.diff(b))[0, 1] # what the changes report
print(f"levels r = {r_lvl:.2f}, changes r = {r_chg:.2f}")
fig, ax = plt.subplots(1, 2, figsize=(10, 4.6))
ax[0].scatter(a, b, s=7, color=RED, alpha=0.5)
ax[0].set_title(f"Levels, r = {r_lvl:.2f}", fontsize=12)
ax[1].scatter(np.diff(a), np.diff(b), s=7, color=BLUE, alpha=0.5)
ax[1].set_title(f"Changes, r = {r_chg:.2f}", fontsize=12)
for p in ax:
p.set_xlabel("series A"); p.set_ylabel("series B")
plt.show()levels r = 0.86, changes r = -0.02
The left panel plots the two price levels against each other, and the right panel plots their daily changes. The daily changes have no trend in them, so the correlation falls from 0.86 to -0.02. Granger and Newbold (1974) showed the same problem in regressions: two trending series that share nothing can still look strongly related.
Safeguard. Start with the economics. Ask whether there is any reason these two series should move together, because no test supplies that reason, and every test will sometimes report a relationship where there is none. Then, if we want to know whether two series move together over the long run, a cointegration test says more than a correlation does. It also reports relationships that are not there, less often than a correlation, and often enough that the economics still has to come first.
3. Confirmation bias
Say we have a strategy that spreads money across thirty assets and, in truth, none of those assets can be predicted. The first test disappoints, so we change how much the strategy holds of one asset and test it again on the same data. The Sharpe ratio improves, so we keep the change. Then we try another change. If we now do that fifteen hundred times, keeping every change that improves the Sharpe ratio, we get 3.79.
So, the issue is that, keeping only the changes that improve the Sharpe ratio (confirmation bias), we falsely believe that we have a strategy with a Sharpe ratio of 3.79. When we run the same strategy on data that the tuning has never seen, the Sharpe ratio comes out at -0.26.
rng = np.random.default_rng(3)
N = 30
tuned_on = rng.normal(0, 0.01, (3 * D, N)) # the data we keep testing on
fresh = rng.normal(0, 0.01, (10 * D, N)) # data the tuning never sees
w = np.zeros(N); w[rng.integers(N)] = 1.0
best = sharpe(tuned_on @ w)
in_path, out_path = [], []
for _ in range(1500):
j = rng.integers(N)
trial = w.copy(); trial[j] += rng.normal(0, 0.25) # one random adjustment
s = sharpe(tuned_on @ trial)
if s > best: # keep it only if it helps
w, best = trial, s
in_path.append(best)
out_path.append(sharpe(fresh @ w))
print(f"tuned {in_path[-1]:.2f}, fresh {out_path[-1]:.2f}")
fig, ax = plt.subplots(figsize=(9, 5))
ax.plot(in_path, color=RED, lw=2.0, label="the data we tuned on")
ax.plot(out_path, color=BLUE, lw=2.0, label="fresh data")
ax.axhline(0, color="#888", lw=0.8)
ax.set_xlabel("adjustments tried"); ax.set_ylabel("Sharpe")
ax.set_title("Tuned data vs fresh data", fontsize=13); ax.legend(frameon=False)
plt.show()tuned 3.79, fresh -0.26
The loop does not train a model. Every change is random, and we keep a change only when it raises the Sharpe ratio on the data we test on. Repeating that fifteen hundred times is enough to push the Sharpe ratio to 3.79. So, we can overfit without any machine learning, just by choosing which changes to keep.
Safeguard. Before the test, write down what result would make us drop the idea. Count every change we make along the way. And keep a part of the data that none of those changes has touched, because that part is the only one that can still tell us anything.
4. Cherry picking
The bias above came from adjusting one strategy until it worked. The next one comes from running many strategies and reporting only one.
Say we build two hundred versions of a strategy and, in truth, none of them has an edge. If we now take whichever version scored highest, we get a Sharpe ratio of 1.53, against an average across all two hundred of 0.04.
So, the issue is that, reporting only the best of the two hundred versions (cherry picking), we falsely believe that we have a strategy with a Sharpe ratio of 1.53. The 1.53 comes from picking the highest number out of two hundred tries, and the more versions we test, the higher that number gets, even though no version, in truth, has an edge.
rng = np.random.default_rng(4)
strat = rng.normal(0, 0.01, (5 * D, 200)) # 200 strategies with no edge
sh = np.array([sharpe(strat[:, i]) for i in range(200)])
print(f"best {sh.max():.2f}, average {sh.mean():.2f}")
fig, ax = plt.subplots(figsize=(9, 5))
ax.hist(sh, bins=25, color=BLUE, alpha=0.85)
ax.axvline(sh.max(), color=RED, lw=2.4)
ax.text(sh.max() - 0.04, ax.get_ylim()[1] * 0.9, "the one we report",
ha="right", color=RED, fontsize=11)
ax.axvline(0, color="#888", lw=0.8)
ax.set_xlabel("Sharpe"); ax.set_ylabel("strategies")
ax.set_title("200 strategies, no edge", fontsize=13)
plt.show()best 1.53, average 0.04
Each of the two hundred versions gets one Sharpe ratio, and the histogram shows all of them. They centre on zero, because that is the truth I built in. The red line is the version we would have reported, and it comes out at 1.53 even though, in truth, no version has an edge.
Safeguard. Count every version we test, including the ones we throw away. The count matters, because luck alone raises the best of many versions: here, the best of two hundred worthless versions came out at 1.53. So, before we trust the best version, work out what the best of that many worthless versions produces, and trust the 1.53 only if it is far above that number. Bailey and López de Prado (2014) use the number of trials and the properties of the Sharpe-ratio estimates to calculate a deflated Sharpe ratio.
5. Regression to the mean
Say we have a thousand fund managers and, in truth, every one of them has the same skill, which is none. We rank them on their returns over the first five years and keep the best tenth, who returned 12.3% a year. If we now follow those same managers through the next five years, we get -1.5%.
So, the issue is that, treating the strong record as skill and ignoring regression to the mean, we falsely believe that the top tenth will keep returning 12.3% a year. In this simulation, every manager has a skill of zero, so the ranking is pure luck, and the next five years start from zero again. In real data, a record can contain both skill and luck, and even then, part of what put a manager at the top is luck.
rng = np.random.default_rng(5)
M = 1000
first = ((1 + rng.normal(0, 0.01, (5 * D, M))).prod(axis=0) ** 0.2 - 1) * 100
nxt = ((1 + rng.normal(0, 0.01, (5 * D, M))).prod(axis=0) ** 0.2 - 1) * 100
top = first >= np.percentile(first, 90) # the best tenth on the first period
print(f"top decile {first[top].mean():+.1f}% then {nxt[top].mean():+.1f}%")
fig, ax = plt.subplots(figsize=(9, 5))
for i in np.where(top)[0]:
ax.plot([0, 1], [first[i], nxt[i]], color=GREY, lw=0.7)
ax.plot([0, 1], [first[top].mean(), nxt[top].mean()], color=RED, lw=3,
marker="o", ms=8, label="top decile")
ax.plot([0, 1], [first.mean(), nxt.mean()], color=BLUE, lw=2.4,
marker="o", ms=7, label="everyone")
ax.axhline(0, color="#888", lw=0.8)
ax.set_xticks([0, 1]); ax.set_xticklabels(["first 5 years", "next 5 years"])
ax.set_ylabel("annual return (%)")
ax.set_title("The top decile, five years on", fontsize=13); ax.legend(frameon=False)
plt.show()top decile +12.3% then -1.5%
Each grey line is one manager from the top tenth. They start bunched together at the top, because that is how they were chosen, and they spread back across the whole range once the next five years arrive.
Safeguard. When a strong record turns weak, first work out what plain luck would have produced. Look for a reason only if the fall is larger than that.
6. Outcome bias
The five biases above all made a worthless result look valuable. The last one works the other way round, and throws away a strategy that should have been kept.
Say we have a strategy that makes money on average, at a Sharpe ratio of 0.5, so in truth the right decision is to keep trading it. We do not know that yet, so we give it one year and judge it on what that year returns. If we now simulate five thousand separate first years for this same strategy, a third of them lose money.
So, the issue is that, stopping the strategy because its first year lost money (outcome bias), we falsely believe we have removed a bad strategy. All five thousand of those years came from the same strategy, with the same edge and the same risk, and chance is the only thing that separates a winning year from a losing one. A rule that stops after one losing year therefore throws away a third of the good strategies it is applied to. Baron and Hershey (1988) found that people rate identical decisions more favourably once they learn the outcome was favourable.
rng = np.random.default_rng(6)
mu = 0.5 * 0.01 / np.sqrt(D) # daily drift for a true Sharpe of 0.5
year = ((1 + rng.normal(mu, 0.01, (D, 5000))).prod(axis=0) - 1) * 100
print(f"{(year < 0).mean()*100:.0f}% of first years end down")
fig, ax = plt.subplots(figsize=(9, 5))
bins = np.linspace(year.min(), year.max(), 45)
ax.hist(year[year >= 0], bins=bins, color=BLUE, alpha=0.85, label="kept")
ax.hist(year[year < 0], bins=bins, color=RED, alpha=0.85, label="cut")
ax.axvline(0, color="#888", lw=1.0)
ax.set_xlabel("first-year return (%)"); ax.set_ylabel("cases")
ax.set_title("One year from the same process", fontsize=13); ax.legend(frameon=False)
plt.show()34% of first years end down
Every bar in the chart, red and blue, is a first year from the same strategy. The red ones are the years we would stop after, and the only thing that puts a year in the red group is chance.
Safeguard. Keep a journal. Before the first year, write down what we believe about the strategy and what we expect it to return. When the year is over, compare the expectation with what actually happened, and if the two deviate, ask where the deviation comes from. It can come from randomness, as in the chart above, or from something real that breaks the reasoning we wrote down. Without the journal, the outcome is the only thing left to judge.
What this does not settle
Simulated data shows the mechanism and not the size. Each market above was built so that I knew the answer before the test ran, which is what lets me measure the bias. How large the same bias is in real data depends on the data, on how freely we search, and on how often we check a result.
The biases overlap. Cherry picking and confirmation bias are the same move at two different points, one while building a strategy and one while choosing which strategy to show. A single decision usually carries several of these at once. Six sections is a way to see them one at a time, and it is not the way they arrive.
A safeguard is not a method. Writing something down before the answer is known helps, and it does not turn a judgment into a measurement. It puts the judgment somewhere it can be argued with later.
To conclude
Knowing the names does not remove the biases. A person is present at every stage where these biases can enter: choosing the sample, defining the test, changing the model, and deciding what the result means.
Most of the safeguards above are about timing. We write something down before we see the result: the rule that would make us drop the idea, the count of the versions we test, and the journal with what we expect the strategy to return. The rest compare the result with what luck alone produces: the best of two hundred worthless versions, the strong record that turns weak, and the first year that ends in a loss. And the safeguard for survivorship bias puts the stocks that stopped trading back into the data.
None of this takes much time, and all of it has to be in place before we see the result. After that, the biases above decide what we believe.
Read next
- When out of sample stops being out of sample The confirmation-bias section above, taken apart properly.
- Does your backtested Sharpe ratio show a real edge? The cherry-picking section above, and a bootstrap that measures it.
- Does the price of butter predict the S&P 500? The spurious-correlation section above, on two real series, with the test that settles it.
Disclaimer: a teaching example on simulated data. Every number here comes from a market I built myself, where the truth was fixed before any test ran. Not investment advice.
Sources: Rolf Dobelli, The Art of Thinking Clearly, Harper, 2013, which prompted this post. Stephen Brown, William Goetzmann, Roger Ibbotson and Stephen Ross, Survivorship Bias in Performance Studies, Review of Financial Studies (1992). Clive Granger and Paul Newbold, Spurious Regressions in Econometrics, Journal of Econometrics (1974). Peter Wason, On the Failure to Eliminate Hypotheses in a Conceptual Task, Quarterly Journal of Experimental Psychology (1960). Tyler Shumway, The Delisting Bias in CRSP Data, Journal of Finance (1997). David Bailey and Marcos López de Prado, The Deflated Sharpe Ratio, Journal of Portfolio Management (2014). Jonathan Baron and John Hershey, Outcome Bias in Decision Evaluation (1988).