import numpy as np, itertools, matplotlib.pyplot as plt
RED, BLUE, GREY = "#C44E52", "#1F77B4", "#CCCCCC"
T, N, RHO, M = 180, 300, 0.90, 12
EDGE, HURDLE = 0.0014, 0.6
def make_panel(seed, edge=EDGE):
rng = np.random.default_rng(seed)
size = np.exp(rng.normal(0, 1.0, N))
sig = np.empty((T, N)); sig[0] = rng.normal(0, 1, N)
for t in range(1, T):
sig[t] = RHO*sig[t-1] + np.sqrt(1 - RHO**2)*rng.normal(0, 1, N)
ret = rng.normal(0, 0.06, (T, N))
ret[1:] += edge * sig[:-1] # last month's signal pays off this month
return size, sig, ret
panel = make_panel(0)Same data, same trading signal, different answer
Python
Backtesting
Code
On simulated data where the edge is real, the reported return runs from -1.6% to +6.3% a year, depending on eight ordinary portfolio choices.
Menkveld and co-authors (2024) gave 164 research teams the same data, 17 years of trading in EuroStoxx 50 futures, and the same six hypotheses to test. The teams came back with different answers. For one hypothesis, 125 of the 164 teams found a significant result. For another, six teams did. The authors call the spread the nonstandard error: the uncertainty that comes from who runs the analysis, on top of the uncertainty that comes from the data. In their experiment, the two were about the same size.
The spread has a plain source. Each hypothesis had to be turned into a measure, and each team chose its own: what to compute, how often to sample the data, and what to do with the outliers. The data leaves those choices open, the researcher fills them in, and Menkveld’s teams filled them in differently.
Their teams tested hypotheses. My question is whether the same thing happens when the output is a trading strategy. I cannot test that the way Menkveld did, because, as far as I know, no one has given 164 quants the same data and collected their backtests. So, in this case, I show the mechanism on a market where I know the truth. I simulate 300 stocks where a signal, in truth, predicts returns, and I build one long-short portfolio on that signal. Then I change how I build the portfolio, one choice at a time, the way another quant might have chosen, and read off what happens.
The plain build earns 2.74% a year at a Sharpe ratio of 0.73, and I would trade it. One ordinary change, a higher trading cost, takes the Sharpe ratio to 0.25, and I would not. Across every combination of the eight choices that build the portfolio, the same signal on the same data earns anywhere from -1.6% to +6.3% a year. The edge stays the same through all of it. Only the conclusion moves: trade the portfolio, or leave it.
The setup
- Market: 300 simulated stocks over 180 months.
- Signal: one number per stock that moves slowly, and, in truth, last month’s value predicts this month’s return.
- Portfolio: long the stocks with the highest signal, short the lowest, rebalanced monthly.
- Costs: charged on turnover, in basis points, where one basis point is 0.01%.
- Decision: I trade the portfolio if its Sharpe ratio clears 0.6. The level is my choice, and I vary it later.
The portfolio
Building the portfolio takes eight choices: how far back to average the signal, whether to trim extreme returns, which stocks to drop, how to weight the rest, how many groups to sort them into, how long to wait after the signal, how often to rebalance, and what one trade costs. The signal decides none of them.
def build(size, sig, ret, window, winsor, liquidity, weighting,
groups, lag, rebalance, cost_bps):
keep = size >= np.quantile(size, liquidity)
s, r, w = sig[:, keep].copy(), ret[:, keep], size[keep]
n = s.shape[1]
if window > 1: # average the signal
cs = np.vstack([np.zeros((1, n)), np.cumsum(s, axis=0)])
s[window-1:] = (cs[window:] - cs[:-window]) / window
if winsor: # trim extreme returns
lo, hi = np.percentile(r, [winsor, 100-winsor], axis=1, keepdims=True)
r = np.clip(r, lo, hi)
start, k, prev, pnl = lag + window - 1, max(1, n // groups), None, []
for t in range(start, T):
if prev is None or (t - start) % rebalance == 0:
o = np.argsort(s[t-lag]); wts = np.zeros(n)
if weighting == "equal":
wts[o[-k:]] = 1/k; wts[o[:k]] = -1/k
else:
hi_i, lo_i = o[-k:], o[:k]
wts[hi_i] = w[hi_i]/w[hi_i].sum(); wts[lo_i] = -w[lo_i]/w[lo_i].sum()
turn = np.abs(wts - (0 if prev is None else prev)).sum()
prev = wts
else:
turn = 0.0
pnl.append(wts @ r[t] - turn*cost_bps/10000)
return np.asarray(pnl)
def report(p):
return p.mean()*M*100, p.mean()/p.std(ddof=1)*np.sqrt(M) # return % a year, Sharpe
BASE = dict(window=1, winsor=0.0, liquidity=0.0, weighting="equal",
groups=5, lag=1, rebalance=1, cost_bps=10.0)
a0, s0 = report(build(*panel, **BASE))
print(f"the plain build: {a0:+.2f}% a year, Sharpe ratio {s0:.2f}")the plain build: +2.74% a year, Sharpe ratio 0.73The plain build takes a simple option at every choice: the raw one-month signal, no trimming, every stock, equal weights, quintiles, trade one month after the signal, rebalance monthly, and charge 10 basis points. It earns 2.74% a year at a Sharpe ratio of 0.73, which clears my cutoff. I would trade this.
One change at a time
Now I change one choice and leave the other seven alone. Two of the choices, which stocks to drop and what one trade costs, have two alternatives each, so the eight choices give ten single changes.
FORKS = dict(window=[1, 3], winsor=[0.0, 1.0], liquidity=[0.0, 0.2, 0.4],
weighting=["equal", "size"], groups=[5, 10], lag=[1, 2],
rebalance=[1, 3], cost_bps=[0.0, 10.0, 25.0])
KEYS = list(FORKS)
LABELS = {"window": "average the signal over 3 months", "winsor": "trim returns at 1%",
"liquidity": "drop the smallest", "weighting": "weight by size",
"groups": "hold deciles", "lag": "trade a month later",
"rebalance": "rebalance quarterly", "cost_bps": "charge"}
rows = []
for k in KEYS:
for v in FORKS[k]:
if v == BASE[k]:
continue
a, s = report(build(*panel, **{**BASE, k: v}))
name = LABELS[k]
if k == "liquidity": name = f"drop the smallest {int(v*100)}%"
if k == "cost_bps": name = f"charge {int(v)} bp"
rows.append((name, a, s))
for name, a, s in sorted(rows, key=lambda x: x[2]):
print(f"{name:32s} return {a:+5.2f}% Sharpe {s:.2f}")charge 25 bp return +0.93% Sharpe 0.25
weight by size return +2.68% Sharpe 0.45
trade a month later return +2.10% Sharpe 0.52
drop the smallest 40% return +2.52% Sharpe 0.52
hold deciles return +3.36% Sharpe 0.57
drop the smallest 20% return +2.58% Sharpe 0.61
rebalance quarterly return +2.71% Sharpe 0.66
average the signal over 3 months return +2.78% Sharpe 0.72
trim returns at 1% return +2.78% Sharpe 0.75
charge 0 bp return +3.95% Sharpe 1.04rows_sorted = sorted(rows, key=lambda x: x[2])
fig, ax = plt.subplots(figsize=(9, 5))
y = np.arange(len(rows_sorted))
for i, (name, a, s) in enumerate(rows_sorted):
ax.plot([s0, s], [i, i], color=GREY, lw=1.4, zorder=1)
ax.scatter(s, i, s=70, color=RED if s < HURDLE else BLUE, zorder=3)
ax.axvline(s0, color="#444", lw=1.4, ls="-", label="the plain build (0.73)")
ax.axvline(HURDLE, color=RED, lw=1.0, ls="--", label=f"my cutoff ({HURDLE})")
ax.set_yticks(y); ax.set_yticklabels([r[0] for r in rows_sorted])
ax.set_xlabel("Sharpe ratio")
ax.set_title("One change to the portfolio, ten times over", fontsize=13)
ax.legend(frameon=False, loc="lower right")
plt.tight_layout(); plt.savefig("nse_single.png", dpi=140, bbox_inches="tight"); plt.show()
Each grey line starts at the plain build and ends where one change leaves the Sharpe ratio. Five of the ten changes take the portfolio below my cutoff. Charging 25 basis points instead of 10 takes the Sharpe ratio from 0.73 to 0.25. Weighting by size takes it to 0.45. Holding deciles raises the return to 3.36% and lowers the Sharpe ratio to 0.57, so the headline return improves while the portfolio turns worse. And each of these changes is one line of code, with a reason a reviewer would accept.
Two quants
A quant makes all eight choices at once, so I now make them the way a second quant might: one who builds the same idea to hold more money. This quant drops the smallest 40% of stocks because small stocks are hard to trade, weights by size because big positions need big stocks, rebalances quarterly to trade less, and charges 25 basis points because trading at size costs more.
SECOND = {**BASE, "liquidity": 0.4, "weighting": "size",
"rebalance": 3, "cost_bps": 25.0}
a2, s2 = report(build(*panel, **SECOND))
print(f"the second quant: {a2:+.2f}% a year, Sharpe ratio {s2:.2f}")
print(f"volatility: the plain build {a0/s0:.1f}% a year, the second quant {a2/s2:.1f}%")
fig, ax = plt.subplots(figsize=(9, 4.4))
ax.plot(np.cumsum(build(*panel, **BASE))*100, color=BLUE, lw=2.2, label="the plain build")
ax.plot(np.cumsum(build(*panel, **SECOND))*100, color=RED, lw=2.2, label="the second quant")
ax.axhline(0, color="#888", lw=0.8)
ax.set_xlabel("months"); ax.set_ylabel("cumulative return (%)")
ax.set_title("The same signal, built two ways", fontsize=13); ax.legend(frameon=False)
plt.show()the second quant: +2.90% a year, Sharpe ratio 0.40
volatility: the plain build 3.8% a year, the second quant 7.2%
The second portfolio earns 2.90% a year at a Sharpe ratio of 0.40, which sits below my cutoff. The edge is still in the data, at the size I set. So, the first quant trades this strategy, the second quant drops it, and the whole gap between them comes from how the portfolio is built. The second build even earns more, 2.90% against 2.74%, and it takes nearly twice the volatility to do it, 7.2% against 3.8% a year. So, the higher return and the lower Sharpe ratio describe the same portfolio, and which build looks better depends on which number the reader compares.
Every combination
Two builds are two points. Running every combination of the eight choices gives 576 versions of the same portfolio. Reporting all of them instead of one preferred build follows the specification curve of Simonsohn and co-authors (2020).
COMBOS = list(itertools.product(*FORKS.values()))
res = np.array([report(build(*panel, **dict(zip(KEYS, c)))) for c in COMBOS])
ann, sharpe = res[:, 0], res[:, 1]
print(f"{len(COMBOS)} builds: {ann.min():+.2f}% to {ann.max():+.2f}% a year,"
f" Sharpe {sharpe.min():.2f} to {sharpe.max():.2f}")
for h in (0.5, 0.6, 0.7):
print(f" above a Sharpe of {h}: {(sharpe > h).sum()} of {len(sharpe)}")576 builds: -1.60% to +6.31% a year, Sharpe -0.17 to 1.07
above a Sharpe of 0.5: 211 of 576
above a Sharpe of 0.6: 126 of 576
above a Sharpe of 0.7: 70 of 576The reported return spans -1.60% to +6.31% a year, on one dataset with one edge. Moving the cutoff does not remove the spread: at a Sharpe ratio of 0.5, 211 of the 576 builds pass, and at 0.7, 70 pass.
Does the pattern hold?
One dataset is one draw, so I repeat the whole exercise on 100 fresh datasets, each built with the same edge.
SINGLES = [(k, v) for k in KEYS for v in FORKS[k] if v != BASE[k]]
flips = []
for i in range(100):
p = make_panel(4000+i)
b = report(build(*p, **BASE))[1]
flips.append(sum((report(build(*p, **{**BASE, k: v}))[1] > HURDLE) != (b > HURDLE)
for k, v in SINGLES))
flips = np.array(flips)
print(f"at least one single change crosses the cutoff: {(flips > 0).mean():.0%} of datasets")
print(f"median number that do: {np.median(flips):.0f} of {len(SINGLES)}")at least one single change crosses the cutoff: 79% of datasets
median number that do: 2 of 10On the first dataset, five of the ten changes crossed my cutoff, which is a high count. Across the 100 datasets, the median is two changes, and in 79% of them at least one change moves the portfolio across the cutoff. So, which change flips the conclusion depends on the dataset, and in most datasets at least one change does.
What this test does not settle
Eight parameters are not 164 teams. Menkveld’s teams disagreed about what the hypothesis meant, which test to run, and how often to sample the data, and one team reported a trend of +74,491%. My eight choices copy none of that. Real quants also cluster on the same conventions instead of spreading evenly across every combination. So the test shows the mechanism, and it does not measure how much real quants disagree.
Some builds are different strategies. A size-weighted decile portfolio can be called a different strategy from an equal-weighted quintile one. Part of the spread is disagreement about one portfolio, and part is two portfolios.
The cost numbers are choices too. I charge one flat rate on turnover. Novy-Marx and Velikov (2016) show that realistic costs vary across strategies and across stocks, so my 10 and 25 basis points are choices of mine, like everything else in the build.
Simulated data shows the mechanism and not the size. I set the edge so that the plain build clears 0.6 with something to spare. A stronger edge survives all ten changes, and a weaker one fails all of them.
To conclude
The edge was in the data the whole time. How I built the portfolio decided the conclusion, and every choice in the build was ordinary.
That is what the nonstandard error means for a backtest. A single Sharpe ratio is one build out of hundreds the reader never sees, and two quants with the same data can come back with a 0.73 and a 0.40, one on each side of the cutoff, without either of them making an error. Menkveld found one more thing worth carrying over: after the peer-feedback stages of his experiment, where teams read reviews of their work and saw the best papers, the spread across teams fell by 47%. Seeing the other builds made the spread smaller.
So, report the choices next to the result. Give the turnover and the cost you charged. Show the result with and without the stocks you dropped. And when the portfolio sits near the cutoff, say which choices would carry it across, because the next quant will make some of them differently.
The takeaway is that a backtest reports one build of the strategy, so before trading it, ask how many of the other reasonable builds agree.
Read next
- Backtests tell you sweet little lies What trading costs did to a momentum portfolio on Swedish stocks.
- Does your backtested Sharpe ratio show a real edge? The other problem with one number: too few observations behind it.
Disclaimer: a simplified, hypothetical backtest on simulated data, for discussion purposes only. I put the edge into the data myself. Not investment advice. Past performance does not predict future returns.
Sources: Albert Menkveld and co-authors, Nonstandard Errors, Journal of Finance (2024). Amar Soebhag, Bart van Vliet and Patrick Verwijmeren, Non-standard errors in asset pricing: Mind your sorts, Journal of Empirical Finance (2024). Robert Novy-Marx and Mihail Velikov, A Taxonomy of Anomalies and Their Trading Costs, Review of Financial Studies (2016). Uri Simonsohn, Joseph Simmons and Leif Nelson, Specification curve analysis, Nature Human Behaviour (2020).