How do I test many parameters at once?

Run the backtest function once for every value in a list of parameters and collect the results in one table.

A sweep runs the same backtest once for every value in a list and puts the results in one table. The list here is the SMA window, and each run gives back a total return, a Sharpe ratio and a turnover.

In this lesson I sweep five windows on AAA, put buy and hold in the same table, then run the sweep on a second ticker and on each half of the sample.

Step 1. The prices and the function from Lesson 34

prices.csv sits next to this lesson and holds simulated data: six years of daily closes for four tickers. close_of pulls one ticker out as a Series indexed by date.

import numpy as np
import pandas as pd

prices = pd.read_csv("prices.csv", parse_dates=["Date"])    # long form: one row per ticker per day

def close_of(ticker):                                       # one ticker out as a dated price Series
    rows = prices[prices["Ticker"] == ticker]               # keep only that ticker's rows
    return rows.set_index("Date")["Close"].sort_index()     # Date as the index, oldest first

aapl = close_of("AAA")                                     # six years of daily closes

print(len(aapl))          # -> 1566

print(aapl.head(2))
# -> Date
# -> 2020-01-01    75.08
# -> 2020-01-02    76.94
# -> Name: Close, dtype: float64

1566
Date
2020-01-01    75.08
2020-01-02    76.94
Name: Close, dtype: float64

Here is the function from Lesson 34 again. It takes a price Series and a window, buys when the close is above its own moving average, lags the signal by a day, and returns three numbers.

def backtest(close, window, cost=0.0):                          # one price Series in, three numbers out
    ret      = close.pct_change()                               # the stock's own daily return
    sma      = close.rolling(window).mean()                     # trailing mean over the last window days
    signal   = (close > sma).astype(float)                      # 1.0 on days the close sits above it
    position = signal.shift(1)                                  # yesterday's signal is today's position
    trades   = position.diff().abs()                            # 1.0 on each day the position flips
    strat    = (position * ret - cost * trades).dropna()        # earned on the days we were in
    return {
        "total":    (1 + strat).prod() - 1,                     # compounded growth over the whole run
        "sharpe":   strat.mean() / strat.std() * np.sqrt(252),  # daily ratio scaled up to a year
        "turnover": trades.sum(),                               # count of switches in and out
    }

one = backtest(aapl, 50)                                        # a single run before sweeping a list

print(f"total {one['total']:.2%}   sharpe {one['sharpe']:.2f}   turnover {one['turnover']:.0f}")
# -> total 58.61%   sharpe 0.50   turnover 113

total 58.61%   sharpe 0.50   turnover 113

I leave cost at zero for the whole lesson so the only thing changing between runs is the window.

Step 2. Five windows, one table

The sweep is a loop over the list. Each pass calls backtest, tags the result with the window it came from, and appends the dictionary to rows. pd.DataFrame turns a list of dictionaries into a table with one row per dictionary.

windows = [20, 50, 100, 150, 200]             # the five parameter values to try
rows    = []                                  # one result dictionary per run lands here

for w in windows:
    result = backtest(aapl, w)                # a full backtest for this one window
    result["rule"] = f"sma {w}"               # tag the result with the window it came from
    rows.append(result)

sweep = pd.DataFrame(rows).set_index("rule")  # list of dictionaries becomes one row each

print(sweep.round(4))
# ->           total  sharpe  turnover
# -> rule
# -> sma 20   1.0932  0.7576     173.0
# -> sma 50   0.5861  0.5040     113.0
# -> sma 100  0.3079  0.3312      95.0
# -> sma 150  0.3359  0.3530      63.0
# -> sma 200  0.1826  0.2408      69.0

          total  sharpe  turnover
rule                             
sma 20   1.0932  0.7576     173.0
sma 50   0.5861  0.5040     113.0
sma 100  0.3079  0.3312      95.0
sma 150  0.3359  0.3530      63.0
sma 200  0.1826  0.2408      69.0

total is a fraction, so 1.0932 is a gain of 109.32%. The short window returns most and trades most: 173 switches in and out against 69 for the 200-day window.

Step 3. Buy and hold in the same table

A sweep with nothing to compare against only ranks the tries against each other. Buy and hold is a position of 1.0 every day, so its total is the compounded daily return and its turnover is zero.

ret = aapl.pct_change().dropna()                        # the stock's returns, no rule applied

sweep.loc["buy and hold"] = {                           # a sixth row on the finished table
    "total":    (1 + ret).prod() - 1,                   # in every day, so no lag and no trades
    "sharpe":   ret.mean() / ret.std() * np.sqrt(252),  # annualised the same way as in backtest
    "turnover": 0.0,                                    # never traded, so nothing to charge
}

print(sweep.sort_values("total", ascending=False).round(4))
# ->                total  sharpe  turnover
# -> rule
# -> buy and hold  1.6545  0.7865       0.0
# -> sma 20        1.0932  0.7576     173.0
# -> sma 50        0.5861  0.5040     113.0
# -> sma 150       0.3359  0.3530      63.0
# -> sma 100       0.3079  0.3312      95.0
# -> sma 200       0.1826  0.2408      69.0

               total  sharpe  turnover
rule                                  
buy and hold  1.6545  0.7865       0.0
sma 20        1.0932  0.7576     173.0
sma 50        0.5861  0.5040     113.0
sma 150       0.3359  0.3530      63.0
sma 100       0.3079  0.3312      95.0
sma 200       0.1826  0.2408      69.0

All five windows return less than buy and hold on this sample, and all five have a lower Sharpe. The best of the five, 20 days, earns 109.32% against 165.45% for holding the stock.

Step 4. The same sweep on a second ticker

Now the same loop over both AAA and DDD, with a 10-day window added to the list. I keep only the total so the two columns sit side by side.

nvda = close_of("DDD")                               # a second ticker, same six years
wide = [10, 20, 50, 100, 200]                         # a 10-day window joins the list

both = pd.DataFrame(index=wide, columns=["AAA", "DDD"], dtype=float)
for w in wide:
    both.loc[w, "AAA"] = backtest(aapl, w)["total"]  # keep the total, drop the other two
    both.loc[w, "DDD"] = backtest(nvda, w)["total"]  # same window, other ticker

both.index.name = "window"

print(both.round(4))
# ->           AAA     DDD
# -> window
# -> 10      0.9631  4.0776
# -> 20      1.0932  2.7945
# -> 50      0.5861  1.0602
# -> 100     0.3079  0.9117
# -> 200     0.1826  0.8039

           AAA     DDD
window                
10      0.9631  4.0776
20      1.0932  2.7945
50      0.5861  1.0602
100     0.3079  0.9117
200     0.1826  0.8039

idxmax() returns the index label of the largest value in each column.

print(both.idxmax())
# -> AAA    20
# -> DDD    10
# -> dtype: int64

AAA    20
DDD    10
dtype: int64

The winner on AAA is 20 days and the winner on DDD is 10 days. On AAA the 10-day window comes second, and on DDD it beats the 20-day window by 128 percentage points.

Step 5. First half against second half

aapl.index[len(aapl) // 2] is the date halfway down the sample. Slicing the price Series at that date gives two stretches of roughly equal length, and I sweep each one on its own.

cut = aapl.index[len(aapl) // 2]            # the date halfway down the sample

first  = aapl.loc[:cut]                     # opening stretch, up to and including cut
second = aapl.loc[cut:]                     # closing stretch, so cut sits in both

print(cut.date(), len(first), len(second))   # -> 2023-01-02 784 783

2023-01-02 784 783

halves = pd.DataFrame(index=windows, columns=["first", "second"], dtype=float)

for w in windows:
    halves.loc[w, "first"]  = backtest(first, w)["total"]   # the whole sweep on the early years
    halves.loc[w, "second"] = backtest(second, w)["total"]  # the same sweep on the late years

halves.index.name = "window"                               # name the index so it prints

print(halves.round(4))
# ->          first  second
# -> window
# -> 20      0.5117  0.4056
# -> 50      0.2763  0.3098
# -> 100     0.1151  0.1942
# -> 150    -0.0119  0.4593
# -> 200    -0.1202  0.4306

         first  second
window                
20      0.5117  0.4056
50      0.2763  0.3098
100     0.1151  0.1942
150    -0.0119  0.4593
200    -0.1202  0.4306

print(halves.idxmax())
# -> first      20
# -> second    150
# -> dtype: int64

first      20
second    150
dtype: int64

The 20-day window wins the first half with 51.17% and the 150-day window wins the second with 45.93%. The 150-day window lost 1.19% in the first half, and the 200-day window lost 12.02% in the first half and made 43.06% in the second.

A sweep tells you how each parameter did on the sample you ran it on. It does not tell you which one will do best next. Picking the highest number from a list of tries measures the list as well as the rule, because the more values you try the higher that top number goes on its own.

Further reading: Why a model with no predictive power can score 90% accuracy.

Your turn

Run the Step 2 sweep on CCC over the same five windows, add buy and hold, and sort by total. Which window returns most, and does any window beat holding CCC?

Show answer

msft = close_of("CCC")                       # a third ticker, same six years
rows = []

for w in [20, 50, 100, 150, 200]:
    result = backtest(msft, w)                # one full run per window
    result["rule"] = f"sma {w}"
    rows.append(result)

sweep = pd.DataFrame(rows).set_index("rule")  # five runs stacked into one table

ret = msft.pct_change().dropna()              # returns for the buy and hold row
sweep.loc["buy and hold"] = {
    "total":    (1 + ret).prod() - 1,
    "sharpe":   ret.mean() / ret.std() * np.sqrt(252),
    "turnover": 0.0,
}

print(sweep.sort_values("total", ascending=False).round(4))
# ->                total  sharpe  turnover
# -> rule
# -> buy and hold  4.1698  1.3756       0.0
# -> sma 20        3.9881  1.6637     155.0
# -> sma 50        3.9022  1.5744      75.0
# -> sma 200       3.0213  1.3704      19.0
# -> sma 100       2.8959  1.3392      49.0
# -> sma 150       2.5776  1.2506      45.0

The 20-day window returns most at 398.81% and none of the five beats holding CCC at 416.98%.