Why do I lag the signal?

Shift a signal down one row so it sits in front of the return it earns, and see how much the total changes.

A signal computed from today’s close cannot be traded at today’s close. It is known only once the day is over, so the first return it can earn is tomorrow’s. Shifting the signal down one row, signal.shift(1), puts it in front of that return.

In this lesson I put the same signal against the same returns twice, once unlagged and once lagged, on five returns I can check by hand and then on AAPL, and I show that signal * ret.shift(-1) is the lagged version written the other way round.

Step 1. Five returns and a signal that already knows

Here is a signal that is 1 on every up day. It is built from the returns themselves, so in hindsight it is perfect.

import pandas as pd

ret    = pd.Series([0.10, -0.05, 0.04, -0.10, 0.06])
signal = (ret > 0).astype(float)             # 1 on up days, 0 on down days

print(pd.DataFrame({"ret": ret, "signal": signal, "position": signal.shift(1)}))
# ->     ret  signal  position
# -> 0  0.10     1.0       NaN
# -> 1 -0.05     0.0       1.0
# -> 2  0.04     1.0       0.0
# -> 3 -0.10     0.0       1.0
# -> 4  0.06     1.0       0.0
    ret  signal  position
0  0.10     1.0       NaN
1 -0.05     0.0       1.0
2  0.04     1.0       0.0
3 -0.10     0.0       1.0
4  0.06     1.0       0.0

shift(1) moves every value down one row. The 1 that row 0 read off its own return now sits on row 1, where it meets the next return instead.

Multiply each version by the returns.

same_day = signal * ret
lagged   = signal.shift(1) * ret

print(pd.DataFrame({"ret": ret, "same_day": same_day, "lagged": lagged}))
# ->     ret  same_day  lagged
# -> 0  0.10      0.10     NaN
# -> 1 -0.05     -0.00   -0.05
# -> 2  0.04      0.04    0.00
# -> 3 -0.10     -0.00   -0.10
# -> 4  0.06      0.06    0.00
    ret  same_day  lagged
0  0.10      0.10     NaN
1 -0.05     -0.00   -0.05
2  0.04      0.04    0.00
3 -0.10     -0.00   -0.10
4  0.06      0.06    0.00

The -0.00 entries are a zero position times a negative return. Read the two columns down. The same_day column keeps every gain and drops every loss. The lagged column keeps the losses, because the signal only turns on after an up day and this series alternates.

Compound each column with (1 + r).prod() - 1, and put buy and hold next to them.

print(f"same day: {(1 + same_day).prod() - 1:.4%}")           # -> same day: 21.2640%
print(f"lagged  : {(1 + lagged.dropna()).prod() - 1:.4%}")    # -> lagged  : -14.5000%
print(f"hold    : {(1 + ret).prod() - 1:.4%}")                # -> hold    : 3.6807%
same day: 21.2640%
lagged  : -14.5000%
hold    : 3.6807%

Same signal, same returns, 21.26% against -14.50%. The gap is entirely the row the signal sits on.

Step 2. Price above the 50-day average on AAPL

prices.csv sits next to this lesson and holds simulated daily closes for four tickers from 2020 to 2025. The rule is the one from Lesson 26: hold when the close is above its own 50-day average, hold nothing otherwise. The first 49 days have no average yet, so the signal is NaN there.

import numpy as np
import pandas as pd

prices = pd.read_csv("prices.csv", parse_dates=["Date"])
close  = prices[prices["Ticker"] == "AAPL"].set_index("Date")["Close"]

ret = close.pct_change()
sma = close.rolling(50).mean()

signal = (close > sma).astype(float)
signal[sma.isna()] = np.nan                  # no average, no signal

print(signal.dropna().head(3))
# -> Date
# -> 2020-03-10    1.0
# -> 2020-03-11    1.0
# -> 2020-03-12    1.0
# -> Name: Close, dtype: float64

print(len(signal.dropna()))                  # -> 1517
Date
2020-03-10    1.0
2020-03-11    1.0
2020-03-12    1.0
Name: Close, dtype: float64
1517

Now both versions.

same_day = (signal * ret).dropna()
lagged   = (signal.shift(1) * ret).dropna()

print(f"same day: {(1 + same_day).prod() - 1:.2%}")      # -> same day: 1162.54%
print(f"lagged  : {(1 + lagged).prod() - 1:.2%}")        # -> lagged  : 58.61%
print(f"buy hold: {(1 + ret.dropna()).prod() - 1:.2%}")  # -> buy hold: 165.45%
same day: 1162.54%
lagged  : 58.61%
buy hold: 165.45%

58.61% is the number this rule would have earned. 1162.54% is what you get by letting each day’s position be chosen with that day’s close already in hand. Buy and hold returned 165.45% over the same stretch, so the rule lost to holding the stock.

Step 3. The same thing written the other way

Instead of shifting the signal forward into tomorrow, you can shift the return backwards to meet today’s signal: signal * ret.shift(-1). Every product is identical, it just sits one row higher.

look = pd.DataFrame({"ret": ret,
                     "signal": signal,
                     "lagged": signal.shift(1) * ret,
                     "next_day": signal * ret.shift(-1)})

print(look.iloc[49:54].round(6))
# ->                  ret  signal    lagged  next_day
# -> Date
# -> 2020-03-10  0.031592     1.0       NaN -0.016424
# -> 2020-03-11 -0.016424     1.0 -0.016424 -0.012680
# -> 2020-03-12 -0.012680     1.0 -0.012680 -0.035732
# -> 2020-03-13 -0.035732     0.0 -0.035732 -0.000000
# -> 2020-03-16 -0.001451     0.0 -0.000000  0.000000
                 ret  signal    lagged  next_day
Date                                            
2020-03-10  0.031592     1.0       NaN -0.016424
2020-03-11 -0.016424     1.0 -0.016424 -0.012680
2020-03-12 -0.012680     1.0 -0.012680 -0.035732
2020-03-13 -0.035732     0.0 -0.035732 -0.000000
2020-03-16 -0.001451     0.0 -0.000000  0.000000

The -0.016424 that lagged reports on 11 March is the same figure next_day reports on 10 March. Both say the position taken on 10 March earned the return of 11 March. Multiply either column out and the total is the same.

next_day = (signal * ret.shift(-1)).dropna()

print(len(lagged), len(next_day))                     # -> 1516 1516
print(f"{(1 + lagged).prod() - 1:.10f}")              # -> 0.5861063842
print(f"{(1 + next_day).prod() - 1:.10f}")            # -> 0.5861063842
1516 1516
0.5861063842
0.5861063842

Pick one convention and keep it. I use signal.shift(1), because the shifted series is the position held on that date and everything later, from costs to drawdown, is computed date by date.

The rule: a signal built from data up to and including day t earns the return of day t + 1.

Your turn

Run the same 100-day rule on MSFT, both ways. How far apart are the two totals, and how does the lagged one compare with buy and hold?

import numpy as np
import pandas as pd

prices = pd.read_csv("prices.csv", parse_dates=["Date"])
close  = prices[prices["Ticker"] == "MSFT"].set_index("Date")["Close"]

ret = close.pct_change()
sma = close.rolling(100).mean()

signal = (close > sma).astype(float)
signal[sma.isna()] = np.nan

same_day = (signal * ret).dropna()
lagged   = (signal.shift(1) * ret).dropna()

print(f"same day: {(1 + same_day).prod() - 1:.2%}")      # -> same day: 819.93%
print(f"lagged  : {(1 + lagged).prod() - 1:.2%}")        # -> lagged  : 289.59%
print(f"buy hold: {(1 + ret.dropna()).prod() - 1:.2%}")  # -> buy hold: 416.98%