Correlations with US stocks depend on which five years you measure
Diversification comes from taking uncorrelated bets with positive expected returns. But the correlation itself varies over time, so depending on which five years you look at, you get different answers.
I take US stocks and six asset classes and show how the correlation of each asset class with US stocks changes over time. The six are US Treasuries, corporate bonds, gold, commodities, developed markets outside the US and emerging markets. I could have picked more or fewer asset classes, but I chose these six to keep it simple.
Over the five years to July 2013, the correlation with US stocks was +0.75 for commodities and −0.31 for US Treasuries. Over the five years to July 2026, it was +0.01 for commodities and +0.53 for US Treasuries, the opposite order.
The correlation also changes with how often you measure the return, monthly, weekly or daily.
The data
US stocks are Kenneth French’s daily market portfolio. Gold, commodities, developed markets outside the US and emerging markets come from LSEG Workspace. My LSEG account has no total-return bond index, so I build the US Treasury and corporate bond series from yields: the 10-year Treasury yield and Moody’s Baa corporate yield from FRED. The table gives the series for each asset class and what its return includes. Appendix A has the downloads, how I build the bond series and a first check, and Appendix B has two more checks.
| Asset class | Series | What it includes |
|---|---|---|
| US stocks | Kenneth French’s daily market portfolio | dividends, US dollars |
| US Treasuries | a 10-year par bond rebuilt from FRED’s DGS10 | coupon and price change |
| Corporate bonds | a modelled 10-year par bond, priced off Moody’s Baa yield, FRED’s DBAA | coupon and price change |
| Gold | LSEG XAU=, spot, US dollars an ounce | price only, no income |
| Commodities | S&P GSCI Total Return | spot, roll and Treasury bill collateral |
| Developed ex-US | MSCI EAFE gross, large and mid cap outside the US and Canada | dividends reinvested, US dollars |
| Emerging markets | MSCI Emerging Markets gross, monthly before 2000 | dividends reinvested, US dollars |
The correlation
For each series, I take the price or index level on the last trading day of each month and compute monthly returns. I drop any month whose last trading day falls more than a week before month-end, so a partial month never enters as a full month.
def period_level(daily_level, rule="M", stale_days=STALE_DAYS):
"""One level per calendar period, taken from the last day the series traded in that period.
It drops a period when that last trading day is more than `stale_days` before the period
ends, so a series that starts or stops midway never contributes a part-month return as though
it were a whole month, and it drops a period the calendar has not finished yet.
"""
traded = daily_level.dropna()
period = traded.index.to_period(rule)
last_level = traded.groupby(period).last()
last_day = pd.Series(traded.index, index=period).groupby(level=0).last()
period_end = last_level.index.to_timestamp(how="end").normalize()
stale = (period_end - pd.DatetimeIndex(last_day.to_numpy())).days > stale_days
unfinished = period_end >= TODAY
# reindex onto every period between the first and the last, so a hole breaks the chain
# and a two-month return never enters as a one-month return
every_period = pd.period_range(last_level.index[0], last_level.index[-1], freq=rule)
return last_level.where(~(stale | unfinished)).reindex(every_period)
monthly_level = pd.DataFrame({name: period_level(levels[name]) for name in levels.columns})
monthly_return = monthly_level.pct_change(fill_method=None)
print(f"monthly returns: {monthly_return.index[0]} to {monthly_return.index[-1]}")
print(f" US stocks {int(monthly_return['stocks'].notna().sum()):,} months, "
f"the last one {monthly_return['stocks'].last_valid_index()}")monthly returns: 1926-07 to 2026-09
US stocks 1,200 months, the last one 2026-07
I take a five-year window of 60 monthly returns and compute each asset class’s correlation with US stocks. Then I make it rolling, moving the window forward one month at a time and computing the correlation again. The first window ends in December 1992, because MSCI’s emerging-market index starts at the end of 1987, and the last ends in July 2026, so I get 404 windows.
def rolling_correlation(period_return, months=WINDOW_MONTHS):
"""Correlation with US stocks over the last `months` periods, one estimate per period end."""
estimates = {}
for name in ASSET_NAMES:
pair = period_return[["stocks", name]].dropna() # keep a month only if both reported
pair = pair.reindex(period_return.index) # put the gaps back where they were
# rolling() counts the months it can see, so a window holding a gap gives no estimate
estimates[name] = pair[name].rolling(months).corr(pair["stocks"])
return pd.DataFrame(estimates)
correlation = rolling_correlation(monthly_return).dropna() # start where all six have a window
correlation = correlation.loc[:LAST_MONTH] # and stop at the reported snapshot
assert correlation.abs().max().max() <= 1, "a correlation outside -1 to +1"
# every twelfth row is every twelfth calendar month only while no month in between is missing
unbroken = pd.period_range(correlation.index[0], correlation.index[-1], freq="M")
assert correlation.index.equals(unbroken), "a month is missing between the first and the last"
print(f"{len(correlation)} monthly estimates, each from {WINDOW_MONTHS} monthly returns, "
f"ending {correlation.index[0]} to {correlation.index[-1]}")404 monthly estimates, each from 60 monthly returns, ending 1992-12 to 2026-07
One window a year
Then I put the correlations on a video, so you can see how they change. The video rolls one year at a time, showing 34 of the 404 windows: the five years to July 1993, the five years to July 1994, and so on to the five years to July 2026. The numbers in this section come from these 34 windows.
def window_label(month):
"""The months one window covers: 2026-07 with a five-year window gives Aug 2021 to Jul 2026."""
first_month = month - WINDOW_MONTHS + 1
return f"{first_month.strftime('%b %Y')} – {month.strftime('%b %Y')}"
# counting back from the latest window, so the video always ends on the most recent one
anchor_positions = sorted(range(len(correlation) - 1, -1, -STEP_MONTHS))
anchors = correlation.iloc[anchor_positions]
assert anchors.notna().all().all(), "a window with no estimate would break the interpolation"
print(f"{len(anchors)} windows a year apart, {window_label(anchors.index[0])} "
f"to {window_label(anchors.index[-1])}")
print(pd.DataFrame({
"Lowest": {ASSET_LABEL[n]: f"{anchors[n].min():+.2f}" for n in ASSET_NAMES},
"Lowest ends": {ASSET_LABEL[n]: anchors[n].idxmin().strftime("%b %Y") for n in ASSET_NAMES},
"Highest": {ASSET_LABEL[n]: f"{anchors[n].max():+.2f}" for n in ASSET_NAMES},
"Highest ends": {ASSET_LABEL[n]: anchors[n].idxmax().strftime("%b %Y") for n in ASSET_NAMES},
"Latest": {ASSET_LABEL[n]: f"{anchors[n].iloc[-1]:+.2f}" for n in ASSET_NAMES}}).to_string())
print(f"lowest of all {len(correlation)} monthly windows: "
f"Developed ex-US {correlation['developed'].min():+.2f}, "
f"Emerging markets {correlation['emerging'].min():+.2f}")34 windows a year apart, Aug 1988 – Jul 1993 to Aug 2021 – Jul 2026
Lowest Lowest ends Highest Highest ends Latest
US Treasuries -0.56 Jul 2014 +0.53 Jul 2026 +0.53
Corporate bonds -0.30 Jul 2015 +0.70 Jul 2024 +0.69
Gold -0.35 Jul 1993 +0.22 Jul 2024 +0.17
Commodities -0.40 Jul 1993 +0.75 Jul 2013 +0.01
Developed ex-US +0.32 Jul 1997 +0.92 Jul 2012 +0.78
Emerging markets +0.39 Jul 1997 +0.86 Jul 2012 +0.64
lowest of all 404 monthly windows: Developed ex-US +0.27, Emerging markets +0.36
Developed markets outside the US and emerging markets had a positive correlation with US stocks in all 34 windows. The correlation was between +0.32 and +0.92 for developed markets outside the US, and between +0.39 and +0.86 for emerging markets.
For US Treasuries and commodities, the order reversed. I compare the latest window, the five years to July 2026, with the five years to July 2013, the window in which commodities had their highest correlation with US stocks.
EARLIER = pd.Period("2013-07", "M")
LATEST = correlation.index[-1]
print(pd.DataFrame({
window_label(EARLIER): {ASSET_LABEL[n]: f"{anchors.loc[EARLIER, n]:+.2f}" for n in ASSET_NAMES},
window_label(LATEST): {ASSET_LABEL[n]: f"{anchors.loc[LATEST, n]:+.2f}" for n in ASSET_NAMES}}).to_string()) Aug 2008 – Jul 2013 Aug 2021 – Jul 2026
US Treasuries -0.31 +0.53
Corporate bonds +0.26 +0.69
Gold +0.17 +0.17
Commodities +0.75 +0.01
Developed ex-US +0.91 +0.78
Emerging markets +0.86 +0.64
In the five years to July 2013, US Treasuries had the lower correlation with US stocks. In the five years to July 2026, commodities had the lower correlation. So, if you compare the two asset classes over only the latest five years, you do not see the opposite order from the five years to July 2013.
Monthly, weekly and daily returns
So far, I have used monthly correlations, and monthly returns absorb the daily variation I could also have measured. So I use the sample period from August 2021 to July 2026 and show how correlation varies across monthly, weekly, and daily returns.
def correlation_in_window(period_return, month_of_row, first_month, last_month):
"""Correlation with US stocks for every asset class, over one window of calendar months.
month_of_row gives the calendar month each row belongs to, so a weekly or a daily panel can
be cut by the same window as the monthly one. Each asset keeps every period it and US stocks
both traded, so a market with its own holidays does not shorten the others.
"""
inside = (month_of_row >= first_month) & (month_of_row <= last_month)
window = period_return[inside]
estimates = {}
for name in ASSET_NAMES:
pair = window[["stocks", name]].dropna()
estimates[ASSET_LABEL[name]] = pair["stocks"].corr(pair[name])
return pd.Series(estimates)
weekly_level = pd.DataFrame({name: period_level(levels[name], rule="W-FRI", stale_days=4)
for name in levels.columns})
weekly_return = weekly_level.pct_change(fill_method=None)
daily_return = pd.DataFrame({name: levels[name].dropna().pct_change() for name in levels.columns})
FIRST_MONTH = LATEST - WINDOW_MONTHS + 1
weekly_month = weekly_return.index.to_timestamp(how="end").to_period("M")
measured = pd.DataFrame({
"Monthly": correlation_in_window(monthly_return, monthly_return.index, FIRST_MONTH, LATEST),
"Weekly": correlation_in_window(weekly_return, weekly_month, FIRST_MONTH, LATEST),
"Daily": correlation_in_window(daily_return, daily_return.index.to_period("M"), FIRST_MONTH, LATEST)})
print(f"{window_label(LATEST)}, the same five years measured three ways:")
print(measured.map(lambda value: f"{value:+.2f}").to_string())Aug 2021 – Jul 2026, the same five years measured three ways:
Monthly Weekly Daily
US Treasuries +0.53 +0.12 +0.07
Corporate bonds +0.69 +0.30 +0.16
Gold +0.17 +0.16 +0.14
Commodities +0.01 +0.11 +0.10
Developed ex-US +0.78 +0.71 +0.39
Emerging markets +0.64 +0.53 +0.24
For five of the six asset classes, the correlation decreases as the returns get more frequent. For US Treasuries, it was +0.53 on monthly returns, +0.12 on weekly returns and +0.07 on daily returns. For emerging markets, it was +0.64, +0.53 and +0.24, and for gold, +0.17, +0.16 and +0.14. Commodities are the exception, with +0.10 on daily returns against +0.01 on monthly ones.
So the correlation, or more correctly the estimated correlation, changes over time, but it also changes with the frequency you use. The higher the frequency, the more variation you include, but that extra variation can also be noise.
There is no right or wrong when it comes to which frequency you should use, because in the end, monthly, weekly and daily returns all measure the correlation. But when you estimate a correlation, you can get noise, and that noise can come from the sample period and also from the frequency. Daily returns, for example, can contain mechanical effects of different closing times. A foreign market closes before the US market, so its reaction to the same news can show up a day later in daily returns, while a monthly return puts both moves in the same month. Appendix B checks this closing-time explanation.
The video
FRAMES_PER_STEP = 36 # frames between two estimates, 0.6s at 60 fps
FPS = 60
HOLD_END = 3.0 # seconds the last window stays on screen
# One shape-preserving curve through all of the yearly estimates, read off once per frame.
# PchipInterpolator passes exactly through every estimate and does not overshoot between
# neighbouring ones. It interpolates the values in between for the animation, and those are not
# correlation estimates. Easing each pair of years separately would stop every dot at every one.
curve = PchipInterpolator(np.arange(len(anchors)), anchors.to_numpy(), axis=0)
position = np.arange((len(anchors) - 1) * FRAMES_PER_STEP + 1) / FRAMES_PER_STEP # counted in estimates
path = curve(position)
on_estimate = np.flatnonzero(np.isclose(position, np.round(position))) # frames on an estimate
assert np.allclose(path[on_estimate], anchors.to_numpy()), "the curve misses an estimate"
for column, name in enumerate(ASSET_NAMES):
lowest, highest = anchors[name].min(), anchors[name].max()
assert path[:, column].min() >= lowest - 1e-9, f"{name}: the curve goes below every estimate"
assert path[:, column].max() <= highest + 1e-9, f"{name}: the curve goes above every estimate"
named_window = np.round(position).astype(int) # the window whose months the corner shows
frames = [(path[row], named_window[row]) for row in range(len(position))]
frames = frames + [frames[-1]] * round(HOLD_END * FPS)
# While the corner holds one window, the number beside a dot is already travelling towards the
# next year's estimate. Only the frame on an estimate is the estimate for that window.
drift = np.abs(path - anchors.to_numpy()[named_window])
print(f"{len(frames)} frames, {len(frames) / FPS:.1f} seconds, "
f"{window_label(anchors.index[0])} to {window_label(anchors.index[-1])}")
print("between one estimate and the next the number beside a dot differs")
print(f" from the window named by {drift.mean():.3f} on average and {drift.max():.3f} at most")1369 frames, 22.8 seconds, Aug 1988 – Jul 1993 to Aug 2021 – Jul 2026
between one estimate and the next the number beside a dot differs
from the window named by 0.017 on average and 0.249 at most
import cv2 # pip install --no-deps opencv-python
VIDEO_WIDTH, VIDEO_HEIGHT, DPI = 1440, 960, 120
RECT = [0.088, 0.105, 0.875, 0.775] # the plot, with room above it for the dates
COLUMN = np.arange(len(ASSETS)) # each asset keeps its own column, whatever it does
figure = plt.figure(figsize=(VIDEO_WIDTH / DPI, VIDEO_HEIGHT / DPI), dpi=DPI)
def draw_frame(values, label):
"""One video frame as a BGR array: one dot per asset class, the window's months in the corner.
Redrawing the whole figure each frame is slower than updating the dots in place, and it keeps
the code readable, which matters more here.
"""
figure.clf()
figure.patch.set_facecolor(BG)
ax = figure.add_axes(RECT)
ax.set_facecolor(BG)
ax.set_xlim(-0.65, len(ASSETS) - 0.35)
ax.set_ylim(-1, 1) # a correlation cannot leave this range
ax.set_yticks([-1, -0.5, 0, 0.5, 1])
ax.yaxis.set_major_formatter(mticker.FuncFormatter(lambda v, _: "0" if v == 0 else f"{v:+.1f}"))
ax.set_xticks(COLUMN)
ax.set_xticklabels([label for _, label, _ in ASSETS])
ax.grid(True, axis="y", color=GRID, lw=1.1)
ax.set_axisbelow(True)
for side in ("top", "right"):
ax.spines[side].set_visible(False)
for side in ("left", "bottom"):
ax.spines[side].set_color(SPINE)
ax.tick_params(axis="both", length=0, labelsize=13, colors=INK, pad=10)
ax.axhline(0, color=INK, lw=1.8, zorder=3) # the line the dots move around
ax.set_ylabel("Correlation with US stocks", fontsize=17, color=AXIS_TEXT, labelpad=14)
for column, (_, _, colour), value in zip(COLUMN, ASSETS, values):
ax.plot(column, value, "o", ms=16, color=colour, zorder=4, clip_on=False)
ax.annotate(f"{value:+.2f}", (column, value), xytext=(19, 0), textcoords="offset points",
ha="left", va="center", fontsize=18, fontweight="semibold", color=colour,
annotation_clip=False, zorder=5,
# the patch keeps the number legible when a dot is on the zero line
bbox=dict(facecolor=BG, edgecolor="none", pad=1.5))
figure.text(0.966, 0.945, label, ha="right", va="center", fontsize=25, color=INK)
figure.canvas.draw()
return cv2.cvtColor(np.asarray(figure.canvas.buffer_rgba()), cv2.COLOR_RGBA2BGR)# Media Foundation is Windows only, and its H.264 output is what a browser can decode. On another
# platform pass cv2.CAP_FFMPEG here instead, which needs an H.264 encoder available to FFmpeg. On
# this machine that would be OpenH264, and it is not installed.
writer = cv2.VideoWriter(RAW_MP4, cv2.CAP_MSMF, cv2.VideoWriter_fourcc(*"H264"),
FPS, (VIDEO_WIDTH, VIDEO_HEIGHT))
# a writer that failed to open discards every frame in silence, so check it rather than trust it
assert writer.isOpened(), "OpenCV could not open the video writer"
started = time.time()
for number, (values, window) in enumerate(frames):
writer.write(draw_frame(values, window_label(anchors.index[window])))
if (number + 1) % 300 == 0:
print(f" {number + 1} of {len(frames)} frames ({time.time() - started:.0f}s)", flush=True)
writer.release()
plt.close(figure)
print(f"rendered {len(frames)} frames in {(time.time() - started) / 60:.1f} min") 300 of 1369 frames (29s)
600 of 1369 frames (58s)
900 of 1369 frames (80s)
1200 of 1369 frames (105s)
rendered 1369 frames in 2.0 min
def mp4_boxes(buffer, start=0, end=None):
"""Every box in one stretch of an MP4 file: its type, where it starts, its size, its header."""
end = len(buffer) if end is None else end
found, position = [], start
while position + 8 <= end:
size, kind = struct.unpack(">I4s", buffer[position:position + 8])
header = 8
if size == 1: # a box above 4 GB states its size in 64 bits
size, header = struct.unpack(">Q", buffer[position + 8:position + 16])[0], 16
elif size == 0: # a size of zero means the box runs to the end
size = end - position
found.append((kind.decode("latin-1"), position, size, header))
if size < header:
break
position += size
return found
def move_index_to_front(source, target):
"""Write the file again with its index ahead of the frames, so playback can start early.
An MP4 keeps the frames in one box (mdat) and the index of where each chunk of them starts in
another (moov). OpenCV writes mdat first, and an index at the end can delay the start of
playback. Moving moov in front pushes every chunk that much further into the file, so each
offset it holds gains moov's own size.
"""
buffer = bytearray(open(source, "rb").read())
top_level = mp4_boxes(buffer)
order = [kind for kind, _, _, _ in top_level]
assert "moov" in order and "mdat" in order, order
_, moov_at, moov_size, moov_header = top_level[order.index("moov")]
index = bytearray(buffer[moov_at:moov_at + moov_size])
def add_to_offsets(box, start, end):
for kind, position, size, header in mp4_boxes(box, start, end):
if kind in ("trak", "mdia", "minf", "stbl", "edts"):
add_to_offsets(box, position + header, position + size) # these hold other boxes
elif kind in ("stco", "co64"): # the chunk offsets, 32 bit and 64 bit versions
count = struct.unpack(">I", box[position + header + 4:position + header + 8])[0]
pattern, width = (">I", 4) if kind == "stco" else (">Q", 8)
at = position + header + 8
for _ in range(count):
moved = struct.unpack(pattern, box[at:at + width])[0] + moov_size
box[at:at + width] = struct.pack(pattern, moved)
at += width
add_to_offsets(index, moov_header, moov_size)
mdat_at = top_level[order.index("mdat")][1]
rebuilt = buffer[:mdat_at] + index + buffer[mdat_at:moov_at] + buffer[moov_at + moov_size:]
open(target, "wb").write(rebuilt)
return [kind for kind, _, _, _ in mp4_boxes(rebuilt)]
print(f"boxes in order: {move_index_to_front(RAW_MP4, OUT_MP4)}")
os.remove(RAW_MP4)
print(f"{os.path.basename(OUT_MP4)}: {VIDEO_WIDTH}x{VIDEO_HEIGHT} at {FPS} fps, {len(frames)} frames, "
f"{len(frames) / FPS:.1f}s, {os.path.getsize(OUT_MP4) / 1e6:.1f} MB")boxes in order: ['ftyp', 'uuid', 'moov', 'mdat']
correlation.mp4: 1440x960 at 60 fps, 1369 frames, 22.8s, 5.6 MB
Each dot is one asset class’s correlation with US stocks over the five years named in the corner, and each asset class keeps its own column. Between two yearly windows, the video interpolates the dots and the numbers beside them, so only the frames on a yearly window show a calculated correlation. So, the order of the dots changes from one five-year window to the next.
Limitations
I focus only on correlation to connect it to diversification. But volatility also matters, because a portfolio’s volatility depends on the volatility of each asset, separately from the correlation.
I derive the prices of both bond series from their yields because the data I have access to contains only yields.
Conclusion
Diversification, when you look only at correlation, changes over time, and depending on which five years you look at, you will get different answers. It also changes depending on how often you measure returns. So it is better to be careful about concluding too fast by looking only at the correlation over the past five years.
Appendix A: downloading and building the data
I set the window length, video speed, and the six asset classes in one block, so I only need to change them in one place.
import io
import os
import struct
import time
import urllib.request
import warnings
import zipfile
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
from scipy.interpolate import PchipInterpolator # the curve the dots follow between two years
warnings.filterwarnings("ignore")
plt.rcParams["font.family"] = ["Segoe UI", "Arial", "DejaVu Sans"] # the video's typography
DATA_DIR = os.path.expanduser("~") # anywhere you can write
STOCK_CSV = os.path.join(DATA_DIR, "us_stock_daily.csv")
TREASURY_YIELD_CSV = os.path.join(DATA_DIR, "treasury_yield_daily.csv")
CORPORATE_YIELD_CSV = os.path.join(DATA_DIR, "corporate_yield_daily.csv")
LSEG_CSV = os.path.join(DATA_DIR, "lseg_levels_daily.csv")
RAW_MP4 = os.path.join(DATA_DIR, "correlation_raw.mp4") # the encoder writes here first
OUT_MP4 = os.path.join(DATA_DIR, "correlation.mp4")
START_DATE = "1960-01-01" # ask early, each source clips to what it has
END_DATE = str(pd.Timestamp.now().date())
TODAY = pd.Timestamp.now().normalize()
WINDOW_MONTHS = 60 # five years of monthly returns in every estimate
LAST_MONTH = pd.Period("2026-07", "M") # the snapshot reported here, so a later download
# cannot move the windows the prose quotes
STEP_MONTHS = 12 # the video passes through one estimate a year
STALE_DAYS = 7 # how old a month's last trade may be before it is dropped
BOND_MATURITY = 10.0 # constant maturity, in years
# One row per asset class: the name used in the code, the label on the axis, and its colour.
ASSETS = [("treasury", "US Treasuries", "#17868A"),
("corporate", "Corporate bonds", "#6B8E23"),
("gold", "Gold", "#D2822B"),
("commodities", "Commodities", "#C0392B"),
("developed", "Developed ex-US", "#2B6CB0"),
("emerging", "Emerging markets", "#7C5CBF")]
ASSET_NAMES = [name for name, _, _ in ASSETS]
ASSET_LABEL = {name: label for name, label, _ in ASSETS}
BG, INK, GRID, SPINE, AXIS_TEXT = "#FCEFE3", "#1f1f1f", "#EADCCC", "#D5C6B4", "#4a4a4a"I download US stocks from Kenneth French’s website and the two yields from FRED. Moody’s holds the copyright on the Baa yield, so I do not republish it.
FRENCH_ZIP = ("https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/"
"F-F_Research_Data_Factors_daily_CSV.zip")
FRED_CSV = "https://fred.stlouisfed.org/graph/fredgraph.csv?id={}&cosd=1900-01-01"
def download(url, tries=4):
"""Fetch one file, retrying with a growing pause, then raising so a failure is never silent."""
for attempt in range(tries):
try:
request = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (research script)"})
with urllib.request.urlopen(request, timeout=120) as response:
return response.read()
except Exception as error:
if attempt == tries - 1:
raise
wait = 15 * (attempt + 1)
print(f" download failed ({type(error).__name__}), retrying in {wait}s")
time.sleep(wait)
def us_stock_level():
"""Daily wealth index of the US market portfolio, dividends included, starting at 1.
Kenneth French publishes the market return in excess of the risk-free rate (Mkt-RF) and the
risk-free rate (RF) as separate columns, both in percent, so this function adds the two back
together to get the return on the market itself.
"""
archive = zipfile.ZipFile(io.BytesIO(download(FRENCH_ZIP)))
text = archive.read(archive.namelist()[0]).decode("latin-1").splitlines()
header = 0 # the file opens with a paragraph of notes
for row, line in enumerate(text):
if "Mkt-RF" in [cell.strip() for cell in line.split(",")]:
header = row
break
# an annual table follows the daily one, and only the daily rows start with a date
dated = [line for line in text[header + 1:] if line.strip()[:8].isdigit()]
factors = pd.read_csv(io.StringIO("\n".join(dated)), header=None,
names=["date"] + [cell.strip() for cell in text[header].split(",")[1:]])
factors.index = pd.to_datetime(factors.pop("date").astype(str), format="%Y%m%d")
market_return = (factors["Mkt-RF"] + factors["RF"]) / 100
return (1 + market_return).cumprod()
def save_fred_yield(series_id, path):
"""Save one daily yield series from FRED, in percent as it is published."""
text = download(FRED_CSV.format(series_id)).decode("utf-8").splitlines()
dated = [line for line in text if not line.endswith(".")] # FRED prints a missing day as a dot
table = pd.read_csv(io.StringIO("\n".join(dated)))
yields = pd.to_numeric(table.iloc[:, 1], errors="coerce")
yields.index = pd.to_datetime(table.iloc[:, 0])
yields.dropna().sort_index().rename("yield").rename_axis("Date").to_csv(path, encoding="utf-8-sig")
if not os.path.exists(STOCK_CSV):
us_stock_level().rename("level").rename_axis("Date").to_csv(STOCK_CSV, encoding="utf-8-sig")
if not os.path.exists(TREASURY_YIELD_CSV):
save_fred_yield("DGS10", TREASURY_YIELD_CSV) # 10-year constant-maturity Treasury yield
if not os.path.exists(CORPORATE_YIELD_CSV):
save_fred_yield("DBAA", CORPORATE_YIELD_CSV) # Moody's seasoned Baa corporate bond yield
stock_level = pd.read_csv(STOCK_CSV, parse_dates=["Date"]).set_index("Date")["level"]
treasury_yield = pd.read_csv(TREASURY_YIELD_CSV, parse_dates=["Date"]).set_index("Date")["yield"] / 100
corporate_yield = pd.read_csv(CORPORATE_YIELD_CSV, parse_dates=["Date"]).set_index("Date")["yield"] / 100
print(f"US stocks {len(stock_level):,} days {stock_level.index[0]:%d %b %Y} to {stock_level.index[-1]:%d %b %Y}")
print(f"10-year yield {len(treasury_yield):,} days {treasury_yield.index[0]:%d %b %Y} to {treasury_yield.index[-1]:%d %b %Y}")
print(f"Baa yield {len(corporate_yield):,} days {corporate_yield.index[0]:%d %b %Y} to {corporate_yield.index[-1]:%d %b %Y}")US stocks 26,296 days 01 Jul 1926 to 31 Jul 2026
10-year yield 16,161 days 02 Jan 1962 to 15 Sep 2026
Baa yield 10,214 days 02 Jan 1986 to 17 Sep 2026
Then I download gold, commodities, developed markets outside the US, and emerging markets from LSEG, along with two bond funds, IEF and LQD, which I use to check my bond series. The LSEG data are licensed, so the block prints only the number of days and the start date for each series.
In this extract, LSEG quotes half the gold price before 16 October 1989, so I double every gold price before that day.
LSEG_SERIES = {"gold": ("XAU=", "MID_PRICE"), # gold spot, US dollars an ounce
"commodities": (".SPGSCITR", "TR.PriceClose"), # S&P GSCI, spot, roll and collateral
"developed": (".MIEA00000GUS", "TRDPRC_1"), # MSCI EAFE, dividends reinvested
"emerging": (".MIEF00000GUS", "TRDPRC_1"), # MSCI Emerging Markets, dividends reinvested
"treasury_etf": ("IEF.O", "TRDPRC_1"), # iShares 7-10 year Treasury, from 2002
"corporate_etf": ("LQD.P", "TRDPRC_1")} # iShares investment grade corporates, from 2002
# The largest daily step to accept from a feed without looking at it. A bigger step is a reason to
# inspect the series before trusting it. MSCI publishes the gross emerging index monthly
# before 2000, so one of its steps can cover a whole month.
LARGEST_STEP = {"emerging": 0.45}
GOLD_SCALE_BREAK = pd.Timestamp("1989-10-16") # the day XAU= starts quoting the whole gold price
def patch_httpx_proxy():
"""lseg-data 2.1.1 passes proxy={} into httpx, and httpx 0.28 and later reject a dict."""
import httpx
def coerce(kwargs):
proxy = kwargs.get("proxy")
if isinstance(proxy, dict): # {} or {"http": None} both mean no proxy at all
kwargs["proxy"] = next((value for value in proxy.values() if value), None)
return kwargs
for client_class in (httpx.Client, httpx.AsyncClient):
if getattr(client_class.__init__, "_patched", False):
continue # idempotent, so re-running this block is harmless
original = client_class.__init__
def make(original):
def __init__(self, *args, **kwargs):
return original(self, *args, **coerce(kwargs))
__init__._patched = True
return __init__
client_class.__init__ = make(original)
def download_lseg(ric, field, tries=6):
"""Download one daily series from LSEG, retrying on a rate limit.
Anything other than a rate limit stops the download, so a real error surfaces instead of
being retried into silence.
"""
for attempt in range(tries):
try:
history = ld.get_history(universe=ric, fields=[field],
start=START_DATE, end=END_DATE, interval="daily")
return history.iloc[:, 0].dropna().sort_index()
except Exception as error:
rate_limited = ("Too many requests" in str(error)) or ("429" in str(error))
if not rate_limited or attempt == tries - 1:
raise
wait = 30 * (2 ** attempt) # 30s, 60s, 120s, 240s, 480s
print(f" rate-limited, waiting {wait}s", flush=True)
time.sleep(wait)
if not os.path.exists(LSEG_CSV):
patch_httpx_proxy()
import lseg.data as ld # needs Workspace running and logged in on this machine
ld.open_session() # no try block: without a session this has to stop
feeds = {name: download_lseg(ric, field) for name, (ric, field) in LSEG_SERIES.items()}
ld.close_session()
pd.concat(feeds, axis=1).sort_index().rename_axis("Date").to_csv(LSEG_CSV, encoding="utf-8-sig")
lseg = pd.read_csv(LSEG_CSV, parse_dates=["Date"]).set_index("Date").sort_index()
# In this extract XAU= quotes half the gold price before 16 October 1989, so the block doubles
# everything before that day to put the whole series on one scale. It measures the step on the days
# gold itself traded, because the frame also holds rows from the other five feeds. It keeps raw_gold
# because the check block reports what the correction is worth.
raw_gold = lseg["gold"].copy()
traded_gold = lseg["gold"].dropna()
step = (traded_gold / traded_gold.shift(1)).dropna()
doubling = step[(step > 1.6) & (step < 2.4)]
assert list(doubling.index) == [GOLD_SCALE_BREAK], f"doublings found: {list(doubling.index.date)}"
lseg.loc[lseg.index < GOLD_SCALE_BREAK, "gold"] *= 2
for name in lseg.columns:
biggest_step = lseg[name].dropna().pct_change().abs().max()
limit = LARGEST_STEP.get(name, 0.25)
assert biggest_step < limit, f"a step of {biggest_step:.0%} in {name}: check for a change of scale"
series = lseg[name].dropna()
print(f" {name:<14} {LSEG_SERIES[name][0]:<16} {len(series):6,} days from {series.index[0]:%d %b %Y}")
print(f"gold: scale corrected before {doubling.index[0]:%d %b %Y}") gold XAU= 14,747 days from 22 Mar 1968
commodities .SPGSCITR 14,296 days from 02 Jan 1970
developed .MIEA00000GUS 14,797 days from 31 Dec 1969
emerging .MIEF00000GUS 6,866 days from 31 Dec 1987
treasury_etf IEF.O 6,075 days from 26 Jul 2002
corporate_etf LQD.P 5,743 days from 30 Aug 2002
gold: scale corrected before 16 Oct 1989
For the bonds, I have to build the returns myself from the yields. At each date, I reprice the bond from the previous date at the new yield, add the interest it earned, and then buy a new ten-year bond, so the series always holds a ten-year bond. For the corporate bonds, I use Moody’s Baa yield as if it were the yield of a ten-year bond, although it is an average over bonds with twenty years or more left.
def par_price(yield_now, coupon_rate, maturity):
"""Price per 100 of a bond with semiannual coupons, given an annual yield and coupon rate."""
yield_now = np.maximum(yield_now, 1e-8) # a zero yield divides by zero in the annuity term
discount = (1 + yield_now / 2) ** (-2 * maturity)
return 100 * ((coupon_rate / yield_now) * (1 - discount) + discount)
def total_return_from_yield(bond_yield, maturity=BOND_MATURITY):
"""Total return of a modelled bond held at a constant maturity, rebuilt from its yield.
At each observation this function reprices the bond at the new yield with the elapsed calendar
time taken off its maturity, then adds the coupon accrued over that time. It sets the coupon
rate to the previous observation's yield and treats that as the par yield for `maturity`, so
the bond keeps a ten-year maturity instead of ageing like one particular note. Where the input is not a yield on
a bond of that maturity, as with Moody's Baa average, the result is a model and not an index.
"""
# observations can be days apart over a weekend, so the step is the elapsed calendar time.
# clip(lower=1) stops a zero-day gap, which would make the accrued coupon zero
year_fraction = bond_yield.index.to_series().diff().dt.days.clip(lower=1) / 365.0
coupon = bond_yield.shift(1) # the previous observation's yield, used as a par rate
price_change = (par_price(bond_yield.to_numpy(), coupon.to_numpy(),
maturity - year_fraction.to_numpy()) - 100) / 100
accrued = coupon.to_numpy() * year_fraction.to_numpy()
return pd.Series(price_change + accrued, index=bond_yield.index).dropna()
treasury_level = (1 + total_return_from_yield(treasury_yield)).cumprod()
corporate_level = (1 + total_return_from_yield(corporate_yield)).cumprod()
levels = pd.concat([stock_level.rename("stocks"), treasury_level.rename("treasury"),
corporate_level.rename("corporate"), lseg], axis=1).sort_index()
print(f"daily levels: {len(levels):,} days {levels.index[0]:%d %b %Y} to {levels.index[-1]:%d %b %Y}")
for name, label, _ in ASSETS:
print(f" {label:<18} from {levels[name].first_valid_index():%d %b %Y}")daily levels: 26,857 days 01 Jul 1926 to 17 Sep 2026
US Treasuries from 03 Jan 1962
Corporate bonds from 03 Jan 1986
Gold from 22 Mar 1968
Commodities from 02 Jan 1970
Developed ex-US from 31 Dec 1969
Emerging markets from 31 Dec 1987
To check my series, I compare them with Damodaran’s annual returns from NYU Stern, which cover the 10-year Treasury, Baa corporate bonds and gold from 1988. Both bond series and the corrected gold series have a correlation above 0.99 with Damodaran’s returns over the 38 years. For 1989, the corrected gold series gives −1.5% against Damodaran’s −2.8%, while the uncorrected LSEG series gives +96.9%.
# Damodaran / NYU Stern, January 2026 vintage: annual returns on the 10-year Treasury, on Baa
# corporate bonds and on gold. Public, and checked against here before trusting any of the
# series built above.
DAMODARAN_ANNUAL = {
1988: ( 0.0822, 0.1568, -0.1526), 1989: ( 0.1769, 0.1631, -0.0284),
1990: ( 0.0624, 0.0565, -0.0311), 1991: ( 0.1500, 0.1640, -0.0856),
1992: ( 0.0936, 0.1368, -0.0573), 1993: ( 0.1421, 0.1644, 0.1768),
1994: (-0.0804, -0.0123, -0.0217), 1995: ( 0.2348, 0.2009, 0.0098),
1996: ( 0.0143, 0.0528, -0.0459), 1997: ( 0.0994, 0.1130, -0.2141),
1998: ( 0.1492, 0.0810, -0.0083), 1999: (-0.0825, 0.0097, 0.0085),
2000: ( 0.1666, 0.0938, -0.0544), 2001: ( 0.0557, 0.0860, 0.0075),
2002: ( 0.1512, 0.1206, 0.2557), 2003: ( 0.0038, 0.1238, 0.1989),
2004: ( 0.0449, 0.1033, 0.0465), 2005: ( 0.0287, 0.0513, 0.1777),
2006: ( 0.0196, 0.0527, 0.2320), 2007: ( 0.1021, 0.0490, 0.3192),
2008: ( 0.2010, -0.0344, 0.0432), 2009: (-0.1112, 0.1996, 0.2504),
2010: ( 0.0846, 0.0940, 0.2924), 2011: ( 0.1604, 0.1226, 0.1202),
2012: ( 0.0297, 0.0940, 0.0568), 2013: (-0.0910, -0.0113, -0.2761),
2014: ( 0.1075, 0.1075, 0.0012), 2015: ( 0.0128, -0.0150, -0.1211),
2016: ( 0.0069, 0.1152, 0.0810), 2017: ( 0.0280, 0.0915, 0.1266),
2018: (-0.0002, -0.0318, -0.0093), 2019: ( 0.0964, 0.1525, 0.1908),
2020: ( 0.1133, 0.1060, 0.2417), 2021: (-0.0442, 0.0102, -0.0375),
2022: (-0.1783, -0.1523, 0.0055), 2023: ( 0.0388, 0.0874, 0.1326),
2024: (-0.0164, 0.0174, 0.2596), 2025: ( 0.0780, 0.0696, 0.6622),
}
published = pd.DataFrame(DAMODARAN_ANNUAL, index=["treasury", "corporate", "gold"]).T
rebuilt = levels[["treasury", "corporate", "gold"]].resample("YE").last().pct_change(fill_method=None)
rebuilt.index = rebuilt.index.year
years = published.index.intersection(rebuilt.index)
print(f"against Damodaran, annual returns December to December, {years[0]} to {years[-1]}:")
for name in ("treasury", "corporate", "gold"):
gap = (rebuilt.loc[years, name] - published.loc[years, name]).abs()
agreement = rebuilt.loc[years, name].corr(published.loc[years, name])
print(f" {ASSET_LABEL[name]:<16} mean gap {100 * gap.mean():.2f}pp, worst {100 * gap.max():.2f}pp "
f"({gap.idxmax()}), correlation {agreement:.3f}")
assert agreement > 0.95, f"{name} does not track Damodaran's series"
# 1989 is the year of the gold scale break, so it is the year the correction shows up in.
raw_1989 = raw_gold.resample("YE").last().pct_change(fill_method=None).loc["1989-12-31"]
print(f" Gold in 1989 {100 * rebuilt.loc[1989, 'gold']:+.1f}% corrected, "
f"{100 * raw_1989:+.1f}% as the feed arrives, {100 * published.loc[1989, 'gold']:+.1f}% Damodaran")
assert abs(rebuilt.loc[1989, "gold"] - published.loc[1989, "gold"]) < 0.03, "the gold correction is wrong"against Damodaran, annual returns December to December, 1988 to 2025:
US Treasuries mean gap 0.70pp, worst 3.97pp (1991), correlation 0.993
Corporate bonds mean gap 0.58pp, worst 3.50pp (1991), correlation 0.993
Gold mean gap 0.76pp, worst 1.97pp (2011), correlation 0.999
Gold in 1989 -1.5% corrected, +96.9% as the feed arrives, -2.8% Damodaran
Appendix B: two more checks
An annual check can miss month-to-month differences, so I also compare my bond series with the two bond funds on the five-year correlation with US stocks. My Treasury series and IEF differ by 0.019 on average and by 0.053 at most. My corporate series and LQD differ by 0.077 on average and by 0.368 at most. The funds report only prices, without the interest they pay, and they hold different bonds than my series, so the differences do not come from my series alone.
# A series can agree once a year and differ month by month, and it is the monthly path this post
# measures. I therefore also compare both bond series with a traded fund, on the quantity this
# post reports. The funds are price only, so they leave out distributions, and each one holds a
# different set of bonds from the model. They are a check, and an imperfect one.
TRADED_FUND = {"treasury": ("treasury_etf", "IEF"), "corporate": ("corporate_etf", "LQD")}
for name, (fund, fund_name) in TRADED_FUND.items():
both_reported = monthly_return.loc[:LAST_MONTH, ["stocks", name, fund]].dropna()
# put any missing month back as a gap, so a five-year window is always five calendar years
every_month = pd.period_range(both_reported.index[0], both_reported.index[-1], freq="M")
pair = both_reported.reindex(every_month)
rebuilt_path = pair[name].rolling(WINDOW_MONTHS).corr(pair["stocks"])
fund_path = pair[fund].rolling(WINDOW_MONTHS).corr(pair["stocks"])
gap = (rebuilt_path - fund_path).abs().dropna()
print(f"{ASSET_LABEL[name]} against {fund_name}, {len(both_reported)} months "
f"from {both_reported.index[0]}:")
print(f" the five-year correlation with US stocks differs by {gap.mean():.3f} on average and "
f"{gap.max():.3f} at worst ({gap.idxmax()})")
print(f" latest window {rebuilt_path.iloc[-1]:+.3f} rebuilt against {fund_path.iloc[-1]:+.3f} traded")US Treasuries against IEF, 288 months from 2002-08:
the five-year correlation with US stocks differs by 0.019 on average and 0.053 at worst (2014-02)
latest window +0.531 rebuilt against +0.535 traded
Corporate bonds against LQD, 272 months from 2002-09:
the five-year correlation with US stocks differs by 0.077 on average and 0.368 at worst (2014-01)
latest window +0.691 rebuilt against +0.711 traded
Finally, I check the closing-time explanation. I pair each asset class’s return on the next trading day with the US return. Then emerging markets have a correlation of +0.44 against +0.24 on the same day, and developed markets outside the US +0.44 against +0.39. The other four asset classes have a lower correlation one day later, so only the two foreign indices fit a closing-time gap.
# Many of the markets in the two foreign indices close hours before New York, so a day there and
# the same day in New York do not cover the same hours. Pairing each index with its own next
# session is one way to look at that: a higher correlation is consistent with a closing-time gap.
# The US return date sets the window, so at its edge a July return can pair with an August one.
print("daily returns over the same window, as traded and one session later:")
for name in ASSET_NAMES:
own = levels[name].dropna().pct_change()
trio = pd.concat({"stocks": daily_return["stocks"], "same": own, "next": own.shift(-1)},
axis=1).dropna()
in_window = (trio.index.to_period("M") >= FIRST_MONTH) & (trio.index.to_period("M") <= LATEST)
trio = trio[in_window]
as_traded = trio["stocks"].corr(trio["same"])
one_later = trio["stocks"].corr(trio["next"])
print(f" {ASSET_LABEL[name]:<18} {as_traded:+.2f} as traded, {one_later:+.2f} one session later "
f"({one_later - as_traded:+.2f}), {len(trio):,} days")daily returns over the same window, as traded and one session later:
US Treasuries +0.07 as traded, +0.05 one session later (-0.02), 1,246 days
Corporate bonds +0.16 as traded, +0.14 one session later (-0.02), 1,247 days
Gold +0.14 as traded, +0.05 one session later (-0.08), 1,255 days
Commodities +0.10 as traded, +0.06 one session later (-0.05), 1,255 days
Developed ex-US +0.39 as traded, +0.44 one session later (+0.05), 1,255 days
Emerging markets +0.24 as traded, +0.44 one session later (+0.20), 1,255 days
Read next
- Why the 60/40 portfolio is not dead. The same two questions on one pair of assets: what a five-year window does to a stock-bond curve, and what the return frequency does to it.
- How did diversified portfolios do when US stocks rose or fell? All Weather and two other diversified portfolios against the S&P 500, across the whole range of monthly returns.
- Butter prices can appear related to the S&P 500 What a high correlation between two series does and does not establish.
Disclaimer: an empirical study for discussion purposes, not investment advice. The design is descriptive and not causal. Past performance does not guarantee future returns.
Sources: Kenneth R. French, daily research factors, Tuck School of Business at Dartmouth College. Federal Reserve Bank of St Louis (FRED), series DGS10 for the 10-year Treasury yield and DBAA for Moody’s Baa corporate yield. Aswath Damodaran, NYU Stern, annual returns on the 10-year Treasury, Baa corporate bonds and gold, January 2026 vintage. Daily gold, S&P GSCI Total Return, MSCI EAFE gross and MSCI Emerging Markets gross index levels, and the IEF and LQD funds, from LSEG Workspace (licensed). I do not reproduce the raw data here.