Correlations with US stocks depend on which five years you measure

Analysis
Python
On monthly returns, US Treasuries had a correlation of −0.31 with US stocks over the five years to July 2013 and +0.53 over the five years to July 2026.
Published

September 19, 2026

Diversification comes from taking uncorrelated bets with positive expected returns. But the correlation itself varies over time, so depending on which five years you look at, you get different answers.

I take US stocks and six asset classes and show how the correlation of each asset class with US stocks changes over time. The six are US Treasuries, corporate bonds, gold, commodities, developed markets outside the US and emerging markets. I could have picked more or fewer asset classes, but I chose these six to keep it simple.

Over the five years to July 2013, the correlation with US stocks was +0.75 for commodities and −0.31 for US Treasuries. Over the five years to July 2026, it was +0.01 for commodities and +0.53 for US Treasuries, the opposite order.

The correlation also changes with how often you measure the return, monthly, weekly or daily.

The data

US stocks are Kenneth French’s daily market portfolio. Gold, commodities, developed markets outside the US and emerging markets come from LSEG Workspace. My LSEG account has no total-return bond index, so I build the US Treasury and corporate bond series from yields: the 10-year Treasury yield and Moody’s Baa corporate yield from FRED. The table gives the series for each asset class and what its return includes. Appendix A has the downloads, how I build the bond series and a first check, and Appendix B has two more checks.

Asset class Series What it includes
US stocks Kenneth French’s daily market portfolio dividends, US dollars
US Treasuries a 10-year par bond rebuilt from FRED’s DGS10 coupon and price change
Corporate bonds a modelled 10-year par bond, priced off Moody’s Baa yield, FRED’s DBAA coupon and price change
Gold LSEG XAU=, spot, US dollars an ounce price only, no income
Commodities S&P GSCI Total Return spot, roll and Treasury bill collateral
Developed ex-US MSCI EAFE gross, large and mid cap outside the US and Canada dividends reinvested, US dollars
Emerging markets MSCI Emerging Markets gross, monthly before 2000 dividends reinvested, US dollars

The correlation

For each series, I take the price or index level on the last trading day of each month and compute monthly returns. I drop any month whose last trading day falls more than a week before month-end, so a partial month never enters as a full month.

def period_level(daily_level, rule="M", stale_days=STALE_DAYS):
    """One level per calendar period, taken from the last day the series traded in that period.

    It drops a period when that last trading day is more than `stale_days` before the period
    ends, so a series that starts or stops midway never contributes a part-month return as though
    it were a whole month, and it drops a period the calendar has not finished yet.
    """
    traded = daily_level.dropna()
    period = traded.index.to_period(rule)
    last_level = traded.groupby(period).last()
    last_day = pd.Series(traded.index, index=period).groupby(level=0).last()
    period_end = last_level.index.to_timestamp(how="end").normalize()
    stale = (period_end - pd.DatetimeIndex(last_day.to_numpy())).days > stale_days
    unfinished = period_end >= TODAY
    # reindex onto every period between the first and the last, so a hole breaks the chain
    # and a two-month return never enters as a one-month return
    every_period = pd.period_range(last_level.index[0], last_level.index[-1], freq=rule)
    return last_level.where(~(stale | unfinished)).reindex(every_period)


monthly_level = pd.DataFrame({name: period_level(levels[name]) for name in levels.columns})
monthly_return = monthly_level.pct_change(fill_method=None)
print(f"monthly returns: {monthly_return.index[0]} to {monthly_return.index[-1]}")
print(f"  US stocks {int(monthly_return['stocks'].notna().sum()):,} months, "
      f"the last one {monthly_return['stocks'].last_valid_index()}")
monthly returns: 1926-07 to 2026-09
  US stocks 1,200 months, the last one 2026-07

I take a five-year window of 60 monthly returns and compute each asset class’s correlation with US stocks. Then I make it rolling, moving the window forward one month at a time and computing the correlation again. The first window ends in December 1992, because MSCI’s emerging-market index starts at the end of 1987, and the last ends in July 2026, so I get 404 windows.

def rolling_correlation(period_return, months=WINDOW_MONTHS):
    """Correlation with US stocks over the last `months` periods, one estimate per period end."""
    estimates = {}
    for name in ASSET_NAMES:
        pair = period_return[["stocks", name]].dropna()        # keep a month only if both reported
        pair = pair.reindex(period_return.index)               # put the gaps back where they were
        # rolling() counts the months it can see, so a window holding a gap gives no estimate
        estimates[name] = pair[name].rolling(months).corr(pair["stocks"])
    return pd.DataFrame(estimates)


correlation = rolling_correlation(monthly_return).dropna()     # start where all six have a window
correlation = correlation.loc[:LAST_MONTH]                     # and stop at the reported snapshot
assert correlation.abs().max().max() <= 1, "a correlation outside -1 to +1"
# every twelfth row is every twelfth calendar month only while no month in between is missing
unbroken = pd.period_range(correlation.index[0], correlation.index[-1], freq="M")
assert correlation.index.equals(unbroken), "a month is missing between the first and the last"
print(f"{len(correlation)} monthly estimates, each from {WINDOW_MONTHS} monthly returns, "
      f"ending {correlation.index[0]} to {correlation.index[-1]}")
404 monthly estimates, each from 60 monthly returns, ending 1992-12 to 2026-07

One window a year

Then I put the correlations on a video, so you can see how they change. The video rolls one year at a time, showing 34 of the 404 windows: the five years to July 1993, the five years to July 1994, and so on to the five years to July 2026. The numbers in this section come from these 34 windows.

def window_label(month):
    """The months one window covers: 2026-07 with a five-year window gives Aug 2021 to Jul 2026."""
    first_month = month - WINDOW_MONTHS + 1
    return f"{first_month.strftime('%b %Y')} – {month.strftime('%b %Y')}"


# counting back from the latest window, so the video always ends on the most recent one
anchor_positions = sorted(range(len(correlation) - 1, -1, -STEP_MONTHS))
anchors = correlation.iloc[anchor_positions]
assert anchors.notna().all().all(), "a window with no estimate would break the interpolation"

print(f"{len(anchors)} windows a year apart, {window_label(anchors.index[0])} "
      f"to {window_label(anchors.index[-1])}")
print(pd.DataFrame({
    "Lowest": {ASSET_LABEL[n]: f"{anchors[n].min():+.2f}" for n in ASSET_NAMES},
    "Lowest ends": {ASSET_LABEL[n]: anchors[n].idxmin().strftime("%b %Y") for n in ASSET_NAMES},
    "Highest": {ASSET_LABEL[n]: f"{anchors[n].max():+.2f}" for n in ASSET_NAMES},
    "Highest ends": {ASSET_LABEL[n]: anchors[n].idxmax().strftime("%b %Y") for n in ASSET_NAMES},
    "Latest": {ASSET_LABEL[n]: f"{anchors[n].iloc[-1]:+.2f}" for n in ASSET_NAMES}}).to_string())
print(f"lowest of all {len(correlation)} monthly windows: "
      f"Developed ex-US {correlation['developed'].min():+.2f}, "
      f"Emerging markets {correlation['emerging'].min():+.2f}")
34 windows a year apart, Aug 1988 – Jul 1993 to Aug 2021 – Jul 2026
                 Lowest Lowest ends Highest Highest ends Latest
US Treasuries     -0.56    Jul 2014   +0.53     Jul 2026  +0.53
Corporate bonds   -0.30    Jul 2015   +0.70     Jul 2024  +0.69
Gold              -0.35    Jul 1993   +0.22     Jul 2024  +0.17
Commodities       -0.40    Jul 1993   +0.75     Jul 2013  +0.01
Developed ex-US   +0.32    Jul 1997   +0.92     Jul 2012  +0.78
Emerging markets  +0.39    Jul 1997   +0.86     Jul 2012  +0.64
lowest of all 404 monthly windows: Developed ex-US +0.27, Emerging markets +0.36

Developed markets outside the US and emerging markets had a positive correlation with US stocks in all 34 windows. The correlation was between +0.32 and +0.92 for developed markets outside the US, and between +0.39 and +0.86 for emerging markets.

For US Treasuries and commodities, the order reversed. I compare the latest window, the five years to July 2026, with the five years to July 2013, the window in which commodities had their highest correlation with US stocks.

EARLIER = pd.Period("2013-07", "M")
LATEST = correlation.index[-1]
print(pd.DataFrame({
    window_label(EARLIER): {ASSET_LABEL[n]: f"{anchors.loc[EARLIER, n]:+.2f}" for n in ASSET_NAMES},
    window_label(LATEST): {ASSET_LABEL[n]: f"{anchors.loc[LATEST, n]:+.2f}" for n in ASSET_NAMES}}).to_string())
                 Aug 2008 – Jul 2013 Aug 2021 – Jul 2026
US Treasuries                  -0.31               +0.53
Corporate bonds                +0.26               +0.69
Gold                           +0.17               +0.17
Commodities                    +0.75               +0.01
Developed ex-US                +0.91               +0.78
Emerging markets               +0.86               +0.64

In the five years to July 2013, US Treasuries had the lower correlation with US stocks. In the five years to July 2026, commodities had the lower correlation. So, if you compare the two asset classes over only the latest five years, you do not see the opposite order from the five years to July 2013.

Monthly, weekly and daily returns

So far, I have used monthly correlations, and monthly returns absorb the daily variation I could also have measured. So I use the sample period from August 2021 to July 2026 and show how correlation varies across monthly, weekly, and daily returns.

def correlation_in_window(period_return, month_of_row, first_month, last_month):
    """Correlation with US stocks for every asset class, over one window of calendar months.

    month_of_row gives the calendar month each row belongs to, so a weekly or a daily panel can
    be cut by the same window as the monthly one. Each asset keeps every period it and US stocks
    both traded, so a market with its own holidays does not shorten the others.
    """
    inside = (month_of_row >= first_month) & (month_of_row <= last_month)
    window = period_return[inside]
    estimates = {}
    for name in ASSET_NAMES:
        pair = window[["stocks", name]].dropna()
        estimates[ASSET_LABEL[name]] = pair["stocks"].corr(pair[name])
    return pd.Series(estimates)


weekly_level = pd.DataFrame({name: period_level(levels[name], rule="W-FRI", stale_days=4)
                             for name in levels.columns})
weekly_return = weekly_level.pct_change(fill_method=None)
daily_return = pd.DataFrame({name: levels[name].dropna().pct_change() for name in levels.columns})

FIRST_MONTH = LATEST - WINDOW_MONTHS + 1
weekly_month = weekly_return.index.to_timestamp(how="end").to_period("M")
measured = pd.DataFrame({
    "Monthly": correlation_in_window(monthly_return, monthly_return.index, FIRST_MONTH, LATEST),
    "Weekly": correlation_in_window(weekly_return, weekly_month, FIRST_MONTH, LATEST),
    "Daily": correlation_in_window(daily_return, daily_return.index.to_period("M"), FIRST_MONTH, LATEST)})
print(f"{window_label(LATEST)}, the same five years measured three ways:")
print(measured.map(lambda value: f"{value:+.2f}").to_string())
Aug 2021 – Jul 2026, the same five years measured three ways:
                 Monthly Weekly  Daily
US Treasuries      +0.53  +0.12  +0.07
Corporate bonds    +0.69  +0.30  +0.16
Gold               +0.17  +0.16  +0.14
Commodities        +0.01  +0.11  +0.10
Developed ex-US    +0.78  +0.71  +0.39
Emerging markets   +0.64  +0.53  +0.24

For five of the six asset classes, the correlation decreases as the returns get more frequent. For US Treasuries, it was +0.53 on monthly returns, +0.12 on weekly returns and +0.07 on daily returns. For emerging markets, it was +0.64, +0.53 and +0.24, and for gold, +0.17, +0.16 and +0.14. Commodities are the exception, with +0.10 on daily returns against +0.01 on monthly ones.

So the correlation, or more correctly the estimated correlation, changes over time, but it also changes with the frequency you use. The higher the frequency, the more variation you include, but that extra variation can also be noise.

There is no right or wrong when it comes to which frequency you should use, because in the end, monthly, weekly and daily returns all measure the correlation. But when you estimate a correlation, you can get noise, and that noise can come from the sample period and also from the frequency. Daily returns, for example, can contain mechanical effects of different closing times. A foreign market closes before the US market, so its reaction to the same news can show up a day later in daily returns, while a monthly return puts both moves in the same month. Appendix B checks this closing-time explanation.

The video

FRAMES_PER_STEP = 36                             # frames between two estimates, 0.6s at 60 fps
FPS = 60
HOLD_END = 3.0                                   # seconds the last window stays on screen

# One shape-preserving curve through all of the yearly estimates, read off once per frame.
# PchipInterpolator passes exactly through every estimate and does not overshoot between
# neighbouring ones. It interpolates the values in between for the animation, and those are not
# correlation estimates. Easing each pair of years separately would stop every dot at every one.
curve = PchipInterpolator(np.arange(len(anchors)), anchors.to_numpy(), axis=0)
position = np.arange((len(anchors) - 1) * FRAMES_PER_STEP + 1) / FRAMES_PER_STEP   # counted in estimates
path = curve(position)

on_estimate = np.flatnonzero(np.isclose(position, np.round(position)))   # frames on an estimate
assert np.allclose(path[on_estimate], anchors.to_numpy()), "the curve misses an estimate"
for column, name in enumerate(ASSET_NAMES):
    lowest, highest = anchors[name].min(), anchors[name].max()
    assert path[:, column].min() >= lowest - 1e-9, f"{name}: the curve goes below every estimate"
    assert path[:, column].max() <= highest + 1e-9, f"{name}: the curve goes above every estimate"

named_window = np.round(position).astype(int)    # the window whose months the corner shows
frames = [(path[row], named_window[row]) for row in range(len(position))]
frames = frames + [frames[-1]] * round(HOLD_END * FPS)

# While the corner holds one window, the number beside a dot is already travelling towards the
# next year's estimate. Only the frame on an estimate is the estimate for that window.
drift = np.abs(path - anchors.to_numpy()[named_window])
print(f"{len(frames)} frames, {len(frames) / FPS:.1f} seconds, "
      f"{window_label(anchors.index[0])} to {window_label(anchors.index[-1])}")
print("between one estimate and the next the number beside a dot differs")
print(f"  from the window named by {drift.mean():.3f} on average and {drift.max():.3f} at most")
1369 frames, 22.8 seconds, Aug 1988 – Jul 1993 to Aug 2021 – Jul 2026
between one estimate and the next the number beside a dot differs
  from the window named by 0.017 on average and 0.249 at most
import cv2                                       # pip install --no-deps opencv-python

VIDEO_WIDTH, VIDEO_HEIGHT, DPI = 1440, 960, 120
RECT = [0.088, 0.105, 0.875, 0.775]              # the plot, with room above it for the dates
COLUMN = np.arange(len(ASSETS))                  # each asset keeps its own column, whatever it does

figure = plt.figure(figsize=(VIDEO_WIDTH / DPI, VIDEO_HEIGHT / DPI), dpi=DPI)


def draw_frame(values, label):
    """One video frame as a BGR array: one dot per asset class, the window's months in the corner.

    Redrawing the whole figure each frame is slower than updating the dots in place, and it keeps
    the code readable, which matters more here.
    """
    figure.clf()
    figure.patch.set_facecolor(BG)
    ax = figure.add_axes(RECT)
    ax.set_facecolor(BG)
    ax.set_xlim(-0.65, len(ASSETS) - 0.35)
    ax.set_ylim(-1, 1)                           # a correlation cannot leave this range
    ax.set_yticks([-1, -0.5, 0, 0.5, 1])
    ax.yaxis.set_major_formatter(mticker.FuncFormatter(lambda v, _: "0" if v == 0 else f"{v:+.1f}"))
    ax.set_xticks(COLUMN)
    ax.set_xticklabels([label for _, label, _ in ASSETS])
    ax.grid(True, axis="y", color=GRID, lw=1.1)
    ax.set_axisbelow(True)
    for side in ("top", "right"):
        ax.spines[side].set_visible(False)
    for side in ("left", "bottom"):
        ax.spines[side].set_color(SPINE)
    ax.tick_params(axis="both", length=0, labelsize=13, colors=INK, pad=10)
    ax.axhline(0, color=INK, lw=1.8, zorder=3)   # the line the dots move around
    ax.set_ylabel("Correlation with US stocks", fontsize=17, color=AXIS_TEXT, labelpad=14)

    for column, (_, _, colour), value in zip(COLUMN, ASSETS, values):
        ax.plot(column, value, "o", ms=16, color=colour, zorder=4, clip_on=False)
        ax.annotate(f"{value:+.2f}", (column, value), xytext=(19, 0), textcoords="offset points",
                    ha="left", va="center", fontsize=18, fontweight="semibold", color=colour,
                    annotation_clip=False, zorder=5,
                    # the patch keeps the number legible when a dot is on the zero line
                    bbox=dict(facecolor=BG, edgecolor="none", pad=1.5))

    figure.text(0.966, 0.945, label, ha="right", va="center", fontsize=25, color=INK)
    figure.canvas.draw()
    return cv2.cvtColor(np.asarray(figure.canvas.buffer_rgba()), cv2.COLOR_RGBA2BGR)
# Media Foundation is Windows only, and its H.264 output is what a browser can decode. On another
# platform pass cv2.CAP_FFMPEG here instead, which needs an H.264 encoder available to FFmpeg. On
# this machine that would be OpenH264, and it is not installed.
writer = cv2.VideoWriter(RAW_MP4, cv2.CAP_MSMF, cv2.VideoWriter_fourcc(*"H264"),
                         FPS, (VIDEO_WIDTH, VIDEO_HEIGHT))
# a writer that failed to open discards every frame in silence, so check it rather than trust it
assert writer.isOpened(), "OpenCV could not open the video writer"

started = time.time()
for number, (values, window) in enumerate(frames):
    writer.write(draw_frame(values, window_label(anchors.index[window])))
    if (number + 1) % 300 == 0:
        print(f"  {number + 1} of {len(frames)} frames ({time.time() - started:.0f}s)", flush=True)
writer.release()
plt.close(figure)
print(f"rendered {len(frames)} frames in {(time.time() - started) / 60:.1f} min")
  300 of 1369 frames (29s)
  600 of 1369 frames (58s)
  900 of 1369 frames (80s)
  1200 of 1369 frames (105s)
rendered 1369 frames in 2.0 min
def mp4_boxes(buffer, start=0, end=None):
    """Every box in one stretch of an MP4 file: its type, where it starts, its size, its header."""
    end = len(buffer) if end is None else end
    found, position = [], start
    while position + 8 <= end:
        size, kind = struct.unpack(">I4s", buffer[position:position + 8])
        header = 8
        if size == 1:                            # a box above 4 GB states its size in 64 bits
            size, header = struct.unpack(">Q", buffer[position + 8:position + 16])[0], 16
        elif size == 0:                          # a size of zero means the box runs to the end
            size = end - position
        found.append((kind.decode("latin-1"), position, size, header))
        if size < header:
            break
        position += size
    return found


def move_index_to_front(source, target):
    """Write the file again with its index ahead of the frames, so playback can start early.

    An MP4 keeps the frames in one box (mdat) and the index of where each chunk of them starts in
    another (moov). OpenCV writes mdat first, and an index at the end can delay the start of
    playback. Moving moov in front pushes every chunk that much further into the file, so each
    offset it holds gains moov's own size.
    """
    buffer = bytearray(open(source, "rb").read())
    top_level = mp4_boxes(buffer)
    order = [kind for kind, _, _, _ in top_level]
    assert "moov" in order and "mdat" in order, order
    _, moov_at, moov_size, moov_header = top_level[order.index("moov")]
    index = bytearray(buffer[moov_at:moov_at + moov_size])

    def add_to_offsets(box, start, end):
        for kind, position, size, header in mp4_boxes(box, start, end):
            if kind in ("trak", "mdia", "minf", "stbl", "edts"):
                add_to_offsets(box, position + header, position + size)   # these hold other boxes
            elif kind in ("stco", "co64"):       # the chunk offsets, 32 bit and 64 bit versions
                count = struct.unpack(">I", box[position + header + 4:position + header + 8])[0]
                pattern, width = (">I", 4) if kind == "stco" else (">Q", 8)
                at = position + header + 8
                for _ in range(count):
                    moved = struct.unpack(pattern, box[at:at + width])[0] + moov_size
                    box[at:at + width] = struct.pack(pattern, moved)
                    at += width

    add_to_offsets(index, moov_header, moov_size)
    mdat_at = top_level[order.index("mdat")][1]
    rebuilt = buffer[:mdat_at] + index + buffer[mdat_at:moov_at] + buffer[moov_at + moov_size:]
    open(target, "wb").write(rebuilt)
    return [kind for kind, _, _, _ in mp4_boxes(rebuilt)]


print(f"boxes in order: {move_index_to_front(RAW_MP4, OUT_MP4)}")
os.remove(RAW_MP4)
print(f"{os.path.basename(OUT_MP4)}: {VIDEO_WIDTH}x{VIDEO_HEIGHT} at {FPS} fps, {len(frames)} frames, "
      f"{len(frames) / FPS:.1f}s, {os.path.getsize(OUT_MP4) / 1e6:.1f} MB")
boxes in order: ['ftyp', 'uuid', 'moov', 'mdat']
correlation.mp4: 1440x960 at 60 fps, 1369 frames, 22.8s, 5.6 MB

Each dot is one asset class’s correlation with US stocks over the five years named in the corner, and each asset class keeps its own column. Between two yearly windows, the video interpolates the dots and the numbers beside them, so only the frames on a yearly window show a calculated correlation. So, the order of the dots changes from one five-year window to the next.

Limitations

I focus only on correlation to connect it to diversification. But volatility also matters, because a portfolio’s volatility depends on the volatility of each asset, separately from the correlation.

I derive the prices of both bond series from their yields because the data I have access to contains only yields.

Conclusion

Diversification, when you look only at correlation, changes over time, and depending on which five years you look at, you will get different answers. It also changes depending on how often you measure returns. So it is better to be careful about concluding too fast by looking only at the correlation over the past five years.

Appendix A: downloading and building the data

I set the window length, video speed, and the six asset classes in one block, so I only need to change them in one place.

import io
import os
import struct
import time
import urllib.request
import warnings
import zipfile

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
from scipy.interpolate import PchipInterpolator   # the curve the dots follow between two years

warnings.filterwarnings("ignore")
plt.rcParams["font.family"] = ["Segoe UI", "Arial", "DejaVu Sans"]   # the video's typography

DATA_DIR = os.path.expanduser("~")               # anywhere you can write
STOCK_CSV = os.path.join(DATA_DIR, "us_stock_daily.csv")
TREASURY_YIELD_CSV = os.path.join(DATA_DIR, "treasury_yield_daily.csv")
CORPORATE_YIELD_CSV = os.path.join(DATA_DIR, "corporate_yield_daily.csv")
LSEG_CSV = os.path.join(DATA_DIR, "lseg_levels_daily.csv")
RAW_MP4 = os.path.join(DATA_DIR, "correlation_raw.mp4")   # the encoder writes here first
OUT_MP4 = os.path.join(DATA_DIR, "correlation.mp4")

START_DATE = "1960-01-01"                        # ask early, each source clips to what it has
END_DATE = str(pd.Timestamp.now().date())
TODAY = pd.Timestamp.now().normalize()

WINDOW_MONTHS = 60                               # five years of monthly returns in every estimate
LAST_MONTH = pd.Period("2026-07", "M")           # the snapshot reported here, so a later download
                                                 # cannot move the windows the prose quotes
STEP_MONTHS = 12                                 # the video passes through one estimate a year
STALE_DAYS = 7                                   # how old a month's last trade may be before it is dropped
BOND_MATURITY = 10.0                             # constant maturity, in years

# One row per asset class: the name used in the code, the label on the axis, and its colour.
ASSETS = [("treasury", "US Treasuries", "#17868A"),
          ("corporate", "Corporate bonds", "#6B8E23"),
          ("gold", "Gold", "#D2822B"),
          ("commodities", "Commodities", "#C0392B"),
          ("developed", "Developed ex-US", "#2B6CB0"),
          ("emerging", "Emerging markets", "#7C5CBF")]
ASSET_NAMES = [name for name, _, _ in ASSETS]
ASSET_LABEL = {name: label for name, label, _ in ASSETS}

BG, INK, GRID, SPINE, AXIS_TEXT = "#FCEFE3", "#1f1f1f", "#EADCCC", "#D5C6B4", "#4a4a4a"

I download US stocks from Kenneth French’s website and the two yields from FRED. Moody’s holds the copyright on the Baa yield, so I do not republish it.

FRENCH_ZIP = ("https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/"
              "F-F_Research_Data_Factors_daily_CSV.zip")
FRED_CSV = "https://fred.stlouisfed.org/graph/fredgraph.csv?id={}&cosd=1900-01-01"


def download(url, tries=4):
    """Fetch one file, retrying with a growing pause, then raising so a failure is never silent."""
    for attempt in range(tries):
        try:
            request = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (research script)"})
            with urllib.request.urlopen(request, timeout=120) as response:
                return response.read()
        except Exception as error:
            if attempt == tries - 1:
                raise
            wait = 15 * (attempt + 1)
            print(f"  download failed ({type(error).__name__}), retrying in {wait}s")
            time.sleep(wait)


def us_stock_level():
    """Daily wealth index of the US market portfolio, dividends included, starting at 1.

    Kenneth French publishes the market return in excess of the risk-free rate (Mkt-RF) and the
    risk-free rate (RF) as separate columns, both in percent, so this function adds the two back
    together to get the return on the market itself.
    """
    archive = zipfile.ZipFile(io.BytesIO(download(FRENCH_ZIP)))
    text = archive.read(archive.namelist()[0]).decode("latin-1").splitlines()
    header = 0                                   # the file opens with a paragraph of notes
    for row, line in enumerate(text):
        if "Mkt-RF" in [cell.strip() for cell in line.split(",")]:
            header = row
            break
    # an annual table follows the daily one, and only the daily rows start with a date
    dated = [line for line in text[header + 1:] if line.strip()[:8].isdigit()]
    factors = pd.read_csv(io.StringIO("\n".join(dated)), header=None,
                          names=["date"] + [cell.strip() for cell in text[header].split(",")[1:]])
    factors.index = pd.to_datetime(factors.pop("date").astype(str), format="%Y%m%d")
    market_return = (factors["Mkt-RF"] + factors["RF"]) / 100
    return (1 + market_return).cumprod()


def save_fred_yield(series_id, path):
    """Save one daily yield series from FRED, in percent as it is published."""
    text = download(FRED_CSV.format(series_id)).decode("utf-8").splitlines()
    dated = [line for line in text if not line.endswith(".")]   # FRED prints a missing day as a dot
    table = pd.read_csv(io.StringIO("\n".join(dated)))
    yields = pd.to_numeric(table.iloc[:, 1], errors="coerce")
    yields.index = pd.to_datetime(table.iloc[:, 0])
    yields.dropna().sort_index().rename("yield").rename_axis("Date").to_csv(path, encoding="utf-8-sig")


if not os.path.exists(STOCK_CSV):
    us_stock_level().rename("level").rename_axis("Date").to_csv(STOCK_CSV, encoding="utf-8-sig")
if not os.path.exists(TREASURY_YIELD_CSV):
    save_fred_yield("DGS10", TREASURY_YIELD_CSV)     # 10-year constant-maturity Treasury yield
if not os.path.exists(CORPORATE_YIELD_CSV):
    save_fred_yield("DBAA", CORPORATE_YIELD_CSV)     # Moody's seasoned Baa corporate bond yield

stock_level = pd.read_csv(STOCK_CSV, parse_dates=["Date"]).set_index("Date")["level"]
treasury_yield = pd.read_csv(TREASURY_YIELD_CSV, parse_dates=["Date"]).set_index("Date")["yield"] / 100
corporate_yield = pd.read_csv(CORPORATE_YIELD_CSV, parse_dates=["Date"]).set_index("Date")["yield"] / 100
print(f"US stocks      {len(stock_level):,} days  {stock_level.index[0]:%d %b %Y} to {stock_level.index[-1]:%d %b %Y}")
print(f"10-year yield  {len(treasury_yield):,} days  {treasury_yield.index[0]:%d %b %Y} to {treasury_yield.index[-1]:%d %b %Y}")
print(f"Baa yield      {len(corporate_yield):,} days  {corporate_yield.index[0]:%d %b %Y} to {corporate_yield.index[-1]:%d %b %Y}")
US stocks      26,296 days  01 Jul 1926 to 31 Jul 2026
10-year yield  16,161 days  02 Jan 1962 to 15 Sep 2026
Baa yield      10,214 days  02 Jan 1986 to 17 Sep 2026

Then I download gold, commodities, developed markets outside the US, and emerging markets from LSEG, along with two bond funds, IEF and LQD, which I use to check my bond series. The LSEG data are licensed, so the block prints only the number of days and the start date for each series.

In this extract, LSEG quotes half the gold price before 16 October 1989, so I double every gold price before that day.

LSEG_SERIES = {"gold": ("XAU=", "MID_PRICE"),                  # gold spot, US dollars an ounce
               "commodities": (".SPGSCITR", "TR.PriceClose"),  # S&P GSCI, spot, roll and collateral
               "developed": (".MIEA00000GUS", "TRDPRC_1"),     # MSCI EAFE, dividends reinvested
               "emerging": (".MIEF00000GUS", "TRDPRC_1"),      # MSCI Emerging Markets, dividends reinvested
               "treasury_etf": ("IEF.O", "TRDPRC_1"),          # iShares 7-10 year Treasury, from 2002
               "corporate_etf": ("LQD.P", "TRDPRC_1")}         # iShares investment grade corporates, from 2002

# The largest daily step to accept from a feed without looking at it. A bigger step is a reason to
# inspect the series before trusting it. MSCI publishes the gross emerging index monthly
# before 2000, so one of its steps can cover a whole month.
LARGEST_STEP = {"emerging": 0.45}
GOLD_SCALE_BREAK = pd.Timestamp("1989-10-16")    # the day XAU= starts quoting the whole gold price


def patch_httpx_proxy():
    """lseg-data 2.1.1 passes proxy={} into httpx, and httpx 0.28 and later reject a dict."""
    import httpx

    def coerce(kwargs):
        proxy = kwargs.get("proxy")
        if isinstance(proxy, dict):              # {} or {"http": None} both mean no proxy at all
            kwargs["proxy"] = next((value for value in proxy.values() if value), None)
        return kwargs

    for client_class in (httpx.Client, httpx.AsyncClient):
        if getattr(client_class.__init__, "_patched", False):
            continue                             # idempotent, so re-running this block is harmless
        original = client_class.__init__

        def make(original):
            def __init__(self, *args, **kwargs):
                return original(self, *args, **coerce(kwargs))
            __init__._patched = True
            return __init__

        client_class.__init__ = make(original)


def download_lseg(ric, field, tries=6):
    """Download one daily series from LSEG, retrying on a rate limit.

    Anything other than a rate limit stops the download, so a real error surfaces instead of
    being retried into silence.
    """
    for attempt in range(tries):
        try:
            history = ld.get_history(universe=ric, fields=[field],
                                     start=START_DATE, end=END_DATE, interval="daily")
            return history.iloc[:, 0].dropna().sort_index()
        except Exception as error:
            rate_limited = ("Too many requests" in str(error)) or ("429" in str(error))
            if not rate_limited or attempt == tries - 1:
                raise
            wait = 30 * (2 ** attempt)           # 30s, 60s, 120s, 240s, 480s
            print(f"  rate-limited, waiting {wait}s", flush=True)
            time.sleep(wait)


if not os.path.exists(LSEG_CSV):
    patch_httpx_proxy()
    import lseg.data as ld                       # needs Workspace running and logged in on this machine
    ld.open_session()                            # no try block: without a session this has to stop
    feeds = {name: download_lseg(ric, field) for name, (ric, field) in LSEG_SERIES.items()}
    ld.close_session()
    pd.concat(feeds, axis=1).sort_index().rename_axis("Date").to_csv(LSEG_CSV, encoding="utf-8-sig")

lseg = pd.read_csv(LSEG_CSV, parse_dates=["Date"]).set_index("Date").sort_index()

# In this extract XAU= quotes half the gold price before 16 October 1989, so the block doubles
# everything before that day to put the whole series on one scale. It measures the step on the days
# gold itself traded, because the frame also holds rows from the other five feeds. It keeps raw_gold
# because the check block reports what the correction is worth.
raw_gold = lseg["gold"].copy()
traded_gold = lseg["gold"].dropna()
step = (traded_gold / traded_gold.shift(1)).dropna()
doubling = step[(step > 1.6) & (step < 2.4)]
assert list(doubling.index) == [GOLD_SCALE_BREAK], f"doublings found: {list(doubling.index.date)}"
lseg.loc[lseg.index < GOLD_SCALE_BREAK, "gold"] *= 2

for name in lseg.columns:
    biggest_step = lseg[name].dropna().pct_change().abs().max()
    limit = LARGEST_STEP.get(name, 0.25)
    assert biggest_step < limit, f"a step of {biggest_step:.0%} in {name}: check for a change of scale"
    series = lseg[name].dropna()
    print(f"  {name:<14} {LSEG_SERIES[name][0]:<16} {len(series):6,} days from {series.index[0]:%d %b %Y}")
print(f"gold: scale corrected before {doubling.index[0]:%d %b %Y}")
  gold           XAU=             14,747 days from 22 Mar 1968
  commodities    .SPGSCITR        14,296 days from 02 Jan 1970
  developed      .MIEA00000GUS    14,797 days from 31 Dec 1969
  emerging       .MIEF00000GUS     6,866 days from 31 Dec 1987
  treasury_etf   IEF.O             6,075 days from 26 Jul 2002
  corporate_etf  LQD.P             5,743 days from 30 Aug 2002
gold: scale corrected before 16 Oct 1989

For the bonds, I have to build the returns myself from the yields. At each date, I reprice the bond from the previous date at the new yield, add the interest it earned, and then buy a new ten-year bond, so the series always holds a ten-year bond. For the corporate bonds, I use Moody’s Baa yield as if it were the yield of a ten-year bond, although it is an average over bonds with twenty years or more left.

def par_price(yield_now, coupon_rate, maturity):
    """Price per 100 of a bond with semiannual coupons, given an annual yield and coupon rate."""
    yield_now = np.maximum(yield_now, 1e-8)      # a zero yield divides by zero in the annuity term
    discount = (1 + yield_now / 2) ** (-2 * maturity)
    return 100 * ((coupon_rate / yield_now) * (1 - discount) + discount)


def total_return_from_yield(bond_yield, maturity=BOND_MATURITY):
    """Total return of a modelled bond held at a constant maturity, rebuilt from its yield.

    At each observation this function reprices the bond at the new yield with the elapsed calendar
    time taken off its maturity, then adds the coupon accrued over that time. It sets the coupon
    rate to the previous observation's yield and treats that as the par yield for `maturity`, so
    the bond keeps a ten-year maturity instead of ageing like one particular note. Where the input is not a yield on
    a bond of that maturity, as with Moody's Baa average, the result is a model and not an index.
    """
    # observations can be days apart over a weekend, so the step is the elapsed calendar time.
    # clip(lower=1) stops a zero-day gap, which would make the accrued coupon zero
    year_fraction = bond_yield.index.to_series().diff().dt.days.clip(lower=1) / 365.0
    coupon = bond_yield.shift(1)                 # the previous observation's yield, used as a par rate
    price_change = (par_price(bond_yield.to_numpy(), coupon.to_numpy(),
                              maturity - year_fraction.to_numpy()) - 100) / 100
    accrued = coupon.to_numpy() * year_fraction.to_numpy()
    return pd.Series(price_change + accrued, index=bond_yield.index).dropna()


treasury_level = (1 + total_return_from_yield(treasury_yield)).cumprod()
corporate_level = (1 + total_return_from_yield(corporate_yield)).cumprod()

levels = pd.concat([stock_level.rename("stocks"), treasury_level.rename("treasury"),
                    corporate_level.rename("corporate"), lseg], axis=1).sort_index()
print(f"daily levels: {len(levels):,} days  {levels.index[0]:%d %b %Y} to {levels.index[-1]:%d %b %Y}")
for name, label, _ in ASSETS:
    print(f"  {label:<18} from {levels[name].first_valid_index():%d %b %Y}")
daily levels: 26,857 days  01 Jul 1926 to 17 Sep 2026
  US Treasuries      from 03 Jan 1962
  Corporate bonds    from 03 Jan 1986
  Gold               from 22 Mar 1968
  Commodities        from 02 Jan 1970
  Developed ex-US    from 31 Dec 1969
  Emerging markets   from 31 Dec 1987

To check my series, I compare them with Damodaran’s annual returns from NYU Stern, which cover the 10-year Treasury, Baa corporate bonds and gold from 1988. Both bond series and the corrected gold series have a correlation above 0.99 with Damodaran’s returns over the 38 years. For 1989, the corrected gold series gives −1.5% against Damodaran’s −2.8%, while the uncorrected LSEG series gives +96.9%.

# Damodaran / NYU Stern, January 2026 vintage: annual returns on the 10-year Treasury, on Baa
# corporate bonds and on gold. Public, and checked against here before trusting any of the
# series built above.
DAMODARAN_ANNUAL = {
    1988: ( 0.0822,  0.1568, -0.1526), 1989: ( 0.1769,  0.1631, -0.0284),
    1990: ( 0.0624,  0.0565, -0.0311), 1991: ( 0.1500,  0.1640, -0.0856),
    1992: ( 0.0936,  0.1368, -0.0573), 1993: ( 0.1421,  0.1644,  0.1768),
    1994: (-0.0804, -0.0123, -0.0217), 1995: ( 0.2348,  0.2009,  0.0098),
    1996: ( 0.0143,  0.0528, -0.0459), 1997: ( 0.0994,  0.1130, -0.2141),
    1998: ( 0.1492,  0.0810, -0.0083), 1999: (-0.0825,  0.0097,  0.0085),
    2000: ( 0.1666,  0.0938, -0.0544), 2001: ( 0.0557,  0.0860,  0.0075),
    2002: ( 0.1512,  0.1206,  0.2557), 2003: ( 0.0038,  0.1238,  0.1989),
    2004: ( 0.0449,  0.1033,  0.0465), 2005: ( 0.0287,  0.0513,  0.1777),
    2006: ( 0.0196,  0.0527,  0.2320), 2007: ( 0.1021,  0.0490,  0.3192),
    2008: ( 0.2010, -0.0344,  0.0432), 2009: (-0.1112,  0.1996,  0.2504),
    2010: ( 0.0846,  0.0940,  0.2924), 2011: ( 0.1604,  0.1226,  0.1202),
    2012: ( 0.0297,  0.0940,  0.0568), 2013: (-0.0910, -0.0113, -0.2761),
    2014: ( 0.1075,  0.1075,  0.0012), 2015: ( 0.0128, -0.0150, -0.1211),
    2016: ( 0.0069,  0.1152,  0.0810), 2017: ( 0.0280,  0.0915,  0.1266),
    2018: (-0.0002, -0.0318, -0.0093), 2019: ( 0.0964,  0.1525,  0.1908),
    2020: ( 0.1133,  0.1060,  0.2417), 2021: (-0.0442,  0.0102, -0.0375),
    2022: (-0.1783, -0.1523,  0.0055), 2023: ( 0.0388,  0.0874,  0.1326),
    2024: (-0.0164,  0.0174,  0.2596), 2025: ( 0.0780,  0.0696,  0.6622),
}
published = pd.DataFrame(DAMODARAN_ANNUAL, index=["treasury", "corporate", "gold"]).T
rebuilt = levels[["treasury", "corporate", "gold"]].resample("YE").last().pct_change(fill_method=None)
rebuilt.index = rebuilt.index.year
years = published.index.intersection(rebuilt.index)

print(f"against Damodaran, annual returns December to December, {years[0]} to {years[-1]}:")
for name in ("treasury", "corporate", "gold"):
    gap = (rebuilt.loc[years, name] - published.loc[years, name]).abs()
    agreement = rebuilt.loc[years, name].corr(published.loc[years, name])
    print(f"  {ASSET_LABEL[name]:<16} mean gap {100 * gap.mean():.2f}pp, worst {100 * gap.max():.2f}pp "
          f"({gap.idxmax()}), correlation {agreement:.3f}")
    assert agreement > 0.95, f"{name} does not track Damodaran's series"

# 1989 is the year of the gold scale break, so it is the year the correction shows up in.
raw_1989 = raw_gold.resample("YE").last().pct_change(fill_method=None).loc["1989-12-31"]
print(f"  Gold in 1989     {100 * rebuilt.loc[1989, 'gold']:+.1f}% corrected, "
      f"{100 * raw_1989:+.1f}% as the feed arrives, {100 * published.loc[1989, 'gold']:+.1f}% Damodaran")
assert abs(rebuilt.loc[1989, "gold"] - published.loc[1989, "gold"]) < 0.03, "the gold correction is wrong"
against Damodaran, annual returns December to December, 1988 to 2025:
  US Treasuries    mean gap 0.70pp, worst 3.97pp (1991), correlation 0.993
  Corporate bonds  mean gap 0.58pp, worst 3.50pp (1991), correlation 0.993
  Gold             mean gap 0.76pp, worst 1.97pp (2011), correlation 0.999
  Gold in 1989     -1.5% corrected, +96.9% as the feed arrives, -2.8% Damodaran

Appendix B: two more checks

An annual check can miss month-to-month differences, so I also compare my bond series with the two bond funds on the five-year correlation with US stocks. My Treasury series and IEF differ by 0.019 on average and by 0.053 at most. My corporate series and LQD differ by 0.077 on average and by 0.368 at most. The funds report only prices, without the interest they pay, and they hold different bonds than my series, so the differences do not come from my series alone.

# A series can agree once a year and differ month by month, and it is the monthly path this post
# measures. I therefore also compare both bond series with a traded fund, on the quantity this
# post reports. The funds are price only, so they leave out distributions, and each one holds a
# different set of bonds from the model. They are a check, and an imperfect one.
TRADED_FUND = {"treasury": ("treasury_etf", "IEF"), "corporate": ("corporate_etf", "LQD")}
for name, (fund, fund_name) in TRADED_FUND.items():
    both_reported = monthly_return.loc[:LAST_MONTH, ["stocks", name, fund]].dropna()
    # put any missing month back as a gap, so a five-year window is always five calendar years
    every_month = pd.period_range(both_reported.index[0], both_reported.index[-1], freq="M")
    pair = both_reported.reindex(every_month)
    rebuilt_path = pair[name].rolling(WINDOW_MONTHS).corr(pair["stocks"])
    fund_path = pair[fund].rolling(WINDOW_MONTHS).corr(pair["stocks"])
    gap = (rebuilt_path - fund_path).abs().dropna()
    print(f"{ASSET_LABEL[name]} against {fund_name}, {len(both_reported)} months "
          f"from {both_reported.index[0]}:")
    print(f"  the five-year correlation with US stocks differs by {gap.mean():.3f} on average and "
          f"{gap.max():.3f} at worst ({gap.idxmax()})")
    print(f"  latest window {rebuilt_path.iloc[-1]:+.3f} rebuilt against {fund_path.iloc[-1]:+.3f} traded")
US Treasuries against IEF, 288 months from 2002-08:
  the five-year correlation with US stocks differs by 0.019 on average and 0.053 at worst (2014-02)
  latest window +0.531 rebuilt against +0.535 traded
Corporate bonds against LQD, 272 months from 2002-09:
  the five-year correlation with US stocks differs by 0.077 on average and 0.368 at worst (2014-01)
  latest window +0.691 rebuilt against +0.711 traded

Finally, I check the closing-time explanation. I pair each asset class’s return on the next trading day with the US return. Then emerging markets have a correlation of +0.44 against +0.24 on the same day, and developed markets outside the US +0.44 against +0.39. The other four asset classes have a lower correlation one day later, so only the two foreign indices fit a closing-time gap.

# Many of the markets in the two foreign indices close hours before New York, so a day there and
# the same day in New York do not cover the same hours. Pairing each index with its own next
# session is one way to look at that: a higher correlation is consistent with a closing-time gap.
# The US return date sets the window, so at its edge a July return can pair with an August one.
print("daily returns over the same window, as traded and one session later:")
for name in ASSET_NAMES:
    own = levels[name].dropna().pct_change()
    trio = pd.concat({"stocks": daily_return["stocks"], "same": own, "next": own.shift(-1)},
                     axis=1).dropna()
    in_window = (trio.index.to_period("M") >= FIRST_MONTH) & (trio.index.to_period("M") <= LATEST)
    trio = trio[in_window]
    as_traded = trio["stocks"].corr(trio["same"])
    one_later = trio["stocks"].corr(trio["next"])
    print(f"  {ASSET_LABEL[name]:<18} {as_traded:+.2f} as traded, {one_later:+.2f} one session later "
          f"({one_later - as_traded:+.2f}), {len(trio):,} days")
daily returns over the same window, as traded and one session later:
  US Treasuries      +0.07 as traded, +0.05 one session later (-0.02), 1,246 days
  Corporate bonds    +0.16 as traded, +0.14 one session later (-0.02), 1,247 days
  Gold               +0.14 as traded, +0.05 one session later (-0.08), 1,255 days
  Commodities        +0.10 as traded, +0.06 one session later (-0.05), 1,255 days
  Developed ex-US    +0.39 as traded, +0.44 one session later (+0.05), 1,255 days
  Emerging markets   +0.24 as traded, +0.44 one session later (+0.20), 1,255 days