Does the price of butter predict the S&P 500? The reason you need cointegration.

Two unrelated series can appear highly correlated, even when there is no real relationship between them.

Computing the correlation between butter prices and the stock market may give us the impression they are related. They are not. Two unrelated series that trend upward can still appear highly correlated.

In this post, I build two series with no relation at all, show how correlation and a naive regression report a strong relationship, and then use cointegration to show there is none. Cointegration is another way of saying that two time series may each wander while the gap between them stays near a stable average.

Butter and the S&P 500

Both series are simulated. I build a “butter price” and an “S&P 500”: each drifts upward with random month-to-month moves, drawn independently so they share nothing. Simulating is the point, because it lets me know the truth up front: the two are unrelated, so any relationship the tests report is spurious.

%matplotlib inline
import numpy as np, pandas as pd, matplotlib.pyplot as plt
SEED, N = 6, 600                                          # 600 months, one fixed draw so the post reproduces
rng = np.random.default_rng(SEED)
butter = 50  + np.cumsum(0.10 + rng.normal(0, 1.0, N))    # butter price: small upward drift, its own noise
spx    = 100 + np.cumsum(0.30 + rng.normal(0, 2.0, N))    # S&P 500: upward drift, noise unrelated to butter
print(f"Correlation of the two levels: {np.corrcoef(butter, spx)[0, 1]:.2f}")

fig, ax1 = plt.subplots(figsize=(9, 5))
ax1.plot(butter, color="#E1A100", lw=1.5); ax1.set_ylabel("butter price", color="#E1A100"); ax1.set_xlabel("month")
ax2 = ax1.twinx()
ax2.plot(spx, color="#1F77B4", lw=1.5); ax2.set_ylabel("S&P 500", color="#1F77B4")
plt.title("Butter and the S&P 500, both drifting up (simulated, unrelated)")
fig.tight_layout(); plt.savefig("nr44_butter_spx.png", dpi=120, bbox_inches="tight"); plt.show()

Correlation of the two levels: 0.93

Correlation 0.93. On a chart the two climb together. Stop here and we would call butter a leading indicator for the S&P.

The naive regression agrees

A regression makes the claim even stronger: put butter on the right, the S&P on the left, and read how much of the S&P butter explains.

import statsmodels.api as sm
ols = sm.OLS(spx, sm.add_constant(butter)).fit()          # the naive move: regress the S&P on butter
print(f"R-squared:    {ols.rsquared:.2f}    (butter 'explains' {ols.rsquared:.0%} of the S&P)")
print(f"t-statistic:  {ols.tvalues[1]:.0f}     (past about 2 counts as significant)")
print(f"p-value:      {ols.pvalues[1]:.0e}   (this says certain)")

R-squared:    0.87    (butter 'explains' 87% of the S&P)
t-statistic:  63     (past about 2 counts as significant)
p-value:      9e-267   (this says certain)

R-squared 0.87, t-statistic 63, a p-value at zero. By the textbook, butter explains the S&P beyond doubt, and we might be tempted to trade on it. Do not: the result is spurious.

The t-statistic is only meaningful when the regression error behaves like noise around a stable average. Here it does not. The error still has a trend, so the regression measures the shared drift of the two series. Granger and Newbold named this the spurious regression in 1974.

What cointegration requires

The regression misled me because the series are non-stationary. Cointegration is built for that case, and it asks three things: both series non-stationary, both integrated of the same order, and a long enough sample to measure.

A stationary series has a constant mean and variance. It moves around a fixed level, and the size of those moves does not grow over time. A non-stationary series has a mean or variance that shifts over time. A price level is a good example. It can drift upward for long periods, so its average level today can be far above where it stood years ago. That does not make every rising price series economically related to every other rising price series.

A price is non-stationary, but its returns, the change from one period to the next, move around a stable mean and are stationary. A series that is non-stationary in levels and stationary after one difference is integrated of order one, written I(1), which is what I check next.

from statsmodels.tsa.stattools import adfuller
def adf(series):                                          # p-value of the Augmented Dickey-Fuller unit-root test
    return adfuller(series, autolag="AIC")[1]             # low (<0.05) = stationary; high = a unit root
print("                  levels    first difference")
print(f"butter  ADF p:    {adf(butter):.2f}      {adf(np.diff(butter)):.0e}")
print(f"S&P 500 ADF p:    {adf(spx):.2f}      {adf(np.diff(spx)):.0e}")
print("\nHigh in levels, about zero in differences, so both series are I(1).")

                  levels    first difference
butter  ADF p:    0.57      1e-17
S&P 500 ADF p:    0.94      0e+00

High in levels, about zero in differences, so both series are I(1).

Both series are I(1). In levels they are non-stationary, and the test says so: p-values of 0.57 and 0.94, too high to call either one stationary. Their month-to-month changes are stationary, with p-values near zero. That is the setup cointegration needs, so the test comes next. For the pandas way of turning a price level into those period-to-period changes, see Lesson 17 in Learn.

The cointegration test

Now the test itself. Cointegration asks whether a fixed combination of the two series, one minus a multiple of the other, is stationary even though each series alone is not. Call that combination the spread. A stationary spread tends to stay near its average. When the two series move apart, the gap has historically closed again. A spread that drifts off and does not come back means there is no stable long-run relationship.

I use the Engle-Granger test: regress the S&P on butter, then test whether the spread is stationary. The null is that the spread is non-stationary, suggesting no cointegration. A low p-value (below 0.05) rejects it: the spread looks stationary, which is evidence of cointegration.

from statsmodels.tsa.stattools import coint
p_coint = coint(spx, butter)[1]                           # Engle-Granger: is the spread (the leftover) stationary?
print(f"Cointegration p-value: {p_coint:.2f}    (low = stationary spread; high = none)")

Cointegration p-value: 0.45    (low = stationary spread; high = none)

The cointegration test returns p = 0.45, far too high to reject the null. So, I find no evidence of a stationary spread. The two are not cointegrated, and butter tells us nothing about the S&P.

What it means

Correlation is a fine measure when both series are stationary. The concern is when the two series trend together. So, a stationarity test is the right first step. If both series are stationary, correlation is at least measuring co-movement, and a shared trend cannot be the cause. If they are not, as prices often are, we need a test that accounts for that.

That test is cointegration, and butter and the S&P stand in for any two series we might compare. When two series are cointegrated, the spread tends to stay around a stable mean instead of drifting away indefinitely. That is what pairs trading uses: when the spread opens up, short the one that has risen and buy the one that has fallen, then close both when they come back together. The same idea applies to longer-run relationships across markets, but the economic story still has to come first. Without that story, a low p-value means nothing.

Do not blindly trust correlation in price levels. First check stationarity. If the series are non-stationary, check whether they are cointegrated.

Disclaimer: This post is a teaching example. Not investment advice.