How do I use yesterday’s value?

Move a column down one row with .shift(1) so every row can read the row above it.

.shift(1) moves every value down one row. The index stays where it is, so row 3 now holds the value that used to sit on row 2. A row can then read the value from the row before it, on the same line.

In this lesson I shift a small Series and print it next to the original, rebuild .pct_change() out of a shift, then put a Close_yesterday column on a price table read from a CSV and shift the other way with .shift(-1).

Step 1. What the move looks like

Four prices. s.shift(1) returns a Series of the same length, with each value one row lower and a NaN at the top.

import pandas as pd

s = pd.Series([100.0, 110.0, 99.0, 108.0], name="Close")    # four closes, oldest first

print(s.shift(1))                                           # every value moved down one row
# -> 0      NaN
# -> 1    100.0
# -> 2    110.0
# -> 3     99.0
# -> Name: Close, dtype: float64

0      NaN
1    100.0
2    110.0
3     99.0
Name: Close, dtype: float64

Put the two side by side in a DataFrame.

side = pd.DataFrame({"Close": s, "shifted": s.shift(1)})    # original and shifted, side by side

print(side)
# ->    Close  shifted
# -> 0  100.0      NaN
# -> 1  110.0    100.0
# -> 2   99.0    110.0
# -> 3  108.0     99.0

   Close  shifted
0  100.0      NaN
1  110.0    100.0
2   99.0    110.0
3  108.0     99.0

Row 1 has today’s 110.0 and yesterday’s 100.0 on the same line. Row 0 has no row above it, so its shifted value is NaN.

Nothing is dropped. Both columns are four rows long, and one value is missing.

print(len(s), len(s.shift(1)))    # -> 4 4   <- same length, nothing dropped
print(s.shift(1).isna().sum())    # -> 1     <- the one gap sits at the top

4 4
1

Step 2. A return is a shift

A return divides today’s price by yesterday’s price and subtracts 1. With s.shift(1) giving yesterday’s price on today’s row, that sentence becomes one line of code.

by_hand  = s / s.shift(1) - 1    # today's price over yesterday's, minus 1
built_in = s.pct_change()        # pandas doing the same arithmetic

print(pd.DataFrame({"by_hand": by_hand, "pct_change": built_in}))
# ->     by_hand  pct_change
# -> 0       NaN         NaN
# -> 1  0.100000    0.100000
# -> 2 -0.100000   -0.100000
# -> 3  0.090909    0.090909

    by_hand  pct_change
0       NaN         NaN
1  0.100000    0.100000
2 -0.100000   -0.100000
3  0.090909    0.090909

print(by_hand.equals(built_in))   # -> True

True

.pct_change() from Lesson 17 is this shift with the arithmetic written for you.

Step 3. Yesterday’s close on a price table

prices.csv holds 40 business days of daily closes for three tickers, simulated rather than real market history.

Shifting mixes tickers together if you shift the whole file at once, so take one ticker first. reset_index(drop=True) renumbers the rows from 0 after the filter.

prices = pd.read_csv("prices.csv")                                  # three tickers, 40 days
aapl   = prices[prices["Ticker"] == "AAA"].reset_index(drop=True)  # one ticker, renumbered from 0

print(aapl.shape)   # -> (40, 4)

(40, 4)

Now two new columns: yesterday’s close, and today minus yesterday.

aapl["Close_yesterday"] = aapl["Close"].shift(1)                   # yesterday's close on this row
aapl["Change"]          = aapl["Close"] - aapl["Close_yesterday"]  # today minus yesterday

print(aapl[["Date", "Close", "Close_yesterday", "Change"]].head(6))
# ->          Date   Close  Close_yesterday  Change
# -> 0  2026-01-02  186.66              NaN     NaN
# -> 1  2026-01-05  186.38           186.66   -0.28
# -> 2  2026-01-06  188.51           186.38    2.13
# -> 3  2026-01-07  187.24           188.51   -1.27
# -> 4  2026-01-08  189.27           187.24    2.03
# -> 5  2026-01-09  185.32           189.27   -3.95

         Date   Close  Close_yesterday  Change
0  2026-01-02  186.66              NaN     NaN
1  2026-01-05  186.38           186.66   -0.28
2  2026-01-06  188.51           186.38    2.13
3  2026-01-07  187.24           188.51   -1.27
4  2026-01-08  189.27           187.24    2.03
5  2026-01-09  185.32           189.27   -3.95

Read row 2 across: the close on 6 January was 188.51, the close on 5 January was 186.38, and the price rose 2.13. Both numbers sit on one row, so the subtraction is a column minus a column.

The first row is the only gap.

print(aapl["Change"].isna().sum())   # -> 1

Step 4. `.shift(-1)` moves the other way

A negative number moves values up. shift(-1) puts tomorrow’s close on today’s row.

aapl["Close_tomorrow"] = aapl["Close"].shift(-1)    # tomorrow's close on today's row

print(aapl[["Date", "Close", "Close_yesterday", "Close_tomorrow"]].head(4))
# ->          Date   Close  Close_yesterday  Close_tomorrow
# -> 0  2026-01-02  186.66              NaN          186.38
# -> 1  2026-01-05  186.38           186.66          188.51
# -> 2  2026-01-06  188.51           186.38          187.24
# -> 3  2026-01-07  187.24           188.51          189.27

         Date   Close  Close_yesterday  Close_tomorrow
0  2026-01-02  186.66              NaN          186.38
1  2026-01-05  186.38           186.66          188.51
2  2026-01-06  188.51           186.38          187.24
3  2026-01-07  187.24           188.51          189.27

The NaN moves to the bottom, because the last row has no day after it.

print(aapl[["Date", "Close", "Close_yesterday", "Close_tomorrow"]].tail(3))
# ->           Date   Close  Close_yesterday  Close_tomorrow
# -> 37  2026-02-24  172.64           171.75          171.42
# -> 38  2026-02-25  171.42           172.64          169.42
# -> 39  2026-02-26  169.42           171.42             NaN

          Date   Close  Close_yesterday  Close_tomorrow
37  2026-02-24  172.64           171.75          171.42
38  2026-02-25  171.42           172.64          169.42
39  2026-02-26  169.42           171.42             NaN

shift(1) puts a value you already knew at the previous row onto the current row. shift(-1) puts a value from a later row onto the current row.

Your turn

Take CCC out of prices.csv, add a Close_yesterday column, build a Return column as Close / Close_yesterday - 1, and check it against pct_change().

Show answer

import pandas as pd

prices = pd.read_csv("prices.csv")
msft   = prices[prices["Ticker"] == "CCC"].reset_index(drop=True)

msft["Close_yesterday"] = msft["Close"].shift(1)                       # yesterday's close
msft["Return"]          = msft["Close"] / msft["Close_yesterday"] - 1  # the return by hand

print(msft[["Date", "Close", "Close_yesterday", "Return"]].head(4))
# ->          Date   Close  Close_yesterday    Return
# -> 0  2026-01-02  416.98              NaN       NaN
# -> 1  2026-01-05  423.19           416.98  0.014893
# -> 2  2026-01-06  423.95           423.19  0.001796
# -> 3  2026-01-07  435.34           423.95  0.026866

print(msft["Return"].equals(msft["Close"].pct_change()))   # -> True

Shifting more than one row

The number is how many rows to move. shift(2) reaches two rows back.

print(pd.DataFrame({"s": s, "shift1": s.shift(1), "shift2": s.shift(2)}))  # one and two rows back
# ->        s  shift1  shift2
# -> 0  100.0     NaN     NaN
# -> 1  110.0   100.0     NaN
# -> 2   99.0   110.0   100.0
# -> 3  108.0    99.0   110.0

       s  shift1  shift2
0  100.0     NaN     NaN
1  110.0   100.0     NaN
2   99.0   110.0   100.0
3  108.0    99.0   110.0

shift(2) leaves two missing values at the top, shift(3) leaves three.