How do I compare each row to the previous one in pandas?

The short version

shift(1) pulls the previous row's value alongside the current one, diff() gives the change since the previous row, and pct_change() gives that change as a proportion. The first row has nothing before it, so it comes out as NaN. Sort the data first, and use groupby when comparisons must stay within each group.

The three tools

import pandas as pd

temps = pd.Series([10, 12, 9, 15])

print("previous", temps.shift(1).tolist())   # [nan, 10.0, 12.0, 9.0]
print("change  ", temps.diff().tolist())      # [nan, 2.0, -3.0, 6.0]
print("percent ", temps.pct_change().tolist())# [nan, 0.2, -0.25, 0.667]

shift(1) moves every value down one row, so each row now sees what came before it. diff() is simply the current value minus that shifted value. pct_change() divides the change by the previous value.

Why the first row is NaN

There is no row before the first, so the previous value is undefined and the result is NaN. This is correct, not a bug. Decide whether to drop that row, fill it with zero, or leave the gap, depending on what a missing comparison means for your analysis.

Looking forward instead

A negative offset looks ahead rather than behind, which is useful for "what happens next" comparisons.

temps.shift(-1)   # the next row's value

The ordering trap

All three functions work on the current row order, so unsorted data compares rows that are not really adjacent. If your rows have a natural order (a date, a sequence number), sort by it first, or every comparison is meaningless.

df = df.sort_values("date")
df["change"] = df["value"].diff()

Keeping comparisons within groups

Across several groups, a plain diff would compare the first row of one group with the last row of the previous group, which is wrong. Group first so each comparison stays inside its own group.

df["change"] = df.sort_values(["city", "date"]).groupby("city")["value"].diff()

Now the first row of each city correctly comes out as NaN, because there is no earlier row for that city to compare against.