How do I get the previous value within each group in pandas?

The short version

Use groupby(key)[col].shift(1) so each row sees the previous value within its own group. Sort by the group key and the ordering column first. The first row of every group correctly comes out as NaN, because it has no predecessor.

The pattern

import pandas as pd

df = pd.DataFrame({
    "city": ["Leeds", "Leeds", "Bath"],
    "amount": [1, 2, 3],
})

df["previous"] = df.groupby("city")["amount"].shift(1)
print(df["previous"].tolist())   # [nan, 1.0, nan]

The second Leeds row sees 1, the first row of each city sees nothing. That is exactly right.

The bug the groupby prevents

Without it, a plain shift(1) carries the last value of one group into the first row of the next.

df["previous"] = df["amount"].shift(1)   # [nan, 1.0, 2.0]

Bath's first row now shows 2, which came from Leeds. It is a silent, plausible-looking error: no exception, just a wrong number. Any change, growth rate or gap computed from it is quietly contaminated by an unrelated group.

Sort first, always

shift works on row order, so the "previous" row is only meaningful if the rows are in the right order.

df = df.sort_values(["city", "date"])
df["previous"] = df.groupby("city")["amount"].shift(1)

Sorting by the group key and the ordering column is the habit to build. Sorting by date alone is not enough, because the groups would be interleaved.

The related operations

The same grouping applies to the whole family of row-to-row comparisons.

df.groupby("city")["amount"].diff()         # change from the previous row
df.groupby("city")["amount"].pct_change()   # proportional change
df.groupby("city")["amount"].shift(-1)      # the NEXT value

Each needs the groupby for exactly the same reason, and each produces NaN at the group boundary, which is the correct answer rather than a defect.

Filling the boundary

Do not reflexively fill those NaN values with zero. A missing "previous value" means there was no previous observation, which is different from a previous value of zero. Filling it with zero would make the first change of each group look like a rise from nothing, inventing growth that did not happen.

If you need a value there, decide deliberately, and consider whether those rows should simply be excluded from the comparison instead.

Using it to detect changes

A common use is flagging where a value changed within a group, which becomes a simple comparison once you have the previous value.

df["changed"] = df["status"] != df.groupby("city")["status"].shift(1)

The first row of each group is flagged as changed (since the comparison with NaN is True), which is usually what you want, because a group's first observation is a new state.