How do I flag rows where a value changed in pandas?

The short version

Compare the column against itself shifted down by one row. Wherever they differ, the value changed. Group first so a change is never detected across a group boundary, and sort so "the previous row" means what you think it does.

The pattern

import pandas as pd

df = pd.DataFrame({
    "city": ["Leeds", "Leeds", "Leeds"],
    "status": ["on", "on", "off"],
})

df["changed"] = df["status"] != df["status"].shift(1)
print(df["changed"].tolist())   # [True, False, True]

Row 0 is flagged because there is nothing before it (the comparison with NaN is True). Row 1 matches the row above, so it is False. Row 2 differs, so it is True.

The first-row behaviour

The first row always comes out as changed, because shift gives it NaN and NaN != anything is True. That is usually what you want: the first observation is a new state, with nothing before it. If you would rather not flag it, set it explicitly.

df.loc[df.index[0], "changed"] = False

But think about whether you really want that, because the first row genuinely does establish a state.

Within groups

Across several groups, an ungrouped comparison detects a spurious "change" wherever one group ends and the next begins, and misses a real change at the start of a group.

df = df.sort_values(["city", "date"])
df["changed"] = df["status"] != df.groupby("city")["status"].shift(1)

Now the first row of each city is correctly flagged as a new state, and no change is detected across the boundary between two cities.

Counting the changes

df["changed"].sum()                            # total changes
df.groupby("city")["changed"].sum()            # changes per city

Turning changes into run identifiers

Taking the cumulative sum of the change flags gives an identifier that increments at each new run, which lets you group consecutive blocks.

df["run"] = df["changed"].cumsum()
df.groupby("run").size()      # how long each run lasted

This is the standard route from "where did it change" to "how long did each state last".

Missing values need care

If the column itself contains NaN, the comparison behaves oddly, because NaN != NaN is True, so two consecutive missing values look like a change. If missing values should count as a state rather than a change, fill them with a sentinel before comparing.

status = df["status"].fillna("unknown")
df["changed"] = status != status.shift(1)

This is a small detail that quietly corrupts change counts in real data, so it is worth handling deliberately.