How do I stop forward fill carrying a value too far in pandas?

The short version

ffill carries the last known value forward across every following gap, however long. Pass limit=n to cap how many consecutive gaps it will fill. And use groupby so a value from one group never leaks into the next, which is a silent and common error.

Limiting the fill

import pandas as pd
import numpy as np

s = pd.Series([1, np.nan, np.nan, np.nan])

s.ffill().tolist()            # [1.0, 1.0, 1.0, 1.0] , filled everything
s.ffill(limit=1).tolist()     # [1.0, 1.0, nan, nan] , only one gap filled

Without a limit, a single reading from January could be carried across an entire year of missing months, presenting a year of invented data as if it were observed.

Why the limit matters

Forward filling encodes an assumption: that the value has not changed since it was last observed. That is reasonable across a short gap (a sensor missed one reading) and unreasonable across a long one (a sensor was offline for six months). The limit argument is where you state how long that assumption stays credible, and it forces you to think about it rather than filling blindly.

The group leak

Across several groups, an unguarded ffill carries the last value of one group into the first rows of the next.

df["value"] = df.groupby("city")["value"].ffill()

Without the groupby, Bath's opening gap would be filled with Leeds's final value. Nothing warns you. The number looks entirely plausible and is entirely wrong.

Sort first

Like every row-order operation, ffill trusts the current order. Sort by group and time before filling, or you carry values forward through rows that are not actually consecutive.

df = df.sort_values(["city", "date"])

Backward fill and interpolation

bfill carries the next known value backwards, which is occasionally right (for example, a value recorded at the end of a period that applies throughout it). interpolate estimates values between the known points rather than repeating one, which usually suits a smoothly changing measure better than a flat carried-forward line.

Choose by what the missing value means: repeat the last known state with ffill, estimate a smooth path with interpolate, or leave it missing when you genuinely do not know.

Say what you did

A forward-filled series looks exactly like an observed one. If a chart or a table contains filled values, note it, or a reader will take invented flat stretches for real measurements. That is a matter of honesty as much as technique.