How do I fill missing values using each group's own values in pandas?

The short version

Use groupby with transform so each group's missing values are filled from that group's own data, not from the whole column. For example, fill each city's gaps with that city's mean. Be aware that a group with no known values at all cannot be filled from itself, so those NaN entries survive and need separate handling.

Filling with each group's mean

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "city": ["Leeds", "Leeds", "Bath"],
    "temp": [np.nan, 20.0, np.nan],
})

df["temp"] = df.groupby("city")["temp"].transform(lambda s: s.fillna(s.mean()))
print(df)

    city  temp
0  Leeds  20.0
1  Leeds  20.0
2   Bath   NaN

Leeds had one known value (20), so its gap is filled with 20. Bath had no known value, so its mean is NaN and the gap remains.

Why transform, not a plain fillna

df["temp"].fillna(df["temp"].mean()) fills every gap with the overall mean across all cities, which blurs real differences between groups. The groupby(...).transform version keeps each group's fill local to that group, which is usually what you want, because a Leeds reading should be estimated from Leeds, not from Bath.

The all-missing group trap

The example shows the catch: a group that is entirely missing cannot produce a mean, so its gaps stay NaN. Never assume the fill leaves no missing values behind. Check afterwards with isna and decide how to handle any group that had nothing to fill from, perhaps by falling back to the overall mean.

df["temp"] = df["temp"].fillna(df["temp"].mean())   # global fallback for leftovers

Filling by carrying values forward within groups

For ordered data, you often want to carry the last known value forward, but only within each group so one group does not leak into the next. Sort first, then forward fill per group.

df = df.sort_values(["city", "date"])
df["value"] = df.groupby("city")["value"].ffill()

Without the groupby, a plain ffill would carry the final value of one city into the first rows of the next, which is a subtle and common bug.

The principle

When filling gaps, ask whether the estimate should come from the whole column or just from similar rows. Grouped filling respects the structure in your data, so prefer it whenever the groups are meaningfully different, and always verify what remains missing afterwards.