Why does groupby on a categorical column produce empty groups in pandas?

The short version

When you group by a category column, pandas includes every category defined on that column, even those with no matching rows. The result is padded with empty groups showing zeros or NaN. Pass observed=True to include only the categories that actually appear in the data.

Seeing the padding

import pandas as pd

df = pd.DataFrame({
    "kind": pd.Categorical(["a", "a"], categories=["a", "b", "c"]),
    "amount": [1, 2],
})

len(df.groupby("kind", observed=False)["amount"].sum())   # 3 groups
len(df.groupby("kind", observed=True)["amount"].sum())    # 1 group

The data only contains a, but the column defines three categories, so the default grouping returns three rows, two of them empty.

Why pandas does this

The category dtype records the full set of valid levels, not just the ones present. Including them all is deliberate: it guarantees every level appears in the output, which is useful when you want a complete table with zeros for the levels that had no activity. It becomes a nuisance when you only want what is really there.

The damage it does

Empty groups distort summaries and charts. A bar chart gains empty bars, a mean over the groups includes meaningless entries, and the row count of your summary no longer matches the number of real categories. With several categorical grouping keys, the effect multiplies: grouping by two categoricals produces every combination of their levels, which can explode into a huge, mostly empty result.

The fix

df.groupby("kind", observed=True)["amount"].sum()

Set observed=True whenever you want the groups that exist in the data. This is usually what people mean.

When the padding is what you want

If you are producing a report where every category must appear, even with zero activity, keep observed=False and fill the gaps.

df.groupby("kind", observed=False)["amount"].sum().fillna(0)

This guarantees a complete table, which is genuinely useful for a fixed set of levels like sizes or priority bands.

Watch for it after filtering

Filtering rows does not remove categories from the dtype. So if you filter down to one city and then group by a categorical city column, you still get every original city as a group. To drop the now-unused levels, call remove_unused_categories.

df["kind"] = df["kind"].cat.remove_unused_categories()

This is the other half of the fix, and it prevents stale levels from following your data around after a filter.

A note on the changing default

The default for observed has been moving towards True in recent pandas versions, and you may see a FutureWarning prompting you to set it explicitly. Setting it deliberately in your code is the right response, because it makes the behaviour clear and version-proof.