Why does groupby drop rows with missing keys in pandas?

The short version

groupby ignores rows whose grouping key is missing, so those rows disappear from the output entirely. Your group totals then quietly under-count, and nothing warns you. Pass dropna=False to keep the missing key as its own group.

Seeing the loss

import pandas as pd

df = pd.DataFrame({
    "city": ["Leeds", None, "Bath"],
    "amount": [1, 2, 3],
})

print(df.groupby("city")["amount"].sum())

city
Bath     3
Leeds    1

The total is 4, but the column sums to 6. The row with the missing city, worth 2, has vanished without a word.

Keeping the missing group

df.groupby("city", dropna=False)["amount"].sum()

city
Bath     3
Leeds    1
NaN      2

Now the missing key forms its own group, the totals reconcile, and you can see exactly how much data has no key.

Why this matters so much

This is one of the most dangerous silent behaviours in pandas, because the result looks perfectly reasonable. A report of sales by city simply omits the sales whose city was never recorded, and the numbers are wrong in a way that no error message reveals. If you have ever had a grouped total fail to match the overall total, this is the first thing to check.

The reconciliation check

Make it a habit to compare the grouped total against the ungrouped one.

df["amount"].sum() == df.groupby("city")["amount"].sum().sum()   # False here!

When these disagree, missing keys are the usual culprit. This single check catches the problem immediately.

Deciding what to do

Keeping the NaN group is not always the right answer; it is the right default for finding out. Once you can see how many rows lack a key, decide deliberately: fill the key if you can infer it, drop the rows if they are unusable, or report the missing group explicitly so readers know it exists. What you should not do is let it disappear by accident.

The same trap when grouping on several keys

With multiple grouping columns, a row is dropped if any one of the keys is missing, so the loss can be larger than you expect. The dropna=False argument applies to all the keys together, so the same fix works.

A related surprise with categoricals

If you group by a categorical column, pandas by default includes every defined category, even those with no rows, producing empty groups. That is the opposite problem: extra rows rather than missing ones. Pass observed=True to include only the categories actually present.