How do I group by more than one column in pandas?
Short answer
Pass a list of columns to groupby and the result gets a MultiIndex of the key combinations. Use as_index=False, or reset_index, to get a flat frame with the keys back as columns.
The short version
Pass a list of column names to groupby. The result is indexed by every observed combination of the keys, which gives you a MultiIndex. Use as_index=False or call reset_index() to get an ordinary flat table with the keys back as columns.
Grouping on two keys
import pandas as pd
df = pd.DataFrame({
"city": ["Leeds", "Leeds", "Bath"],
"kind": ["online", "shop", "online"],
"amount": [1, 2, 3],
})
df.groupby(["city", "kind"])["amount"].sum()
city kind
Bath online 3
Leeds online 1
shop 2
Each row is one combination that actually occurs in the data. Combinations that never happened do not appear at all, which is usually what you want.
Getting a flat table back
df.groupby(["city", "kind"], as_index=False)["amount"].sum()
city kind amount
0 Bath online 3
1 Leeds online 1
2 Leeds shop 2
This is far easier to work with for anything downstream: merging, exporting, plotting. reset_index() after the grouping does the same thing.
Named aggregations still work
df.groupby(["city", "kind"], as_index=False).agg(
total=("amount", "sum"),
orders=("amount", "size"),
)
This gives clearly named columns and a flat result in one step, which is usually the cleanest form.
Turning one key into columns
A grouped result with two keys often reads better as a grid, one key down the side and the other across the top. unstack does that.
df.groupby(["city", "kind"])["amount"].sum().unstack("kind")
This is equivalent to a pivot_table and is the natural next step from a two-key grouping.
The order of the keys matters
The first key becomes the outer level, which affects both the sort order and how easy selection is later. Put the key you will most often filter or select on first.
df.groupby(["kind", "city"]) # kind is now the outer level
Missing keys are still dropped
The same trap applies as with a single key: a row whose any grouping key is missing is excluded entirely, so your totals can silently under-count. Pass dropna=False to keep those rows as their own group.
df.groupby(["city", "kind"], dropna=False)["amount"].sum()
With several keys this matters more, not less, because a row is dropped if even one of its keys is missing, so the loss can be larger than you expect. Always reconcile the grouped total against the ungrouped one.
df["amount"].sum() == df.groupby(["city", "kind"])["amount"].sum().sum()
If those disagree, missing keys are the first thing to check.