Q&A library Python

How do I group by more than one column in pandas?

Python Intermediate as-indexgroupbymultiindex

Short answer

Pass a list of columns to groupby and the result gets a MultiIndex of the key combinations. Use as_index=False, or reset_index, to get a flat frame with the keys back as columns.

The short version

Pass a list of column names to groupby. The result is indexed by every observed combination of the keys, which gives you a MultiIndex. Use as_index=False or call reset_index() to get an ordinary flat table with the keys back as columns.

Grouping on two keys

import pandas as pd

df = pd.DataFrame({
    "city": ["Leeds", "Leeds", "Bath"],
    "kind": ["online", "shop", "online"],
    "amount": [1, 2, 3],
})

df.groupby(["city", "kind"])["amount"].sum()
city   kind
Bath   online    3
Leeds  online    1
       shop      2

Each row is one combination that actually occurs in the data. Combinations that never happened do not appear at all, which is usually what you want.

Getting a flat table back

df.groupby(["city", "kind"], as_index=False)["amount"].sum()
    city    kind  amount
0   Bath  online       3
1  Leeds  online       1
2  Leeds    shop       2

This is far easier to work with for anything downstream: merging, exporting, plotting. reset_index() after the grouping does the same thing.

Named aggregations still work

df.groupby(["city", "kind"], as_index=False).agg(
    total=("amount", "sum"),
    orders=("amount", "size"),
)

This gives clearly named columns and a flat result in one step, which is usually the cleanest form.

Turning one key into columns

A grouped result with two keys often reads better as a grid, one key down the side and the other across the top. unstack does that.

df.groupby(["city", "kind"])["amount"].sum().unstack("kind")

This is equivalent to a pivot_table and is the natural next step from a two-key grouping.

The order of the keys matters

The first key becomes the outer level, which affects both the sort order and how easy selection is later. Put the key you will most often filter or select on first.

df.groupby(["kind", "city"])   # kind is now the outer level

Missing keys are still dropped

The same trap applies as with a single key: a row whose any grouping key is missing is excluded entirely, so your totals can silently under-count. Pass dropna=False to keep those rows as their own group.

df.groupby(["city", "kind"], dropna=False)["amount"].sum()

With several keys this matters more, not less, because a row is dropped if even one of its keys is missing, so the loss can be larger than you expect. Always reconcile the grouped total against the ungrouped one.

df["amount"].sum() == df.groupby(["city", "kind"])["amount"].sum().sum()

If those disagree, missing keys are the first thing to check.

Need this solved on your own data?

The library is free and always will be. When the problem is bigger than an answer, we take on the work.

Start a project