Why can I not assign a new value to a category column in pandas?

The short version

A category column holds a fixed list of allowed values. Assigning a value that is not on that list raises an error. Add the new value to the categories first with add_categories, or convert the column back to plain text if the set of values is genuinely open.

The error

import pandas as pd

s = pd.Series(["a"], dtype="category")

s[0] = "z"
# TypeError: Cannot setitem on a Categorical with a new category

The column knows only about the category a. The value z is not a member, so the assignment is rejected rather than silently expanding the set.

Why this is a feature

The refusal is a guard. A category column represents a closed set of valid values, so an attempt to insert something outside that set usually means a typo or a data error, and the error catches it. Silently accepting new values would defeat the purpose of declaring the column categorical in the first place.

This is the same principle as a database check constraint: it protects you from writing data that violates the schema.

Adding a legitimate new category

If the new value is valid and simply was not present before, add it explicitly.

s = s.cat.add_categories(["z"])
s[0] = "z"    # now allowed

Making this an explicit step means the widening of the allowed set is visible in the code, and a reviewer can see it happened.

Removing unused ones

After filtering, the category list still holds every original level, even ones with no remaining rows. That is why a groupby on a filtered categorical can still produce empty groups.

s = s.cat.remove_unused_categories()

When to convert back

If the column's values are genuinely open-ended and change as you work, the category dtype is fighting you rather than helping. Convert it back.

s = s.astype("string")

Use category for a column with a stable, known set of values (sizes, statuses, regions) where you want the memory saving, the ordering, and the validation. Use plain text for anything freeform.

The same trap when merging

Merging two frames whose category columns have different category lists often results in the column silently reverting to object, losing your memory saving and any ordering you set. Check the dtype after a merge.

merged["kind"].dtype

If you need the category dtype to survive, make both sides share the same category list before merging, or reapply the dtype afterwards.

The fill trap

fillna on a categorical raises if the fill value is not a category, which catches people out.

s.fillna("unknown")            # TypeError unless 'unknown' is a category
s.cat.add_categories(["unknown"]).fillna("unknown")   # correct

This is consistent with everything above: the category list is a contract, and every value entering the column must honour it.