How do I use regular expressions on a pandas column?

The short version

Use .str.extract to pull parts of a value into new columns via capture groups, .str.contains to filter rows by pattern, and .str.replace(..., regex=True) to substitute. These are vectorised, so they run over the whole column without a loop. Rows that do not match extract come back as NaN, which is a useful signal.

Extracting parts into columns

import pandas as pd

codes = pd.Series(["PEN-001", "MUG-002"])

parts = codes.str.extract(r"([A-Z]+)-(\d+)")
print(parts)

     0    1
0  PEN  001
1  MUG  002

Each bracketed group becomes a column. Name the columns directly with named groups, which is far clearer than positional numbers.

codes.str.extract(r"(?P<prefix>[A-Z]+)-(?P<number>\d+)")

Non-matching rows become NaN

If a value does not match the pattern, extract returns NaN for it rather than raising. That makes it an excellent audit tool: count the NaN rows to find entries that break your expected format.

parts["prefix"].isna().sum()   # how many rows did not match

Filtering by pattern

.str.contains returns a boolean column, so it filters directly.

codes[codes.str.contains(r"^PEN", regex=True)]

Two frequent stumbles here. Missing values raise unless you pass na=False, and contains treats its argument as a regex by default, so characters like . and ( have special meaning. Pass regex=False for a literal substring.

Replacing with a pattern

codes.str.replace(r"\d+", "X", regex=True)   # ['PEN-X', 'MUG-X']

Note that regex must be set explicitly in recent pandas; the default changed, and relying on the old behaviour causes surprises. Set it deliberately every time so the intent is visible.

Extracting every match, not just the first

extract takes only the first match per row. To capture all matches, use extractall, which returns a row per match with a MultiIndex.

pd.Series(["a1b2"]).str.extractall(r"(\d)")

Keep patterns readable

Regular expressions get unreadable quickly. Use raw strings (r"...") so backslashes are not double-escaped, name your groups, and for anything complex, add a comment explaining what the pattern is meant to match. A pattern that nobody can read is a bug waiting to happen, and a slightly longer but clearer approach (splitting on a separator, say) is often better than a clever regex.

When not to use a regex

If the structure is simple and fixed, .str.split("-", expand=True) is clearer and faster than a pattern. Reach for regex when the structure genuinely varies, and prefer plain string methods when it does not.