Q&A library Python

How do I filter rows matching any of several text patterns in pandas?

Python Intermediate containsregexstrings

Short answer

Join the patterns with the regex or symbol inside str.contains, or build the pattern from a list. Set na=False so missing values do not raise, and escape special characters when matching literally.

The short version

Use str.contains with the patterns joined by the regex alternation symbol |, which means "or". Build the pattern from a list when there are many. Pass na=False so rows with missing values are treated as not matching rather than raising an error.

Matching any of several patterns

import pandas as pd

s = pd.Series(["pen red", "mug blue", "cup green"])

s.str.contains("red|blue", na=False).tolist()   # [True, True, False]

The | inside the pattern means "red or blue". This is far cleaner than chaining several contains calls with | between them.

Building the pattern from a list

When the patterns come from a variable, join them.

wanted = ["red", "blue"]
pattern = "|".join(wanted)
s[s.str.contains(pattern, na=False)]

This scales to any number of terms and keeps the list of terms separate from the matching logic, which is easier to maintain.

Escaping special characters

If your terms contain regex characters (a full stop, a bracket, a plus), they will be interpreted as pattern syntax and match the wrong things. Escape them.

import re
pattern = "|".join(re.escape(term) for term in wanted)

re.escape makes each term literal. Without it, a term like a.b would match axb, which is a subtle and easily missed bug.

The missing-value trap

Without na=False, a NaN in the column produces NaN in the result rather than False, and using that as a filter raises "cannot mask with non-boolean array". Setting na=False treats missing values as non-matching, which is almost always what you want.

Whole-value matching versus containment

contains looks for the pattern anywhere in the value. If you want an exact match against a set of values, isin is the right tool and is faster, because it does no pattern matching at all.

s.isin(["pen red", "mug blue"])   # exact matches only

Use contains for substrings and isin for whole values. Reaching for a regex when a simple membership test would do is a common over-complication.

Case-insensitive matching

Pass case=False to ignore capitalisation, which matters for real text where the same word appears in several forms.

s.str.contains("RED", case=False, na=False)

Matching the start or end

Anchor the pattern with ^ for the start and $ for the end, or use the dedicated methods, which are clearer.

s.str.startswith("pen", na=False)
s.str.endswith("green", na=False)

These say exactly what they mean and avoid the reader having to parse a regex, so prefer them when they fit.

Need this solved on your own data?

The library is free and always will be. When the problem is bigger than an answer, we take on the work.

Start a project