Why does my missing-value check miss some rows in pandas?

The short version

isna() detects NaN, None and NaT, but not an empty string, a string of spaces, or a placeholder like "NA", "-" or -1. Those are ordinary values as far as pandas is concerned. Convert them into real missing values first, or every missing-value check and fill will silently pass them by.

The problem

import pandas as pd
import numpy as np

s = pd.Series(["Leeds", "", "  ", "NA", None, np.nan])

s.isna().sum()   # 2 , only the None and the NaN

Four of those six values are missing in any meaningful sense, but pandas counts only two. A dropna() would leave the empty string and the "NA" text in place, and a downstream groupby would treat them as real categories.

Converting placeholders at read time

The cleanest fix is to tell the reader what counts as missing in this file.

df = pd.read_csv(
    "orders.csv",
    na_values=["", " ", "NA", "N/A", "-", "unknown"],
    keep_default_na=True,
)

Now those values arrive as genuine NaN, and every missing-value tool works as expected from that point on.

Converting after the fact

If the data is already loaded, replace the placeholders yourself.

s = s.replace(["", " ", "NA", "-"], np.nan)

For whitespace of arbitrary length, strip first so that " " becomes "" and is caught.

s = s.str.strip().replace("", np.nan)

Numeric placeholders are more dangerous

Text placeholders at least look wrong. A numeric sentinel like -1 or 999 for "not recorded" is far worse, because it is a perfectly valid number that will be silently included in every mean, sum and chart.

df["age"] = df["age"].replace(-1, np.nan)

There is no way for pandas to know that -1 means missing. Only you know that, from the data dictionary, which is why reading the documentation of a dataset matters as much as reading the data.

Check what values actually exist

Before trusting any missing-value count, look at what the column really contains.

s.value_counts(dropna=False).head(20)

This surfaces the placeholders immediately: a suspiciously frequent "NA", "-" or 999 will stand out at once, and finding them here is far cheaper than discovering that your average was computed over a column full of sentinel values.

The order of cleaning

Normalise text (strip, standardise case), convert placeholders to NaN, then convert the column to its proper type, and only then count and fill the missing values. Doing these in the wrong order is why a careful-looking pipeline can still produce a column where a tenth of the "values" are the word "unknown".