Why does comparing with pd.NA give NA instead of True or False?

The short version

pd.NA propagates: comparing it with anything gives pd.NA, not False. That is deliberate, because the result of comparing an unknown value genuinely is unknown. So you cannot test for missing values with ==. Use isna(), which is the only correct way.

The behaviour

import pandas as pd

pd.NA == 1        # <NA> , not False
pd.NA == pd.NA    # <NA> , not True
pd.isna(pd.NA)    # True , the correct test

The same applies to np.nan, though for a different reason: np.nan == np.nan is False by the floating point standard, whereas pd.NA == pd.NA is NA. Both mean you cannot use equality to find missing values.

Why propagation is the right design

If a value is unknown, then "is it equal to 1" is also unknown. Answering False would be a claim you cannot justify, because the value might well have been 1. Propagating the uncertainty is logically honest, and it prevents missing data from silently masquerading as a definite negative answer.

This is called three-valued logic, and it is the same rule SQL uses with NULL, which is why WHERE column = NULL never matches anything in SQL either.

The consequence for filtering

An NA in a filter condition is neither True nor False, so it cannot be used to select rows directly and raises.

df[df["value"] == 1]     # works, but rows where value is NA are excluded

Rows with NA are quietly dropped from the result, because they are not definitely equal to 1. That is usually what you want, but be aware it is happening: your filtered frame excludes both the non-matches and the unknowns, and those are different things.

Three-valued logic in and or

The logical operators follow the same principle, and they are cleverer than you might expect.

pd.NA | True     # True , because True or anything is True
pd.NA & False    # False , because False and anything is False
pd.NA | False    # <NA> , the answer genuinely depends on the unknown

Where the answer is determined regardless of the unknown value, you get a definite result. Where it is not, you get NA. This is exactly right, and it is more careful than treating missing as False.

The rule to follow

Never test for missing values with == or !=. Always use isna() and notna().

df[df["value"].isna()]      # rows where the value is missing
df[df["value"].notna()]     # rows where it is present

In conditions, be explicit

When a condition must treat missing values as a specific outcome, say so, rather than relying on whatever the default happens to be.

mask = (df["value"] > 10).fillna(False)    # treat unknown as "does not match"
df[mask]

Filling the condition makes the decision visible in the code, which is far better than a reader having to work out what happened to the missing rows.