Q&A library Python

Why does mean return nan in NumPy, and how do I ignore missing values?

Python Intermediate aggregationnannumpy

Short answer

Any nan in an array poisons a normal mean or sum, because nan propagates through arithmetic. Use the nan-aware functions like nanmean and nansum to skip missing values instead.

The short version

If an array contains even one nan, ordinary mean or sum returns nan, because nan propagates through every calculation it touches. To skip missing values, use the nan-aware versions: np.nanmean, np.nansum, np.nanmax and so on. Unlike pandas, plain NumPy does not ignore missing values for you.

Seeing the problem

import numpy as np

a = np.array([1.0, np.nan, 3.0])

print(np.mean(a))      # nan
print(np.nanmean(a))   # 2.0

The normal mean is poisoned by the single nan. nanmean ignores it and averages the two real values.

Why nan propagates

nan means "not a number", and by design any arithmetic involving it results in nan: 1 + nan is nan, and so is the whole sum. This is correct behaviour, because the true result is genuinely unknown when a value is missing. The nan-aware functions choose instead to drop the missing entries before computing, which is often what you actually want.

The nan-aware family

Most reducing functions have a nan counterpart.

np.nansum(a)     # sum ignoring nan
np.nanmax(a)     # max ignoring nan
np.nanstd(a)     # standard deviation ignoring nan
np.nanmin(a)     # min ignoring nan

Reach for these whenever missing values are expected and you want them skipped rather than propagated.

Detecting nan correctly

You cannot test for nan with ==, because nan is not equal to anything, including itself.

np.nan == np.nan     # False
np.isnan(a)          # [False  True False], the correct test

Use np.isnan to find missing values, for example to count them or to build a mask that drops them.

Dropping instead of using nan-functions

Sometimes you want the clean values themselves, not just an aggregate. Filter them out with a mask.

clean = a[~np.isnan(a)]   # [1. 3.]

This is useful when you need to pass only the real values to a function that has no nan-aware version.

The all-nan edge case

If every value is nan, even nanmean cannot compute an average and returns nan with a warning. So the nan-aware functions solve "some values are missing", not "all values are missing". Check whether any real values remain when that is possible in your data.

The pandas contrast

Pandas silently skips nan in its aggregations by default, so series.mean() ignores missing values without you asking. NumPy does not, which is why you must choose the nan-aware function explicitly. Knowing this difference prevents the surprise of a NumPy result coming back as nan when the pandas equivalent would not.

Need this solved on your own data?

The library is free and always will be. When the problem is bigger than an answer, we take on the work.

Start a project