Q&A library Python

Why do I get NaN when adding two pandas Series together?

Python Advanced alignmentarithmeticindex

Short answer

Pandas aligns on the index before doing arithmetic, so labels present in one Series but not the other produce NaN. Reset the index, or use values, when you mean to combine by position.

The short version

Pandas aligns two Series by their index labels before combining them, not by position. Any label present in one but not the other produces NaN. If you meant to add them position by position, the indexes must match, so reset them or use the underlying values.

Seeing the surprise

import pandas as pd

x = pd.Series([1, 2], index=[0, 1])
y = pd.Series([3, 4], index=[1, 2])

print(x + y)
0    NaN
1    5.0
2    NaN
dtype: float64

Only label 1 exists in both, so only it produces a number (2 + 3). Labels 0 and 2 are missing from one side, so they yield NaN. Nothing was added position by position at all.

Why pandas does this

Alignment is a feature, not a quirk. It means you can add two Series about the same entities without worrying whether they are in the same order, and pandas will match them correctly by label. That is a genuine safeguard against silently combining the wrong rows. The confusion only arises when you think of a Series as a list and expect positional behaviour.

When it bites in real code

The usual cause is arithmetic between a column of a filtered frame and a column of the original.

filtered = df[df["amount"] > 0]      # index now 0, 3, 7
df["ratio"] = filtered["amount"] / df["amount"]   # NaN for the missing labels

The filtered frame kept the original labels, so the two sides do not fully align. This produces a column peppered with NaN for reasons that are not obvious from reading the line.

Combining by position instead

If you genuinely want position-by-position arithmetic, strip the index by taking the raw values.

x.values + y.values     # array([4, 6]) , positional, ignores labels

Or reset both indexes so the labels line up.

x.reset_index(drop=True) + y.reset_index(drop=True)   # [4, 6]

Do this deliberately, since you are overriding a safety mechanism.

Filling instead of NaN

If you want the missing labels treated as a value rather than NaN, use the arithmetic methods, which take a fill value.

x.add(y, fill_value=0)   # [1.0, 5.0, 4.0]

This treats an absent label as zero, so every label produces a result. It is much clearer than adding and then calling fillna, because it states the intent in one place.

The habit to build

When a pandas arithmetic result is unexpectedly full of NaN, check the indexes of both sides first. Nine times in ten, one of them came from a filter or a groupby and carries labels the other does not have. A reset_index(drop=True) after filtering prevents most of these problems before they start.

Need this solved on your own data?

The library is free and always will be. When the problem is bigger than an answer, we take on the work.

Start a project