Q&A library Python

Why does resample show zeros for periods with no data?

Python Advanced missing-dataresampletime-series

Short answer

Summing an empty period gives zero, which is indistinguishable from a real zero, so gaps in your data look like genuine lows. Use asfreq or a count to tell the difference between no data and no activity.

The short version

resample(...).sum() creates a row for every period in the range, and an empty period sums to zero. That zero looks exactly like a genuine zero, so missing data becomes an apparent drop to nothing. Check with a count, or use asfreq, to distinguish "no data" from "no activity".

Seeing the problem

import pandas as pd

s = pd.Series([1, 2], index=pd.to_datetime(["2021-01-01", "2021-01-03"]))

s.resample("D").sum().tolist()   # [1, 0, 2]

There is no data at all for the 2nd of January, but the result shows a confident zero. On a chart, that is a dramatic dip to nothing, and it is entirely fictional.

Seeing the truth

s.resample("D").asfreq().tolist()   # [1.0, nan, 2.0]

asfreq does not aggregate, so the empty period shows as NaN, which is the honest answer: there was no observation.

You can also count the underlying rows.

s.resample("D").count().tolist()    # [1, 0, 1]

A count of zero tells you the period was empty, which the sum alone cannot.

Why this matters

The two situations are completely different, and the sum conflates them.

A genuine zero means "we measured, and there was no activity". A missing period means "we did not measure". Treating the second as the first invents data. It drags averages down, creates false troughs in charts, and can make a monitoring gap look like a business collapse.

The safe pattern

Aggregate, then mark the empty periods explicitly using the count.

agg = s.resample("D").agg(["sum", "count"])
agg.loc[agg["count"] == 0, "sum"] = pd.NA

Now the empty periods are missing rather than zero, and any chart or average handles them correctly instead of silently including a fabricated zero.

Decide what a gap means

Sometimes a gap genuinely does mean zero: if you record every sale, then a day with no rows really was a day with no sales, and zero is correct. If you only receive data when a system is up, a gap means the system was down, and zero is a lie.

Only you know which applies. The point is that resample cannot know, so it defaults to zero and you must decide whether that default is right for your data.

Mean is affected too

mean over an empty period gives NaN rather than zero, which is the honest answer, so mean does not suffer this problem. It is specifically sum and count (which return zero for an empty group) that create the false zeros. That inconsistency is worth remembering when you switch aggregation.

Need this solved on your own data?

The library is free and always will be. When the problem is bigger than an answer, we take on the work.

Start a project