Q&A library Python

How do I aggregate a time series to a coarser frequency in pandas?

Python Advanced datetime-indexresampletime-series

Short answer

Set a datetime index, then use resample with a frequency such as ME or W followed by an aggregation. resample needs a datetime index, and the frequency aliases changed in recent pandas.

The short version

Put the dates in the index, then call resample("ME") (or "W", "QE", and so on) followed by an aggregation like .sum() or .mean(). resample is groupby for time: it buckets rows into calendar periods. It requires a datetime index, and the frequency codes were renamed in pandas 2.2, so ME (month end) now replaces the older M.

A worked example

import pandas as pd

idx = pd.date_range("2021-01-01", periods=60, freq="D")
daily = pd.Series(range(60), index=idx)

monthly = daily.resample("ME").sum()
print(monthly)

This rolls sixty daily values up into monthly totals. Each output row is stamped with the end of its month and holds the sum of that month's days.

The datetime index requirement

resample works on the index, so the dates must be in the index and be a real datetime type, not text. If you get TypeError: Only valid with DatetimeIndex, set the index first.

df = df.set_index(pd.to_datetime(df["date"]))
df.resample("W")["value"].mean()

Alternatively, keep dates in a column and pass on="date" to resample, which avoids changing the index.

The renamed frequency codes

In pandas 2.2 and later, several aliases gained explicit "end" and "start" forms. Month is now ME (month end) or MS (month start); the bare M is deprecated and warns. Quarter is QE or QS, and year is YE or YS. If you see a FutureWarning about the alias, switch to the new form. On older pandas, the short codes still work, so match the alias to your version.

resample versus groupby on a date part

You could group by df["date"].dt.month, but that mixes different years into the same bucket, because it only looks at the month number. resample respects the full date, so January 2021 and January 2022 stay separate. Use resample for true time bucketing and the dt part only when you deliberately want to combine across years.

Filling gaps

resample can also go the other way, to a finer frequency, which creates empty rows you then fill. For coarser aggregation, note that periods with no data appear with a default (often zero for sum), so check whether an empty bucket should really be zero or should be dropped.

Need this solved on your own data?

The library is free and always will be. When the problem is bigger than an answer, we take on the work.

Start a project