Q&A library Python

How do I parse ISO timestamps with time zone offsets in pandas?

Python Advanced datetimeiso8601utc

Short answer

Pass utc=True to to_datetime so mixed offsets are normalised to a single timezone-aware column. Without it, mixed offsets produce an object column of separate timestamps that behaves badly.

The short version

Use pd.to_datetime(..., utc=True) when your timestamps carry time zone offsets. This converts everything to UTC and gives you a single, properly typed, timezone-aware column. Without utc=True, a column with mixed offsets falls back to a slow object column of individual timestamps, which loses the .dt methods and behaves badly.

Parsing with an offset

import pandas as pd

s = pd.Series(["2021-01-01 00:00+01:00", "2021-01-01 00:00+00:00"])

parsed = pd.to_datetime(s, utc=True)
print(parsed[0])   # 2020-12-31 23:00:00+00:00

The first timestamp was one hour ahead of UTC, so in UTC it is the previous day at 23:00. Both values are now on the same clock and can be compared and sorted correctly.

Why mixed offsets are a problem without utc

If different rows carry different offsets and you do not pass utc=True, pandas cannot represent them all in one timezone-aware column, so it produces an object column holding separate Timestamp objects. That column looks fine but is slow, loses vectorised .dt operations, and compares unreliably. In recent pandas you also get a warning telling you to pass utc=True.

The fix is always the same: normalise to UTC at the point of parsing.

The ISO8601 format hint

If every value is genuinely ISO 8601, telling pandas so makes parsing faster and stricter.

pd.to_datetime(s, format="ISO8601", utc=True)

This avoids the inference step and rejects values that do not conform, which is a useful check on your data rather than a silent best guess.

The Z suffix

A trailing Z means UTC (Zulu time) and is parsed correctly.

pd.to_datetime("2021-03-15T10:30:00Z")   # 2021-03-15 10:30:00+00:00

Converting for display

Store in UTC, convert to a local zone only when showing a person.

parsed.dt.tz_convert("Europe/London")

This is the discipline that avoids the whole family of daylight-saving arithmetic bugs: one unambiguous internal representation, local zones only at the edges.

Naive and aware do not mix

Once a column is timezone-aware, you cannot compare it with a naive timestamp, and pandas raises rather than guess.

parsed > pd.Timestamp("2021-01-01")            # TypeError
parsed > pd.Timestamp("2021-01-01", tz="UTC")  # correct

This error is a good thing: it prevents a silent comparison between times that mean different instants. When you hit it, the fix is to make your comparison value aware, not to strip the zone from your data.

Stripping the zone

If a downstream tool cannot cope with aware timestamps, remove the zone deliberately after converting, so you know exactly what the values mean.

parsed.dt.tz_convert("Europe/London").dt.tz_localize(None)

Convert first, then drop the zone. Dropping it without converting leaves you with UTC values labelled as if they were local, which is a real and easily missed error.

Need this solved on your own data?

The library is free and always will be. When the problem is bigger than an answer, we take on the work.

Start a project