Q&A library Python

How do I calculate the time between rows in pandas?

Python Advanced datetimedifftimedelta

Short answer

Subtract datetime columns or use diff to get a Timedelta, then convert to the unit you want with dt.total_seconds. Sort first and group so gaps do not span across groups.

The short version

Subtracting two datetime values, or calling diff on a datetime column, gives a Timedelta. Convert it to a number in your chosen unit with dt.total_seconds() (then divide) or dt.days. Sort the data by time first, and group when the gaps should not cross between groups.

The gap between consecutive rows

import pandas as pd

df = pd.DataFrame({
    "ts": pd.to_datetime([
        "2021-01-01 09:00", "2021-01-01 11:30", "2021-01-02 09:00",
    ])
})

df["gap_hours"] = df["ts"].diff().dt.total_seconds() / 3600
print(df["gap_hours"].tolist())   # [nan, 2.5, 21.5]

diff gives the Timedelta from the previous row, total_seconds() turns it into seconds, and dividing by 3600 gives hours. The first row is NaN because there is no earlier row to measure from.

Why total_seconds and not just the Timedelta

A Timedelta prints nicely but is awkward to compare or plot as a number. Converting to a plain number in a known unit (/ 3600 for hours, / 86400 for days, or dt.days for whole days) gives you something you can filter, average and chart. Choose the unit that suits the scale of your gaps.

Duration between two specific columns

If each row has a start and an end, subtract the columns directly.

df["duration_min"] = (df["end"] - df["start"]).dt.total_seconds() / 60

Both columns must be real datetimes, so convert them with pd.to_datetime first if they arrived as text.

The ordering trap

diff measures against the previous row in current order, so unsorted data gives meaningless gaps. Sort by the timestamp first.

df = df.sort_values("ts")
df["gap"] = df["ts"].diff()

Keeping gaps within groups

Across several groups, a plain diff would measure from the last row of one group to the first of the next. Group so each gap stays inside its own group, and the first row of each group correctly becomes NaN.

df["gap"] = df.sort_values(["user", "ts"]).groupby("user")["ts"].diff()

Building a Timedelta yourself

To add or compare against a fixed duration, construct one with pd.Timedelta.

df["ts"] + pd.Timedelta(hours=1)          # shift every time forward an hour
df[df["gap_hours"] > 24]                    # rows more than a day apart

This is the clean way to express thresholds like "more than a day" without hand-computing seconds.

Need this solved on your own data?

The library is free and always will be. When the problem is bigger than an answer, we take on the work.

Start a project