How do I find and remove duplicate rows in pandas?

The short version

Use duplicated() to flag rows that repeat an earlier one, and drop_duplicates() to remove them. By default a row is a duplicate only if every column matches; pass subset to define a duplicate by specific columns. The keep argument decides which copy survives: the first, the last, or none.

Removing exact duplicate rows

import pandas as pd

df = pd.DataFrame({
    "city": ["Leeds", "Leeds", "Bath"],
    "value": [1, 1, 2],
})

print(df.drop_duplicates())

    city  value
0  Leeds      1
2   Bath      2

The second Leeds row is an exact copy of the first, so it is removed. The first occurrence is kept by default.

Defining a duplicate by some columns only

Often a row counts as a duplicate based on a key, not on every column. Use subset.

df.drop_duplicates(subset="city")

This keeps one row per city, ignoring differences in other columns. Which row it keeps depends on keep, so sort first if you care.

Choosing which copy to keep

keep="first" (the default) keeps the earliest, keep="last" keeps the latest, and keep=False drops every row that has any duplicate, leaving only the genuinely unique ones.

df.drop_duplicates(subset="city", keep="last")   # keep the most recent per city
df.drop_duplicates(keep=False)                    # remove all repeated rows entirely

The keep=False option is useful when a duplicate signals a problem and you want to isolate the rows that are truly one of a kind.

Inspecting before deleting

It is safer to look at the duplicates before removing them, so you understand what you are dropping. duplicated returns a boolean flag per row.

df[df.duplicated(subset="city", keep=False)]   # show all rows involved in a duplicate

Passing keep=False here flags every member of a duplicated group, not just the extra copies, which is what you want when reviewing.

The order trap

Because keep="first" and "last" depend on row order, the "wrong" row can survive if the frame is not sorted the way you assume. If you want to keep, say, the most recent record per key, sort by date first so that "last" really means the latest.

df.sort_values("date").drop_duplicates(subset="city", keep="last")

Counting duplicates

To measure the scale of the problem before cleaning, count the flagged rows.

df.duplicated(subset="city").sum()   # how many rows are repeats