How do I compare the columns or keys of two DataFrames?

The short version

A pandas Index behaves like a set, so intersection, difference and union work directly on it. This is the fastest way to compare two frames' columns, or to find which keys exist on one side of a join and not the other.

The operations

import pandas as pd

a = pd.Index([1, 2, 3])
b = pd.Index([2, 3, 4])

a.intersection(b).tolist()   # [2, 3] , in both
a.difference(b).tolist()     # [1] , in a only
b.difference(a).tolist()     # [4] , in b only
a.union(b).tolist()          # [1, 2, 3, 4] , everything

Note that difference is directional: a.difference(b) is not the same as b.difference(a), and you usually want to look at both.

Diagnosing a merge that lost rows

This is the killer application. When a join drops rows, the question is always "which keys failed to match".

left_keys = pd.Index(orders["customer_id"].unique())
right_keys = pd.Index(customers["customer_id"].unique())

missing = left_keys.difference(right_keys)
print(f"{len(missing)} order customer ids have no matching customer")
print(missing[:10].tolist())

Thirty seconds of this tells you more than an hour of staring at the merge. And it usually reveals the real cause: a type mismatch (numbers on one side, text on the other), trailing whitespace, or a case difference.

Comparing column names

required = pd.Index(["city", "amount", "order_date"])
missing = required.difference(df.columns)

if len(missing):
    raise ValueError(f"missing columns: {missing.tolist()}")

This is a compact and readable validation check, and it names exactly what is absent rather than failing later with a KeyError on one column at a time.

The type-mismatch trap

Set operations compare by value and type, so 1 and "1" do not match.

pd.Index([1, 2]).intersection(pd.Index(["1", "2"]))   # empty!

An empty intersection where you expected a full one is nearly always this. Check the dtypes on both sides before concluding that the data genuinely does not overlap, because a key read as text on one side and as a number on the other is one of the most common causes of a merge silently matching nothing.

Plain Python sets work too

set(orders["customer_id"]) - set(customers["customer_id"])

This is equivalent and just as readable. The Index methods have the advantage of preserving order and returning an Index you can use directly in pandas operations, but for a quick check either is fine.

Symmetric difference

To find everything present on exactly one side, in one call:

a.symmetric_difference(b).tolist()   # [1, 4]

Useful for reconciling two extracts that should be identical, where any output at all indicates a discrepancy worth investigating.