How do I use sets in Python to compare collections?

The short version

A set holds unique items and supports very fast membership tests and comparisons: & for what is in both, - for what is only in the first, | for everything. Use sets to answer "what is missing", "what do they share" and "is this present", all of which are slow and clumsy with lists.

The comparison operators

expected = {"pen", "mug", "cup"}
actual = {"pen", "mug", "pad"}

print(expected - actual)   # {'cup'}  missing from actual
print(actual - expected)   # {'pad'}  unexpected extras
print(expected & actual)   # {'pen', 'mug'}  in both
print(expected | actual)   # everything, no duplicates

These four lines answer most reconciliation questions immediately, where the list equivalent would need nested loops.

Fast membership tests

Checking x in some_list scans the list, so it gets slower as the list grows. Checking x in some_set is effectively instant regardless of size, because sets are hash-based.

big_list = list(range(1_000_000))
big_set = set(big_list)

999_999 in big_list   # scans nearly a million items
999_999 in big_set    # immediate

If you are testing membership repeatedly inside a loop, converting to a set first turns a slow quadratic algorithm into a fast linear one. This is one of the highest-value performance habits in everyday Python.

Removing duplicates

set(items) removes duplicates instantly, but it does not preserve order.

set([3, 1, 3, 2])   # {1, 2, 3}, order not guaranteed

If you need the original order preserved, use list(dict.fromkeys(items)) instead, which keeps first-seen order while still removing repeats.

The hashability requirement

Set members must be hashable, so numbers, strings and tuples work, but lists and dictionaries do not.

{[1, 2]}        # TypeError: unhashable type: 'list'
{(1, 2)}        # fine, tuples are hashable

Convert lists to tuples if you need to put them in a set, for example when deduplicating rows.

Comparing columns of a DataFrame

A practical use is checking that two datasets cover the same keys, which is the quickest way to explain a merge that lost rows.

missing = set(left["id"]) - set(right["id"])   # ids that will not match

This tells you immediately which keys are absent from the other side, before you spend time debugging the merge itself.

Subset and superset checks

<= and >= test containment, which reads clearly for validation.

required = {"city", "amount"}
required <= set(df.columns)   # True if the frame has at least those columns

This is a neat, self-documenting way to assert that a file has the columns your code depends on before you start using them.