How do I find what is using memory in my Python program?

The short version

Use tracemalloc, which is built into Python, to record where memory was allocated and report the biggest consumers by line. As with speed, measure before optimising, because the object using the most memory is frequently not the one you assumed.

Taking a snapshot

import tracemalloc

tracemalloc.start()

# ... run the code you want to measure ...

snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics("lineno")[:5]:
    print(stat)

tracemalloc.stop()

This prints the five lines that allocated the most memory, with the size and the number of allocations. It points you straight at the source rather than leaving you to guess.

The peak matters as much as the total

Current usage tells you what is held now; the peak tells you the most that was ever held at once, which is what determines whether you run out of memory.

current, peak = tracemalloc.get_traced_memory()
print(f"current {current / 1e6:.1f} MB, peak {peak / 1e6:.1f} MB")

A pipeline can end with a small result while having briefly held an enormous intermediate. That transient peak is what kills the process, and only the peak measurement reveals it.

Comparing two points

The most useful technique is diffing two snapshots to see what grew between them.

before = tracemalloc.take_snapshot()
# ... the suspicious operation ...
after = tracemalloc.take_snapshot()

for stat in after.compare_to(before, "lineno")[:5]:
    print(stat)

This isolates the memory attributable to one step, which is far more informative than a total for the whole program.

For DataFrames specifically

tracemalloc tracks Python allocations, and pandas holds much of its data in NumPy buffers, so the accounting can be less direct than you expect. For frames, measure directly.

df.memory_usage(deep=True).sum()

deep=True is essential, or text columns are drastically under-reported. Use this per frame, and tracemalloc for the surrounding Python objects.

The usual culprits

In data work, the memory is nearly always in one of four places: a frame read with default 64-bit dtypes when 16 would do, an object column of strings that should be category, several intermediate frames still referenced by variables, or a list accumulating results that should have been reduced as it went.

Checking those four before profiling will often find the problem in a minute.

Notebooks hold on to everything

Jupyter keeps a reference to every cell's output, so a frame you displayed is still alive even after you rebind the name. If memory only ever grows and never falls, this is usually why. Restarting the kernel is the reliable cure.

Line-by-line tools

For a finer view, the third-party memory_profiler package annotates a function line by line with its memory use, which is useful when one function is clearly the problem and you need to know exactly where inside it. Start with tracemalloc to find the function, then reach for a line profiler only if you need to.