Why does my pandas script use so much more memory than the file size?

The short version

A 1 GB CSV can easily occupy several gigabytes as a DataFrame, because text becomes Python string objects, numbers default to 64-bit, and every intermediate frame you keep is still in memory. Read with narrow dtypes, use category for repetitive text, and stop holding on to copies you have finished with.

Why the expansion happens

Three causes, in rough order of importance.

Text columns stored as object hold pointers to separate Python strings, which carry substantial per-item overhead. A short code that occupies five bytes in the file can take fifty or more in memory.

Numbers default to int64 and float64 even when the values would fit in eight or sixteen bits.

And every intermediate frame lives until nothing references it. A chain of five transformations, each assigned to its own variable, can hold five copies of nearly the same data.

Measure honestly

df.memory_usage(deep=True).sum()

deep=True is essential, because without it, text columns report only the size of their pointers and you will optimise the wrong thing.

Read it small in the first place

The cheapest fix is not to build the oversized frame at all.

import pandas as pd

df = pd.read_csv(
    "orders.csv",
    usecols=["city", "amount", "when"],   # skip columns you never use
    dtype={"city": "category", "amount": "int32"},
    parse_dates=["when"],
)

usecols is often the single biggest win: a wide file where you need four columns should not cost you the other ninety-six.

Category for repetitive text

If a text column has few distinct values relative to its length, category stores each label once and keeps small integer codes per row. On a column of city names repeated across a million rows, this routinely cuts the column's footprint by an order of magnitude.

df["city"] = df["city"].astype("category")

Check the cardinality first with nunique, because on a nearly unique column (an identifier) it does not help.

Release what you no longer need

del raw            # drop the reference
import gc; gc.collect()

More practically, avoid creating the intermediates in the first place by chaining, so no name holds on to a stage you have finished with.

result = (
    pd.read_csv("orders.csv")
    .query("amount > 0")
    .assign(with_vat=lambda d: d["amount"] * 1.2)
)

Only the final frame is kept; the intermediate stages are released as soon as the next step consumes them.

In a notebook, the output cache holds on to things

Jupyter keeps a reference to the output of every cell in Out, so a frame you displayed is still in memory even after you rebind the name. This surprises people whose memory never falls. Restarting the kernel is the reliable cure, which is another reason to restart and run all regularly.

When to change tool

If the data genuinely does not fit even after these measures, chunked reading, Parquet with column selection, or a library built for out-of-core work (Polars, Dask, DuckDB) is the honest answer, rather than fighting to squeeze it into memory.