How do I read a CSV that is too big to fit in memory with pandas?

The short version

Pass chunksize to read_csv and it returns an iterator of smaller DataFrames instead of one huge frame. Process each chunk, keep only the running summary you need, and combine those small results at the end. This keeps memory flat regardless of file size, because only one chunk is in memory at a time.

The pattern

import pandas as pd

total = 0
for chunk in pd.read_csv("orders.csv", chunksize=100_000):
    total += chunk["amount"].sum()

print(total)

Each chunk is an ordinary DataFrame of up to 100,000 rows. You do your work on it, add its contribution to a running result, and let it be discarded before the next chunk loads.

The mistake that defeats the point

The common error is collecting every chunk into a list and concatenating at the end.

# defeats the purpose: rebuilds the whole file in memory
frames = [chunk for chunk in pd.read_csv("orders.csv", chunksize=100_000)]
big = pd.concat(frames)

This holds every chunk at once, so you are back to loading the entire file. Chunking only helps if each chunk is reduced to something small (a sum, a count, a filtered subset) before the next one arrives.

Aggregating across chunks correctly

For a per-group total, accumulate partial group sums and combine them once at the end.

parts = []
for chunk in pd.read_csv("orders.csv", chunksize=100_000):
    parts.append(chunk.groupby("city")["amount"].sum())

result = pd.concat(parts).groupby(level=0).sum()

Here each parts entry is one small per-city total, not a full chunk, so memory stays low. The final groupby(level=0).sum() merges the partial totals into one.

Filtering while reading

If you only need some rows, filter inside the loop so the unwanted rows never accumulate.

wanted = []
for chunk in pd.read_csv("orders.csv", chunksize=100_000):
    wanted.append(chunk[chunk["city"] == "Leeds"])
subset = pd.concat(wanted)

This is safe as long as the surviving subset genuinely fits in memory.

Reduce each chunk's footprint too

Combine chunking with narrow dtypes for the biggest saving, passing dtype and usecols so each chunk is small in the first place.

pd.read_csv("orders.csv", chunksize=100_000,
            usecols=["city", "amount"], dtype={"city": "category"})

When to reach for other tools

If even the summarised result is large, or you are doing this often, a tool built for out-of-core work (such as a database, or a library like Dask or Polars) may fit better. Chunking is the simplest first step, and it is enough for many one-off jobs on files larger than memory.