How do I process a large file without loading it into memory in Python?

The short version

Iterate over the file object directly, and use generator expressions to filter and transform, so only one line is in memory at any moment. Never call readlines() or build a list of all the lines, because that loads the whole file, which is exactly what you were trying to avoid.

The streaming pattern

def read_amounts(path):
    with open(path) as f:
        for line in f:
            parts = line.strip().split(",")
            yield float(parts[1])

total = sum(read_amounts("orders.csv"))

Each line is read, used and discarded. Memory stays flat whether the file is one megabyte or one hundred gigabytes.

What defeats it

with open(path) as f:
    lines = f.readlines()        # the whole file, in memory, at once

This is the mistake. So is any comprehension that builds a full list of the results before you use them.

amounts = [float(l.split(",")[1]) for l in open(path)]   # builds a huge list

Change the brackets to parentheses and it becomes a generator that produces values on demand instead.

amounts = (float(l.split(",")[1]) for l in open(path))   # streams

Chaining generators

Generators compose, so you can build a pipeline where each stage streams into the next and nothing accumulates.

lines = (line.strip() for line in open("orders.csv"))
rows = (line.split(",") for line in lines if line)
amounts = (float(r[1]) for r in rows if r[1])

print(sum(amounts))

Nothing runs until sum starts pulling values, and at any instant only one line is held. This is a genuinely powerful pattern for large data.

The functions that consume lazily

sum, max, min, any and all all consume an iterable one item at a time, so they work perfectly with generators and never build a list. That is why sum(x for x in ...) is the idiomatic way to total a large sequence.

The one-pass limitation

A generator is exhausted after one iteration. If you need two passes (say, a total and then a mean), either compute both in one pass, or reread the file. Trying to iterate a spent generator silently gives you nothing, which is a confusing bug.

total = count = 0
for amount in read_amounts(path):
    total += amount
    count += 1                     # both statistics in one pass

For tabular data, use chunks

For a CSV you are analysing rather than scanning, pandas gives you the same benefit with a friendlier interface.

import pandas as pd
for chunk in pd.read_csv("orders.csv", chunksize=100_000):
    ...

Same principle, same rule: reduce each chunk to a small summary rather than collecting them all, or you have rebuilt the whole file in memory and gained nothing.