What is a generator in Python and when should I use one?

The short version

A generator is a function that uses yield to produce values one at a time, pausing between each, rather than building and returning a full list. Because it holds only one value at a time, it uses almost no memory regardless of how many values it produces. Use generators for large or streaming data, or when the caller might stop early.

A generator function

def squares(n):
    for i in range(n):
        yield i * i

for value in squares(4):
    print(value)   # 0, 1, 4, 9

Each yield hands back one value and freezes the function there. The next iteration resumes right after the yield, keeping the loop's state.

The memory difference

Compare building a list against yielding.

# builds the entire list in memory
def squares_list(n):
    return [i * i for i in range(n)]

# produces one value at a time
def squares_gen(n):
    for i in range(n):
        yield i * i

For n of ten million, the list version allocates ten million numbers at once, while the generator holds one. If you only loop through them, the generator is far lighter.

Generator expressions

For simple cases you do not need a function. A generator expression looks like a list comprehension with parentheses instead of brackets.

total = sum(i * i for i in range(1000))

This computes the sum without ever building the list of squares, which is the idiomatic way to feed a large sequence into sum, max or any.

The one-shot catch

A generator is exhausted once you have iterated it. Looping again yields nothing, because there is nothing left.

g = squares(3)
list(g)   # [0, 1, 4]
list(g)   # []  <- already used up

If you need the values more than once, either build a list or recreate the generator. This surprises people who expect it to behave like a list.

When not to bother

If the data is small and you need random access, indexing, or to iterate several times, a plain list is simpler and the memory saving is irrelevant. Generators earn their place when the data is large, potentially infinite, or produced lazily from a stream, and when you consume it in a single pass.