Why is apply with axis=1 so slow in pandas, and what should I use instead?

The short version

apply(axis=1) calls a Python function once per row, building a Series for each one. That per-row overhead makes it hundreds of times slower than the same logic written as an operation on whole columns. Replace it with column arithmetic, np.where or np.select whenever the logic can be expressed that way.

The scale of the difference

On a frame of 200,000 rows, adding two columns row by row took over a second, while the vectorised version took under a millisecond. That is roughly a thousandfold difference for the identical result.

import pandas as pd
import numpy as np

df = pd.DataFrame({"a": np.arange(200_000), "b": np.arange(200_000)})

# slow: a Python call per row
df.apply(lambda r: r["a"] + r["b"], axis=1)

# fast: one vectorised operation
df["a"] + df["b"]

Both give the same answer. Only one of them scales.

Why it is so slow

A vectorised expression pushes the whole column into compiled code that loops in C. apply(axis=1) instead constructs a Series object for every row, calls back into the Python interpreter, and reassembles the results. You pay the interpreter's overhead once per row, and object creation dominates the actual arithmetic.

Replacing conditional logic

Most apply(axis=1) calls in real code are conditionals, and these have direct vectorised equivalents.

# instead of apply with an if
df["band"] = np.where(df["a"] > 100, "big", "small")

# for several conditions
conditions = [df["a"] < 50, df["a"] < 100]
df["band"] = np.select(conditions, ["low", "mid"], default="high")

np.where handles two outcomes and np.select handles many, both across the whole column at once.

When you cannot vectorise

Sometimes the logic genuinely resists vectorisation, for example a call to an external service per row. Then prefer itertuples, which is lighter than apply(axis=1) because it yields simple named tuples rather than building a Series each time.

results = [expensive(r.a, r.b) for r in df.itertuples(index=False)]
df["result"] = results

The honest caveat about Series.apply

Series.apply (on a single column) is not as bad as apply(axis=1), because it does not build a row Series each time. But it still runs Python per element, so a built-in .str method, a map with a dictionary, or plain arithmetic will beat it. Use it when no vectorised equivalent exists, not as a first reach.

The habit to build

Before writing apply, ask: can this be arithmetic on columns? A np.where or np.select? A map from a dictionary? A merge against a lookup table? One of those covers the overwhelming majority of cases, and each is both faster and clearer than a per-row function.