How do query and eval speed up work on large DataFrames in pandas?

The short version

query filters rows and eval computes new columns using a string expression. On large frames these can save memory and run faster, because pandas evaluates the whole expression at once rather than building a separate temporary array for each sub-part. On small frames the parsing overhead outweighs the benefit, so they are a large-data tool, not a default.

query for filtering

import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3], "b": [10, 20, 30]})

print(df.query("a > 1 and b < 30"))

   a   b
1  2  20

Inside the string you use plain and, or and column names directly, which many people find more readable than the bracket-heavy boolean form df[(df["a"] > 1) & (df["b"] < 30)].

eval for computing

eval builds a new column from an expression.

df.eval("c = a + b", inplace=False)   # adds column c

For a chain of arithmetic on several columns, eval computes the result in one pass instead of creating an intermediate array for each operation, which is where the memory saving comes from on large frames.

Why it can be faster on big data

Ordinary pandas arithmetic like df["a"] + df["b"] + df["c"] builds a temporary array for a + b, then another for the final sum. On a frame with millions of rows, those temporaries cost real memory and time. eval can evaluate the whole expression together and avoid some of that, especially with the optional numexpr library installed.

The catch on small frames

For a few thousand rows, the cost of parsing the string is larger than any saving, so query and eval are actually slower than the normal syntax. Do not reach for them to speed up small data; use them when the frame is large and the expression is non-trivial. Always measure rather than assume.

Referencing outside variables

Inside query, prefix a Python variable with @ to use it, which keeps the distinction between column names and local variables clear.

threshold = 15
df.query("b > @threshold")

Without the @, pandas would look for a column called threshold and raise if it does not exist.

Readability as a reason on its own

Even setting performance aside, some teams prefer query purely because a long filter reads better as a sentence than as a stack of bracketed conditions. That is a legitimate reason to use it, but be aware the string is not checked by your editor, so a typo in a column name only surfaces at run time.