How do I write reusable transformation steps for pandas pipelines?

The short version

Write each step as a function that takes a DataFrame and returns a new DataFrame, then compose them with pipe. Each step becomes independently testable and reusable, and the pipeline reads as a list of named operations rather than a long script of ad hoc mutations.

A reusable step

import pandas as pd

def add_tax(df, rate):
    return df.assign(taxed=df["amount"] * (1 + rate))

orders = pd.DataFrame({"city": ["Leeds", "Bath"], "amount": [1, 2]})

result = orders.pipe(add_tax, rate=0.2)
print(result["taxed"].tolist())   # [1.2, 2.4]

df.pipe(add_tax, rate=0.2) is exactly the same as add_tax(df, rate=0.2), but it reads left to right in a chain rather than nesting calls inside out.

Composing several steps

clean = (
    orders
    .pipe(normalise_names)
    .pipe(add_tax, rate=0.2)
    .pipe(drop_small, minimum=1.0)
)

Read top to bottom, this is a description of the pipeline. Compare that with the usual alternative, a script of thirty lines that reassigns df over and over, where you cannot tell which steps depend on which, and nothing can be tested in isolation.

Why functions, not just a chain of methods

A method chain with assign and loc is fine for a few steps. Functions become worth it when a step is reused across scripts, needs a docstring, or is complex enough to deserve a test.

def test_add_tax():
    given = pd.DataFrame({"amount": [10.0]})
    assert add_tax(given, rate=0.1)["taxed"].tolist() == [11.0]

Now the logic is verified once, and any pipeline that uses it inherits that confidence.

The rules that make steps composable

Take a DataFrame as the first argument and return a new DataFrame. Do not modify the input in place, because a step with hidden side effects breaks when reordered or reused. Use assign rather than direct column assignment inside the function, so the input is left untouched.

# good: returns a new frame
def add_tax(df, rate):
    return df.assign(taxed=df["amount"] * (1 + rate))

# risky: mutates the caller's frame
def add_tax_bad(df, rate):
    df["taxed"] = df["amount"] * (1 + rate)
    return df

The second version quietly changes the frame the caller passed in, which causes bugs that are painful to track down.

Keep steps small and named

A good step does one nameable thing: normalise the column names, drop invalid rows, add a derived column. If you cannot give the function a short honest name, it is doing too much. Small named steps also make the pipeline self-documenting, which is worth as much as the reuse.

The pay-off

This is how a one-off analysis script becomes something a team can maintain: each transformation is a named, tested unit, and the pipeline that assembles them is short enough to read at a glance and easy to reorder or extend.