How do I validate a DataFrame before trusting it?

The short version

Write explicit checks on the data as it enters your pipeline: the columns you need, their types, plausible ranges, and key uniqueness. Fail loudly when an assumption is violated. Every hour spent here saves several tracing a wrong number back through five transformations.

A basic validation function

import pandas as pd

def validate_orders(df: pd.DataFrame) -> pd.DataFrame:
    required = {"order_id", "city", "amount", "order_date"}
    missing = required - set(df.columns)
    if missing:
        raise ValueError(f"missing required columns: {sorted(missing)}")

    if df["order_id"].duplicated().any():
        raise ValueError("order_id is not unique")

    if (df["amount"] < 0).any():
        raise ValueError("negative amounts found")

    if df["amount"].isna().mean() > 0.05:
        raise ValueError("more than 5 per cent of amounts are missing")

    return df

Call it immediately after loading, and every later step can rely on what it guarantees.

df = validate_orders(pd.read_csv("orders.csv"))

What is worth checking

Four things cover most real failures.

Schema. The columns you depend on exist, and have the types you expect. A date column that arrived as text will sort alphabetically and silently corrupt every time-based result.

Uniqueness. A key that should be unique is unique. This is the single most valuable check, because a duplicated key silently multiplies rows in a later merge and inflates every total.

Ranges. Values fall within possible bounds. Negative quantities, percentages above 100, dates in the future: each is a signal the data is not what you assumed.

Missingness. The proportion of gaps is within tolerance. A column that is suddenly 60 per cent empty usually means an upstream change, not a real fall in reporting.

Fail loudly, not quietly

The temptation is to clean up quietly and carry on. Resist it for anything that indicates the data is not what you expected.

A silently dropped row is a silently wrong total. If bad rows are expected and acceptable, remove them explicitly and record how many, so the loss is visible.

bad = df["amount"] < 0
if bad.any():
    logger.warning("dropping %s rows with negative amounts", bad.sum())
    df = df[~bad]

Check again after transformations

Validation is not only for input. After a merge, assert the row count is what you expected. After an aggregation, reconcile the total against the source.

assert len(merged) == len(orders), "merge changed the row count"
assert abs(summary["amount"].sum() - orders["amount"].sum()) < 0.01

These two assertions catch the two most common silent errors in data work: a merge that multiplied rows, and a groupby that dropped rows with missing keys.

Libraries that formalise this

Tools such as Pandera and Great Expectations let you declare a schema and expectations in one place and validate against it, producing readable reports of what failed. They are worth adopting once your hand-written checks start to sprawl, but the thinking is the same, and starting with a plain function costs nothing.

The principle

Data pipelines fail silently. Code that crashes tells you something is wrong; data that is merely wrong does not. Explicit validation is how you convert silent wrongness into a loud, immediate, locatable failure, which is the whole point.