How do I run the same analysis over many files and combine the results?

The short version

Use Path.glob to find the files, read each one, add a column recording which file it came from, collect the frames in a list, and concat once at the end. The source column is the part people forget, and it is what saves you when one file turns out to be malformed.

The pattern

from pathlib import Path
import pandas as pd

frames = []
for path in sorted(Path("data").glob("*.csv")):
    frame = pd.read_csv(path)
    frame["source_file"] = path.name      # tag it
    frames.append(frame)

combined = pd.concat(frames, ignore_index=True)

sorted makes the order deterministic, which matters for reproducibility. ignore_index=True renumbers the result so you do not inherit repeated index labels from each file.

Why concat once, not in the loop

Concatenating inside the loop rebuilds the entire accumulated frame on every iteration, which makes the work grow with the square of the number of files. Collecting and combining once is a single pass, and on a hundred files the difference is dramatic.

Why tag the source

When a total looks wrong, or a column has unexpected missing values, the first question is always "which file did that come from". Without a source column, you have no way to answer it, and you end up rerunning the whole thing file by file to find the culprit.

combined.groupby("source_file").size()          # row count per file
combined.groupby("source_file")["amount"].sum() # totals per file

An outlier here immediately identifies the odd file, which is usually where the problem is.

Guard against the schema drifting

Files that are supposed to be identical often are not: one has an extra column, another spells a header differently. concat aligns on column names and fills the gaps with NaN, so this passes silently.

combined.isna().mean()      # a column that is mostly NaN reveals a mismatch

A better guard is to check each file's columns as you read it, and fail loudly rather than merging a misshapen file into the pile.

expected = {"city", "amount", "date"}
missing = expected - set(frame.columns)
if missing:
    raise ValueError(f"{path.name} is missing columns: {missing}")

Handling a file that fails

If one bad file should not stop the whole run, catch the error, record it, and carry on, but report what was skipped rather than swallowing it.

skipped = []
for path in paths:
    try:
        ...
    except Exception as err:
        skipped.append((path.name, str(err)))

if skipped:
    print(f"skipped {len(skipped)} files:", skipped)

Silently ignoring failures is how a report ends up quietly missing a month of data.

When there are very many files

If the combined data will not fit in memory, summarise each file as you go and combine the small summaries instead, which keeps the footprint flat regardless of how many files there are.