How do I write log messages that are actually useful later?

The short version

Log the values you will need to diagnose a failure, not just that something happened. Use levels consistently so you can filter. Log at the boundaries (reading data, writing results, calling a service) rather than everywhere. A message without context is nearly useless when you are reading it months later.

Context is everything

import logging
logger = logging.getLogger(__name__)

# nearly useless
logger.info("loaded data")

# useful
logger.info("loaded %s rows from %s", len(df), path)

The second tells you how much data arrived and where from, which is exactly what you need when a total looks wrong and you are trying to work out whether the input changed.

Note the %s placeholders with arguments, rather than an f-string. The formatting is then only done if the message is actually emitted, which saves work for debug messages that are filtered out.

Use the levels to mean something

Pick a convention and hold to it. A workable one:

DEBUG for detail useful only when investigating a problem. INFO for the normal milestones of a run (started, loaded, wrote output). WARNING for something unexpected that the code handled (a file was missing and skipped, a value was coerced). ERROR for a failure that stopped something happening. CRITICAL for a failure that stopped everything.

The value of this is that you can run at INFO normally and drop to DEBUG when hunting a bug, without changing any code.

Log the decisions, not the mechanics

The most valuable log lines record decisions your code made that a reader could not otherwise reconstruct.

logger.warning("dropped %s rows with missing city", dropped)
logger.info("filled %s missing amounts with the group mean", filled)

When a number is questioned six months later, these lines are the difference between explaining it in a minute and re-deriving the whole pipeline.

Log exceptions with the traceback

try:
    process(path)
except Exception:
    logger.exception("failed to process %s", path)
    raise

logger.exception records the full traceback automatically. Re-raising afterwards means you have both a record and a genuine failure, rather than a swallowed error.

Do not log secrets or personal data

An obvious rule that is broken constantly. Log identifiers, not the contents of records. A log file is often less well protected than the database it describes, and it frequently ends up somewhere unexpected.

Structure helps at scale

For anything running unattended, logging structured records (key-value pairs, or JSON) rather than prose makes the logs searchable and aggregatable.

logger.info("rows_loaded", extra={"rows": len(df), "source": str(path)})

With a formatter that emits JSON, you can then query "every run where rows_loaded was below the usual level", which prose logs cannot support.

The test of a good log

Ask: if this fails at three in the morning and I read only the logs, could I tell what happened, to which input, and where? If not, the messages need more context. That single question is a better guide than any list of rules.