When should I use pickle to save Python objects, and when should I not?

The short version

Pickle serialises almost any Python object, preserving it exactly, which makes it handy for caching your own intermediate results. But never unpickle a file from an untrusted source, because loading it can execute arbitrary code. It is also tied to Python and can break across versions, so it is a poor choice for long-term storage or for sharing data.

Using it

import pandas as pd

df.to_pickle("cache.pkl")
back = pd.read_pickle("cache.pkl")

For general objects, the standard library works the same way.

import pickle

with open("model.pkl", "wb") as f:
    pickle.dump(my_object, f)

with open("model.pkl", "rb") as f:
    my_object = pickle.load(f)

The security problem

Unpickling is not just reading data; it can construct objects and call code as part of reconstruction. A maliciously crafted pickle file can therefore run arbitrary commands the moment you load it.

pickle.load(open("downloaded.pkl", "rb"))   # can execute code. Do not do this.

The rule is absolute: only load pickles you created yourself, or that come from a source you fully trust. Never unpickle a file from the internet, a user upload, or any untrusted party. There is no safe way to unpickle hostile input.

The durability problem

A pickle is tied to Python and, often, to the exact classes and library versions that created it. Upgrade pandas or your own class definitions, and an old pickle may fail to load or, worse, load subtly wrong. This makes pickle a bad choice for archives or for anything you need to read back in a year.

What to use instead

For tabular data, use Parquet: it is fast, compressed, keeps dtypes, and is readable by many tools and languages. For configuration and simple structures, use JSON: it is text, portable and safe to read. Reserve pickle for short-lived caches inside a single project where you control both ends.

df.to_parquet("data.parquet")   # for data you keep or share

The legitimate use

Pickle earns its place when you want to save something Parquet cannot represent, such as a fitted model object or a complex nested structure, and only for your own use. Even then, treat the file as disposable: something you can regenerate by rerunning your code, not the authoritative copy of anything.

A note on saving models

Many people pickle scikit-learn models. That works, but the same caveats apply: the file is only loadable with compatible library versions, and it is unsafe from untrusted sources. If a model must persist across versions or be shared, prefer an explicit export format and record the library versions alongside it, so it can be reconstructed reliably.