How do I fix a UnicodeDecodeError when reading a file?

The short version

A UnicodeDecodeError means the bytes in the file do not form valid text in the encoding you assumed (normally UTF-8). Pass the correct encoding argument. Do not reach reflexively for latin-1: it never raises, so it silently turns a wrong guess into corrupted text.

The error

import pandas as pd

pd.read_csv("orders.csv")
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 ...

The byte 0xe9 is not valid UTF-8. The file is in some other encoding, and Python is refusing to guess.

The dangerous "fix"

pd.read_csv("orders.csv", encoding="latin-1")   # always "works"

latin-1 maps every possible byte to some character, so it can never fail. That sounds convenient and is actually the problem: if the file is really UTF-8, latin-1 will decode it into visible nonsense rather than raising.

"café".encode("utf-8").decode("latin-1")   # 'café'

This is mojibake. No error, no warning, just quietly corrupted text that then flows into your joins, your groupings and your reports. A city name with an accent becomes a different string, so it fails to match its counterpart in another table, and you lose rows in a merge for reasons that look inexplicable.

Find the real encoding

The common culprits are Windows exports.

pd.read_csv("orders.csv", encoding="cp1252")     # Windows Western Europe
pd.read_csv("orders.csv", encoding="utf-8-sig")  # UTF-8 with a byte order mark

utf-8-sig is worth knowing: Excel often writes a byte order mark at the start of a UTF-8 file, which appears as a stray character on the first column name (city). Using utf-8-sig strips it correctly.

If you do not know, detect it rather than guess.

import chardet
with open("orders.csv", "rb") as f:
    print(chardet.detect(f.read(100_000)))

This reads the raw bytes and reports a best guess with a confidence score, which is a far better starting point than trial and error.

Ask the source

The most reliable fix is not technical. Ask whoever produces the file what encoding it is in, and ideally ask them to export UTF-8. Guessing an encoding is always a fallback, and a file that arrives in an unpredictable encoding will do so again next month.

Checking you got it right

After reading, look at the values that would show the problem: names with accents, currency symbols, anything non-English.

df["city"].unique()[:20]

If you see é, ’ or ü, the encoding is wrong and you have mojibake, whatever the code did or did not raise.

Handling a few bad bytes

If a file is genuinely UTF-8 apart from a handful of corrupt bytes, you can skip them rather than fail entirely.

pd.read_csv("orders.csv", encoding="utf-8", encoding_errors="replace")

This substitutes a replacement character for the bad bytes. Use it knowingly: you are choosing to accept a small amount of corruption in exchange for the file loading, and you should check how many rows are affected rather than assume it is none.