Should I use Parquet instead of CSV in pandas?

The short version

Parquet is a compressed, columnar, typed format. It produces much smaller files than CSV, reads back faster, preserves dtypes (dates stay dates), and lets you read a subset of columns without touching the rest. Prefer it for any data passed between steps of your own pipeline. Keep CSV for when a person or a foreign tool needs to open the file.

Writing and reading

import pandas as pd

df.to_parquet("orders.parquet")
back = pd.read_parquet("orders.parquet")

You need an engine installed, normally pyarrow. Pandas picks it up automatically.

Dtypes survive the round trip

This is the biggest practical win. CSV is just text, so every type must be guessed again on reading.

df.to_csv("orders.csv", index=False)
pd.read_csv("orders.csv")["when"].dtype       # object, a plain string
pd.read_parquet("orders.parquet")["when"].dtype  # datetime64, still a date

With CSV you must re-specify parse_dates and dtype on every read, and a forgotten one silently gives you strings where you expected dates, or floats where you expected integers. Parquet stores the schema with the data, so this whole class of bug disappears.

Smaller files

Parquet compresses by column, and columns of similar values compress extremely well. In a simple test, a frame that took about 300 KB as CSV took roughly 145 KB as Parquet, and the gap widens with repetitive text columns. Less disk means less to read, which is much of the speed benefit.

Reading only the columns you need

Because Parquet is columnar, you can read a few columns without parsing the others at all.

pd.read_parquet("orders.parquet", columns=["amount"])

On a wide table this is dramatically faster and lighter than CSV, where the reader must scan every line in full even to get one column.

When CSV is still the right choice

CSV is universal and human-readable. Use it when a person will open the file in a spreadsheet, when another system requires it, or when you need to eyeball or diff the raw text. It is a poor choice for intermediate storage inside your own pipeline, where nobody reads the raw file and the type-guessing is pure cost.

A practical pattern

Read the messy CSV once, clean and type it properly, then write Parquet and use that from then on.

df = pd.read_csv("raw.csv", parse_dates=["when"], dtype={"code": "string"})
df.to_parquet("clean.parquet")   # every later step reads this

You pay the slow, careful parse once instead of on every run.

A note on compression and appending

Parquet supports several compression codecs (the default is usually fine). It is not designed for appending row by row, so write it as whole files, or as a directory of files split by date or category, which readers can then treat as one dataset and filter efficiently.