Q&A library Python

How do I convert a column to a different type safely in pandas?

Python Intermediate astypedtypesto-numeric

Short answer

astype raises on any value it cannot convert, while to_numeric with errors="coerce" turns the bad ones into NaN so you can inspect them. Convert deliberately and check what failed.

The short version

astype converts a column but raises on the first value it cannot handle. pd.to_numeric(..., errors="coerce") turns unconvertible values into NaN instead, so the job continues and you can see exactly which values were bad. Use to_numeric for messy input, and astype only when you are confident every value is clean.

astype is strict

import pandas as pd

pd.Series(["a"]).astype(int)
# ValueError: invalid literal for int() with base 10: 'a'

The exception tells you something is wrong but not how much, and it stops the whole conversion at the first offender.

to_numeric is diagnostic

s = pd.Series(["1", "2", "bad"])
numbers = pd.to_numeric(s, errors="coerce")

print(numbers.tolist())        # [1.0, 2.0, nan]
print(s[numbers.isna()].tolist())  # ['bad'] , the values that failed

This converts what it can, flags what it cannot, and lets you inspect the failures. On a real file this is far more useful, because it tells you both the scale and the nature of the problem.

The float side effect

Notice the result is floats, not integers. NaN only exists for floats, so the presence of a missing value forces the column to float. If you need genuine integers, use the nullable type after coercing.

pd.to_numeric(s, errors="coerce").astype("Int64")   # capital I, keeps integers with NA

Downcasting to save memory

to_numeric can also choose the smallest sufficient type, which is a cheap memory win on large frames.

pd.to_numeric(pd.Series([1, 2, 3]), downcast="integer").dtype   # int8

Only downcast when you are sure future values will fit the smaller range, because an out-of-range value silently overflows.

Converting to dates and categories

The same principle applies across types. pd.to_datetime(..., errors="coerce") gives NaT for unparseable dates, and astype("category") converts a repetitive text column to save memory. Each conversion is a decision, and each has a failure mode worth checking.

Check the result, not just the call

After any conversion, confirm it did what you expected.

df.dtypes                # did the type actually change
df.isna().sum()          # how many values were lost to coercion

A conversion that silently turned a tenth of your column into NaN is worse than one that failed loudly, so always measure the loss. If coercion dropped more than a handful of values, go back and look at the raw text: there is usually a systematic cause (a thousands separator, a currency symbol, a footnote marker) that you can strip before converting, recovering the data rather than discarding it.

Need this solved on your own data?

The library is free and always will be. When the problem is bigger than an answer, we take on the work.

Start a project