How do I drop rows or columns that are mostly empty in pandas?

The short version

dropna() by default removes any row with even one missing value, which is far too aggressive on a wide frame. Use thresh=n to keep rows with at least n real values, or how="all" to remove only completely empty rows. For dropping sparse columns, filter on the missing proportion directly.

The default is aggressive

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "a": [1, np.nan, np.nan],
    "b": [1, 2, np.nan],
    "c": [1, 2, 3],
})

len(df.dropna())        # 1 , any missing value kills the row

On a frame with twenty columns, a single gap anywhere removes the whole row, so dropna() can wipe out most of your data. It is rarely what you want without qualification.

Requiring a minimum of real values

thresh keeps rows with at least that many non-missing values.

len(df.dropna(thresh=2))   # 2 , rows with at least two real values survive

This is the sensible middle ground: a row missing one field is usable, a row missing almost everything is not.

Dropping only the completely empty

len(df.dropna(how="all"))   # 3 , only rows that are entirely NaN go

Use this to remove blank rows that often appear at the end of exported spreadsheets, without touching rows that have partial data.

Limiting to the columns that matter

Often only a few columns are essential, and missing values elsewhere are tolerable. subset restricts the check.

df.dropna(subset=["c"])   # drop only if the key column c is missing

This is usually the right approach: state which fields a row cannot do without, rather than demanding every field.

Dropping sparse columns

dropna(axis=1) drops columns with any missing value, which is even more brutal. To drop columns that are mostly empty, filter on the missing proportion.

keep = df.loc[:, df.isna().mean() <= 0.5]   # keep columns at most half missing

isna().mean() gives the missing fraction per column, which is a far more useful measure than a raw count, because it is comparable across differently sized frames.

Look before you drop

Always measure the missingness before removing anything, so you know what the drop will cost.

df.isna().mean().round(2).sort_values(ascending=False)

If dropping would remove most of your rows, the right answer is probably to fill or to reconsider which columns you truly need, not to delete the data.

The deeper question

Dropping is a decision about what missing means. If values are missing at random, dropping loses information but does not bias the result much. If they are missing for a reason (a field only filled in certain cases), dropping those rows systematically removes a particular kind of record and biases everything downstream. Think about why the gaps exist before you decide to discard them.