What is the difference between loc and iloc, and when does each fail?
Short answer
loc selects by label, iloc selects by position. They agree only while the index happens to be a clean range, which is exactly why the bug appears after you filter or sort.
The short version
loc uses the labels in the index. iloc uses the integer position, counting
from zero. On a fresh DataFrame these look identical, because the labels happen to
be 0, 1, 2. After a filter or a sort they diverge, and code that used the wrong
one starts returning the wrong row silently.
The moment they diverge
import pandas as pd
df = pd.DataFrame({"state": ["Sokoto", "Kebbi", "Zamfara"], "schools": [412, 380, 265]})
print(df.loc[1, "state"]) # Kebbi
print(df.iloc[1]["state"]) # Kebbi identical, so far
filtered = df[df["schools"] > 300] # drops Zamfara, keeps index labels 0 and 1
print(filtered.loc[1, "state"]) # Kebbi label 1
print(filtered.iloc[1]["state"]) # Kebbi position 1
sorted_df = df.sort_values("schools") # index is now 2, 1, 0
print(sorted_df.loc[1, "state"]) # Kebbi still label 1
print(sorted_df.iloc[1]["state"]) # Kebbi position 1 is label 1 here
print(sorted_df.iloc[0]["state"]) # Zamfara position 0
print(sorted_df.loc[0, "state"]) # Sokoto label 0
Read the last two lines again. Same number, different rows. That is the bug.
Slicing behaves differently too
This trips up people who assume Python slice rules apply everywhere.
df.iloc[0:2] # two rows. The end is exclusive, like a Python list.
df.loc[0:2] # three rows. The end is inclusive, because labels are not positions.
When each one fails
loc raises a KeyError when the label is absent. That is good: it is telling
you the row you asked for does not exist.
iloc raises an IndexError when the position is out of range, and returns the
wrong row when it is in range but you meant a label.
The dangerous failure is the silent one, and only iloc has it.
The rule worth adopting
Use loc unless you specifically want positional access, which in analysis code
is rare. And after any filter, if you plan to use positions at all, reset the
index so positions and labels agree again:
filtered = df[df["schools"] > 300].reset_index(drop=True)
Pass drop=True unless you actually want the old index kept as a column.
The one that catches everyone
df[df["schools"] > 300]["state"] = "unknown" # does nothing, warns
df.loc[df["schools"] > 300, "state"] = "unknown" # assigns
The first line operates on a temporary copy, which is then discarded. Pandas warns
you with SettingWithCopyWarning. Do not silence that warning. Use .loc on the
left hand side of an assignment, always.