Answers, not search results
258 worked answers to the questions data analysts actually ask. Every one written from scratch, with runnable code. Describe your problem in your own words, or filter by stack.
-
Python Advanced
How do I cap extreme values instead of deleting them in pandas?
Use clip with a quantile as the bound to pull extreme values back to a threshold rather than removing the rows. This keeps the observations while limiting their influence, but it does change your data.
Read the answer -
Python Advanced
How do I catch type errors before running my Python code?
Add type hints and run mypy, which reads them and reports mismatches without executing the code. It catches a whole class of bug that tests may miss, especially around None.
Read the answer -
Python Advanced
How do I check that a merge behaved as expected in pandas?
Pass validate to assert the join is one-to-one or one-to-many so an unexpected duplicate key raises instead of silently multiplying rows. Pass indicator to see which side each row matched.
Read the answer -
Python Beginner
How do I check the type of a variable in Python?
Use isinstance, which respects inheritance and accepts a tuple of types, rather than comparing type() directly. Be aware that bool is a subclass of int, so a True value passes an int check.
Read the answer -
Python Beginner
How do I check whether a file exists in Python?
Use Path.exists from pathlib, or os.path.exists. But for the common case of opening a file, it is often better to just try and handle the error, to avoid a race condition.
Read the answer -
Python Intermediate
How do I check whether any or all rows meet a condition in pandas?
Call any or all on a boolean condition to reduce it to a single True or False. Use them for data validation checks, and remember they operate per column on a DataFrame unless you reduce twice.
Read the answer -
Python Intermediate
How do I check whether every column in a row meets a condition?
Apply the condition to the whole frame and reduce with all(axis=1) for a per-row answer, or all(axis=0) for a per-column one. The axis argument decides which direction is collapsed.
Read the answer -
Python Advanced
How do I choose the decision threshold for a classifier?
The default 0.5 cut-off is arbitrary and usually wrong for imbalanced data. Get probabilities with predict_proba and pick the threshold that suits the cost of each error, using the precision-recall curve.
Read the answer -
Python Intermediate
How do I clean messy text values in a pandas column?
Chain the str accessor methods to strip whitespace and normalise case, and strip out stray characters before converting to numbers. Inconsistent whitespace and case are the usual cause of failed joins and duplicate groups.
Read the answer -
Python Advanced
How do I combine many DataFrames efficiently in pandas?
Collect the frames in a list and call concat once at the end. Concatenating inside a loop rebuilds the whole result every iteration, which turns a fast job into a quadratic one.
Read the answer -
Python Intermediate
How do I combine NumPy arrays into one?
Use vstack to stack rows, hstack to join columns, and concatenate with an axis for the general case. The shapes must match on every axis except the one you are joining along.
Read the answer -
Python Intermediate
How do I compare a value to several rows back in pandas?
Pass a periods argument to diff or shift to look back more than one row, such as twelve for a year-on-year monthly comparison. The first rows come out as NaN because there is nothing that far back.
Read the answer -
Python Advanced
How do I compare each row to the previous one in pandas?
Use shift to look at the previous row's value, diff for the change since the previous row, and pct_change for the percentage change. Sort first, and use groupby so comparisons do not cross group boundaries.
Read the answer -
Python Intermediate
How do I compare the columns or keys of two DataFrames?
A pandas Index supports set operations, so use intersection, difference and union to compare column names or key values. This is the quickest way to explain why a merge lost rows.
Read the answer -
Python Advanced
How do I compute an expensive attribute only once in Python?
Use functools.cached_property so the value is computed on first access and stored on the instance. Unlike lru_cache it does not hold the object alive, but it never recomputes, so only use it for values that do not change.
Read the answer -
Python Intermediate
How do I convert a column to a different type safely in pandas?
astype raises on any value it cannot convert, while to_numeric with errors="coerce" turns the bad ones into NaN so you can inspect them. Convert deliberately and check what failed.
Read the answer -
Python Intermediate
How do I convert a column to dates and extract the month or year in pandas?
Convert the column with to_datetime, then read parts through the dt accessor such as dt.year or dt.month. Set the format or dayfirst argument when parsing is ambiguous.
Read the answer -
Python Beginner
How do I convert a string to a number in Python?
Use int for whole numbers and float for decimals. int rejects a string with a decimal point, so convert to float first, and wrap the conversion in try/except to handle input that is not a number.
Read the answer -
Python Intermediate
How do I convert between strings and dates in Python with strptime and strftime?
strptime parses a string into a datetime using a format code, and strftime formats a datetime back into a string. The format codes must match the text exactly, and mixing the two directions is the usual mistake.
Read the answer -
Python Intermediate
How do I convert labels to numeric codes in pandas?
Use factorize to turn labels into integer codes plus the list of unique values, which lets you map back. Do not use the codes as a numeric feature unless the categories genuinely have an order.
Read the answer -
Python Beginner
How do I count distinct values in pandas?
Use nunique for the number of distinct values and unique to list them. nunique ignores missing values by default, and works per group when combined with groupby.
Read the answer -
Python Beginner
How do I count how often each value appears in a column?
Use value_counts on the column for a quick frequency table sorted by count. Pass normalize for proportions, and dropna to include missing values.
Read the answer -
Python Intermediate
How do I create a column that depends on another column I just created in pandas?
Inside a single assign call, use a lambda for the second column so it sees the frame including the first. Referring to the original frame directly fails, because that column does not exist yet.
Read the answer -
Python Intermediate
How do I create a new column based on conditions in pandas?
Use np.where for a two-way choice and np.select for several conditions. Both are vectorised, so they are faster and cleaner than looping or using apply with an if statement.
Read the answer -
Python Intermediate
How do I create every combination of rows from two DataFrames in pandas?
Use merge with how="cross" to pair every row of one frame with every row of the other. The result grows as the product of the two sizes, so it can become very large quickly.
Read the answer