Answers, not search results
258 worked answers to the questions data analysts actually ask. Every one written from scratch, with runnable code. Describe your problem in your own words, or filter by stack.
-
Python Advanced
Are list comprehensions actually faster than for loops in Python?
Slightly, because a comprehension avoids repeated method lookups and function call overhead, but the difference is small. Neither beats a vectorised NumPy or pandas operation on numeric data, which is where the real gains are.
Read the answer -
Python Advanced
How do async and await work in Python?
async def creates a coroutine, await pauses it while waiting so other tasks can run, and asyncio.run starts the event loop. It speeds up code that waits on input and output, not CPU-bound work.
Read the answer -
Python Advanced
How do I add context to an exception without losing the original error?
Use raise NewError(...) from err to attach a meaningful error while keeping the original as its cause. The traceback then shows both, which is what you need to diagnose the failure.
Read the answer -
Python Advanced
How do I aggregate a time series to a coarser frequency in pandas?
Set a datetime index, then use resample with a frequency such as ME or W followed by an aggregation. resample needs a datetime index, and the frequency aliases changed in recent pandas.
Read the answer -
Python Advanced
How do I apply a custom function over a rolling window in pandas?
Use rolling(n).apply(func) with a function that reduces each window to a single number. It is slow because it runs Python per window, so prefer the built-in rolling methods where one exists.
Read the answer -
Python Advanced
How do I apply different preprocessing to numeric and categorical columns in scikit-learn?
Use ColumnTransformer to route numeric columns to a scaler and categorical columns to an encoder in one step. Wrap it in a pipeline so the whole thing fits on training data only and applies consistently.
Read the answer -
Python Advanced
How do I build a scikit-learn Pipeline and why should I?
A Pipeline chains preprocessing steps and a model into one object, so fit and predict apply every step in order. It prevents data leakage during cross-validation and keeps training and prediction consistent.
Read the answer -
Python Advanced
How do I calculate a running total within each group in pandas?
Sort the data in order, then use groupby with cumsum so the running total restarts for each group. Forgetting to sort, or forgetting the groupby, are the two mistakes that give wrong totals.
Read the answer -
Python Advanced
How do I calculate a weighted average in pandas?
Use numpy.average with a weights argument, or multiply and divide the sums yourself. A plain mean ignores the weights, which quietly gives the wrong answer when the quantities differ.
Read the answer -
Python Advanced
How do I calculate quantiles and percentile ranks in pandas?
Use quantile to find the value at a given percentile, and rank with pct=True to find each value's percentile position. They answer opposite questions, and the interpolation method affects the quantile result.
Read the answer -
Python Advanced
How do I calculate the time between rows in pandas?
Subtract datetime columns or use diff to get a Timedelta, then convert to the unit you want with dt.total_seconds. Sort first and group so gaps do not span across groups.
Read the answer -
Python Advanced
How do I calculate year-over-year or period-over-period growth in pandas?
Sort by period, then use pct_change to get the proportional change from the previous period, or pct_change with periods=12 to compare with the same month a year earlier. Group so growth does not cross between series.
Read the answer -
Python Advanced
How do I cap extreme values instead of deleting them in pandas?
Use clip with a quantile as the bound to pull extreme values back to a threshold rather than removing the rows. This keeps the observations while limiting their influence, but it does change your data.
Read the answer -
Python Advanced
How do I catch type errors before running my Python code?
Add type hints and run mypy, which reads them and reports mismatches without executing the code. It catches a whole class of bug that tests may miss, especially around None.
Read the answer -
Python Advanced
How do I check that a merge behaved as expected in pandas?
Pass validate to assert the join is one-to-one or one-to-many so an unexpected duplicate key raises instead of silently multiplying rows. Pass indicator to see which side each row matched.
Read the answer -
Python Advanced
How do I choose the decision threshold for a classifier?
The default 0.5 cut-off is arbitrary and usually wrong for imbalanced data. Get probabilities with predict_proba and pick the threshold that suits the cost of each error, using the precision-recall curve.
Read the answer -
Python Advanced
How do I combine many DataFrames efficiently in pandas?
Collect the frames in a list and call concat once at the end. Concatenating inside a loop rebuilds the whole result every iteration, which turns a fast job into a quadratic one.
Read the answer -
Python Advanced
How do I compare each row to the previous one in pandas?
Use shift to look at the previous row's value, diff for the change since the previous row, and pct_change for the percentage change. Sort first, and use groupby so comparisons do not cross group boundaries.
Read the answer -
Python Advanced
How do I compute an expensive attribute only once in Python?
Use functools.cached_property so the value is computed on first access and stored on the instance. Unlike lru_cache it does not hold the object alive, but it never recomputes, so only use it for values that do not change.
Read the answer -
Python Advanced
How do I detect outliers in a column using the IQR method in pandas?
Compute the interquartile range, then flag values more than 1.5 times the IQR below the first quartile or above the third. This is robust to the outliers themselves, unlike a mean-and-standard-deviation rule.
Read the answer -
Python Advanced
How do I drop rows or columns that are mostly empty in pandas?
Use dropna with thresh to require a minimum number of real values, or how="all" to drop only entirely empty rows. To drop sparse columns, filter on the proportion of missing values instead.
Read the answer -
Python Advanced
How do I evaluate a model with cross-validation in scikit-learn?
Use cross_val_score to train and test on several splits and average the results, giving a more reliable score than a single split. Pass a pipeline so preprocessing is refitted per fold, and stratify for classification.
Read the answer -
Python Advanced
How do I fill a column from a second column where the first is missing in pandas?
Use fillna with another Series, or combine_first, to take the first column's value where present and fall back to the second where missing. This is the pandas equivalent of SQL COALESCE.
Read the answer -
Python Advanced
How do I fill gaps in a series by interpolating in pandas?
Use interpolate to estimate missing values from the ones around them, linearly by default. It suits ordered numeric data like time series, and you should sort first and consider limiting how far it fills.
Read the answer -
Python Advanced
How do I fill missing values using each group's own values in pandas?
Combine groupby with transform to fill each group's gaps from that group's own mean or neighbours, rather than from the whole column. A group that is entirely missing cannot fill itself, so those gaps remain.
Read the answer