How do I test whether two groups differ in Python?

The short version

For two independent groups, use scipy.stats.ttest_ind with equal_var=False (Welch's test), which does not assume the groups have equal variances. Report the difference in means and its confidence interval alongside the p-value, because a p-value alone tells you nothing about how big the difference is.

Running the test

import numpy as np
from scipy import stats

group_a = np.array([12, 15, 14, 16, 13])
group_b = np.array([18, 20, 17, 19, 21])

result = stats.ttest_ind(group_a, group_b, equal_var=False)
print(result.statistic, result.pvalue)

The p-value is the probability of seeing a difference at least this large if the two groups actually had the same mean.

Why equal_var=False by default

The classic Student's t-test assumes both groups have the same variance. Real groups often do not, and when they differ, the classic test gives misleading results. Welch's test (equal_var=False) does not make that assumption and performs nearly as well when the variances are equal. So it is the safer default, and many statisticians recommend using it routinely.

The p-value is not the finding

A small p-value says the difference is unlikely to be pure chance. It does not say the difference is large, or that it matters. With a big enough sample, a trivially small difference produces a tiny p-value. Always report the actual difference.

difference = group_b.mean() - group_a.mean()

A reader needs to know the groups differ by, say, 5 units, not merely that p is below 0.05. The size of the effect is the finding; the p-value is only evidence that it is not noise.

Paired data needs a different test

If the two sets of measurements are on the same subjects (before and after), they are not independent, and ttest_ind is wrong. Use ttest_rel instead, which compares the paired differences and is more powerful.

stats.ttest_rel(before, after)

Using the independent test on paired data throws away the pairing and usually understates the evidence.

The assumptions

The t-test assumes roughly normal data (or a large enough sample for the mean to be approximately normal), and independent observations within each group. It is fairly robust to mild departures from normality, but not to a badly skewed distribution in a small sample. For that, a non-parametric alternative such as mannwhitneyu compares ranks instead and makes fewer assumptions.

The multiple-comparisons trap

If you test many pairs of groups, some will look significant by chance alone. Testing twenty things at the 0.05 level means you would expect one false positive even if nothing is really different. If you run several tests, adjust for it, or at least say clearly how many tests you ran. Reporting only the one that came out significant is how spurious findings enter the world.

Decide before you look

Choose your test, your threshold and your outcome measure before seeing the results. Choosing them afterwards, in the light of what looks promising, invalidates the p-value entirely, however carefully the arithmetic is done.