What does a p value actually mean, and what does it not mean?
Short answer
It is the probability of seeing data at least this extreme if the null hypothesis were true. It is not the probability that the null hypothesis is true, and it never was.
The short version
A p value answers one narrow question: if there were genuinely no effect, how often would random sampling hand me data at least as extreme as what I got?
It does not tell you the probability that your hypothesis is right. It does not tell you the size of the effect. It does not tell you whether the effect matters.
The four things it is routinely mistaken for
"p = 0.03, so there is a 3 percent chance the null is true." No. The p value is calculated assuming the null is true. It cannot then tell you the probability of the thing it assumed.
"p = 0.03, so there is a 97 percent chance the effect is real." No, for the same reason.
"p > 0.05, so there is no effect." No. Absence of evidence is not evidence of absence. A small study can miss a large real effect entirely.
"p = 0.001, so the effect is large." No. With a big enough sample, a trivial difference produces a tiny p value. Significance is about detectability, not size.
What to report instead, or as well
Give the effect size and its confidence interval. They answer the question a director is actually asking.
Say this: enrolment in the intervention districts was 6.2 percentage points higher, 95 percent CI 2.1 to 10.3.
Not this: enrolment was significantly higher, p < 0.05.
The first tells you how much, and how sure. The second tells you almost nothing.
Why this matters on programme data
Development programmes get evaluated on whether an indicator moved. With a large household survey, almost any difference reaches significance. A 0.4 percentage point change in a coverage rate can be "statistically significant" and completely irrelevant to whether the programme worked.
Conversely, a pilot in three districts may show a real, substantial improvement that fails to reach significance because the sample was small. Reporting "no significant effect" invites a funder to conclude the programme failed. It did not fail. The study was not powerful enough to see it.
Both errors have cost real programmes real money.
The honest sentence
If somebody demands a single line, this is defensible:
The difference we observed would occur by chance about three times in a hundred if the programme had no effect at all. The best estimate of the effect is 6.2 percentage points, and the data are consistent with anything between 2.1 and 10.3.
Long, unglamorous, and true.