How do I rank rows within a group using a window function?

The short version

SELECT
    state,
    school,
    enrolment,
    ROW_NUMBER() OVER (PARTITION BY state ORDER BY enrolment DESC) AS position
FROM schools;

PARTITION BY restarts the numbering for each state. ORDER BY decides what "first" means. No GROUP BY is involved, and the detail rows survive.

The three functions, and why the choice matters

Given enrolments of 900, 850, 850, 700 within one state:

ROW_NUMBER never ties. It will pick one of the two 850s arbitrarily, and which one it picks can change between runs unless you add a tiebreaker to the ORDER BY.

RANK ties, then skips. Two second places, no third.

DENSE_RANK ties, then continues. Two second places, then third.

If you are reporting "the top three schools" and two are tied for third, these give you different answers. Decide which is correct for the question, and say so in the report.

Getting the top n per group

WHERE cannot see a window function, because it runs before the window is computed. Wrap it.

WITH ranked AS (
    SELECT
        state,
        school,
        enrolment,
        DENSE_RANK() OVER (PARTITION BY state ORDER BY enrolment DESC) AS position
    FROM schools
)
SELECT state, school, enrolment
FROM ranked
WHERE position <= 3
ORDER BY state, position;

Make ties deterministic

If two rows tie on the ordering column, add a second column that cannot tie. Otherwise the same query returns different rows on different days, and somebody will eventually notice.

ROW_NUMBER() OVER (PARTITION BY state ORDER BY enrolment DESC, school ASC)

The mistake to avoid

SELECT state, MAX(enrolment) FROM schools GROUP BY state;   -- gives the number

That tells you the highest enrolment, not which school had it. Adding school to the select list either errors or, in MySQL's looser modes, silently returns an arbitrary school. Window functions exist because GROUP BY cannot answer "which row".