Papers

Why we measure our own grammar rules

Our site carries a map of English grammar rules, from articles at A1 to inversion and clefts at C2, and behind each rule sits an automatic check that is supposed to recognise the mistake learners actually make. The hard part is not writing such a check. The hard part is knowing whether it works.

Silence has two readings

Suppose one of those checks has found nothing for a month. There are two possible reasons, and from the outside they look identical.

The first is that learners simply do not make this mistake in the texts we see. Some rules are rare: the occasion to get them wrong barely comes up at the level where they are taught.

The second is that the check is written in a way that will never recognise the mistake, no matter how many times it is made.

Both give the same number: zero. A zero that can mean either "nothing to find" or "unable to find" is not a measurement, it is an absence of one. Most of the wrong conclusions we have drawn about our own system started here.

The rule we adopted

Every question of the form "what is holding this rule back" has at least two outcomes, and an answer without its own measurement is not accepted.

In practice that means one of two things has to be produced for every silent rule. Either real places in learner writing where the mistake occurs and the check now recognises them after repair. Or a measured reason why such places are not there to be found, with a number and a sample that a person has actually read, case by case.

The second half of that sentence does the work. A number on its own tends to be believed; a number next to five sentences a person has read tends to be understood. More than once, reading the sample has overturned the number we were about to report, and in every one of those cases the mistake was ours and not the machine's.

What worked

Separating the workshop from the product. Candidate checks live where nothing in the live product reads them. They can be measured, broken, and measured again without anyone who uses the site being affected. A check reaches the product only through a separate step with its own gates, described in the fifth article of this series.

This sounds like ordinary engineering hygiene, and it is. The reason it matters here is that a measurement bench is tempting to run on the live thing: the live thing has all the data. The cost of that temptation is that every experiment becomes a release, and a release cannot be undone as cheaply as an experiment.

What is still open

The honest statement of our position is this. For no single rule do we yet know the full denominator: how many occasions for that particular mistake exist in the writing of learners at that level. Recall against a sample annotated blind, by a person who had never seen our checks, has not been measured by anyone. It is a debt we name rather than a gap we hide, and the third and seventh articles of this series return to it.

The rest of the series is a catalogue: how our own measuring tools misled us, what an honest denominator costs, what surface form can and cannot decide, and which of our habits turned out to be worth keeping.

All articles

Free practice for every rule

Every grammar rule on the site has a trainer behind it. It is free and needs no sign-up.

Report a problem

Page you are on
/en/science/papers/measure-01-why.html en

English is easiest for us, but write in any language you like.

The report carries only your words, the page address and the language. No screenshot, no logs, no account. We cannot reply to a report, so if you need an answer, write to [email protected]