Papers

An honest denominator

Every share is worth exactly what its denominator is worth. "The check catches 14 per cent of cases", "it recognises one place out of three" — such sentences feel like measurements, and most of our wrong readings of our own system were mistakes in the denominator rather than in the count on top.

Four ways a denominator goes wrong

It is more generous than the truth. For one rule the harvesting tool reported seven places where the mistake occurred. Reading them showed that in three of the seven the learner's sentence already contained the word in question: the real error there belonged to a neighbouring rule, and the search had simply caught a correction standing next to it. So the check "stayed silent on seven places", of which fewer than seven were ever its own. A share whose denominator nobody has read is noise.

Zero recall is not zero precision. A check that has recognised no real case can still be perfectly well behaved in the other direction: it stays quiet on correct writing. These are separate axes. Mixing them produced conclusions of the kind "this check is dead", when only one of its two sides was.

A low share is sometimes honest rarity. Before repairing a check that fires on two cases in a hundred, the ceiling of the rule itself has to be measured: how many occasions for that mistake arise at all in writing at that level. If the occasion comes up three times in a hundred pieces of writing, then two per cent may be an honest result, and "repair" will buy nothing but false alarms.

Gains do not add across different bases. An improvement measured on one collection cannot be added to a number obtained on another. Families of mistakes overlap, so the sum of per-family shares exceeds the whole. Only numbers that share a denominator can be compared, and we have caught ourselves breaking this more than once.

The fourth outcome

The usual three outcomes for a construction are: used correctly, used with a mistake, or confused with a neighbouring rule. There is a fourth, and we count it nowhere. The learner avoids the construction altogether and paraphrases around it.

For anyone who teaches, that is a signal in its own right: a rule that is systematically avoided has not been learned, whatever the error counts say. Building a counter for it is harder than finding mistakes, because it means recognising a place where a construction would have been apt and was not used. We have no instrument for this, and we would like one.

What worked

A small denominator is read, not sampled. Up to about ten places, all of them are read, and the share is reported together with a breakdown of each case by the role the words play in it. Larger families are read as a sample, and the sample grows with the yield rather than staying at a convenient five.

Every share arrives with an example. A number without at least one worked case is not accepted into a report. An extreme case is not a rule, but a rule without a single case has not been measured.

What is still open

Making the harvest precise enough that neighbouring mistakes stay out of a rule's denominator means changing the harvesting tool, which then needs gates of its own. And the counter for avoidance described above does not exist. Until it does, every recall number we publish is a statement about the occasions we can see, not about the language the learner is actually producing.

All articles

Free practice for every rule

Every grammar rule on the site has a trainer behind it. It is free and needs no sign-up.

Report a problem

Page you are on
/en/science/papers/measure-03-denominator.html en

English is easiest for us, but write in any language you like.

The report carries only your words, the page address and the language. No screenshot, no logs, no account. We cannot reply to a report, so if you need an answer, write to [email protected]