Papers

Judges, guards, and what we still cannot measure

Not everything can be measured by the shape of a phrase. Where a judgement of meaning is needed — whether a piece of writing is good, whether a construction is apt, whether a style is right — we put an automatic judge in place. Around the bench itself stand guards: for secrets, for contracts, for borrowed wording. Judges and guards are instruments too, and they have their own ways of being wrong.

Where they went wrong

A refusal becomes a verdict about the content. When a judge declines to answer — overload, a filter, a timeout — a pipeline without special handling records the refusal as a score, usually a low one. A learner's sentence then carries a verdict that nobody ever passed on it. The cure is to measure both doors separately: the share of refusals, and the distribution of the real scores. A refusal is a line of statistics, not an opinion.

A timeout set to the average turns the tail into refusals. Set the waiting time by the average response time, and the slow tail of the distribution becomes systematic refusals, which — without the previous lesson in place — become systematic low scores.

A guard can be blind by construction. The guard that checks our texts for borrowed wording extracts only the passages in quotation marks. A fragment in brackets, without quotes, is invisible to it. We caught one such case by hand, an hour before it would have been committed. The blind spot is now written into the guard's own documentation, because until it was, a green light from that guard produced false confidence, which is the worst state an instrument can be in.

An open measure drifts towards easy cases. If the thing being measured is allowed to choose what it is measured on, the sample quietly slides towards the easy end and the success rate rises without any gain in quality. An open measure is a defect in its own right: the sample must be fixed by someone other than the subject.

Automatic proofreading of style cannot be trusted alone. A single style judge is wrong unsystematically, so its word is not enough to change a text. What works is agreement between two independent voices; on the pairs we measured, the share of false alarms falls by an order of magnitude. A lone voice may flag, never edit.

What worked

  • Pipelines that use judges count refusals separately from verdicts, and the share of refusals is a required line in the report.
  • Changes based on automatic proofreading require two voices to agree.
  • Guards carry a written blind-spot note: what this guard cannot see by construction. The green light is read together with that note.

The programme we have not done yet

A sample annotated blind. Recall of our checks has never been measured against writing annotated by a person who had not seen the checks. Without it, what we call recall is recall over places our own tools found, which is a measure of our tools agreeing with themselves.

Calibration of the writing judge against a reference collection scored by people, so that its verdicts sit on a human scale rather than on its own.

Synthesis of mistakes for rules where the collected material contains no natural cases. Synthetic cases would let us measure recall where nature is silent, but first it has to be shown that a synthetic mistake stands for a natural one, and that demonstration needs an instrument of its own.

A measurement bench is never finished. Every new instrument brings its own ways of lying, and the only real hope is that the catalogue of lies grows more slowly than the catalogue of instruments.

All articles

Free practice for every rule

Every grammar rule on the site has a trainer behind it. It is free and needs no sign-up.

Report a problem

Page you are on
/en/science/papers/measure-07-judges.html en

English is easiest for us, but write in any language you like.

The report carries only your words, the page address and the language. No screenshot, no logs, no account. We cannot reply to a report, so if you need an answer, write to [email protected]