Papers

Say the number before you measure

This article is about the part of our practice that works, and about why it is this part and not a cleverer one.

The number goes on the record first

Before a measurement runs, we write down what is being measured, with what, what number we expect, and with what tolerance. Only then is the tool started.

This is not a ritual. Writing the expectation down in advance is what separates a measurement from a fitting exercise. Once the number is known, a person — and a language model — quietly widens the tolerance: "well, six is nearly two to six". Three consequences follow, and all three are uncomfortable on purpose.

The edge of the tolerance is not a hit. A result of six against an expectation of two to six is reported on its own line, as the edge, not folded into "within range".

Misses are reported first. A report opens with what did not match. Anything else turns the report into an advertisement for the person who wrote it.

A predicted failure is also a prediction. "Removing this requirement will be rejected by the gate" was written down before the run, and the gate rejected it. A failure predicted correctly is worth more than a hit arrived at by luck: it means the model we hold of the system's behaviour is right.

The most useful finding about this habit is that misses turned out to be more productive than hits. In the case described in the previous article, it was a missed prediction — "this part will recognise one to five cases", and it recognised none — that identified the real culprit, which was not the one the original diagnosis had named.

Three gates around every change

No change to a check is accepted on the strength of one number. There are three gates.

  • It recognises its own mistake: the check fires on the known bad sentences.
  • It stays quiet on correct text: no firing on corrected versions of those same sentences, and none on the canonical examples of every rule on the map, of which there are several hundred.
  • It stays quiet on ordinary prose: the cost is measured in false alarms across a large fixed collection of published writing, and it is stated for every change, on the collection used historically, for comparability, and on the current full one.

The third gate is the one that kills most candidate checks, and it is the one that would be easiest to leave out, because it is the only one that cannot be passed by trying harder.

What is still open

The discipline rests on the person running the bench. Nothing technically prevents a tool from being run without a declaration first; only the rule does. We would like the refusal to be built into the tool itself: no declared numbers, no measurement. That needs a machine-readable format for declarations which does not turn a working conversation into paperwork, and we have not found it yet.

There is a second, quieter risk. A habit that produces uncomfortable reports is only kept while someone is willing to be uncomfortable. The three consequences above are written down for that reason: a rule on paper survives a bad week better than a good intention.

All articles

Free practice for every rule

Every grammar rule on the site has a trainer behind it. It is free and needs no sign-up.

Report a problem

Page you are on
/en/science/papers/measure-05-declarations.html en

English is easiest for us, but write in any language you like.

The report carries only your words, the page address and the language. No screenshot, no logs, no account. We cannot reply to a report, so if you need an answer, write to [email protected]