Papers
The collection you measured on is part of the instrument
"No false alarms on ordinary prose" means nothing without an answer to "which prose". The collection a measurement was taken on is as much a part of the instrument as the code. We learned this the way such things are usually learned: by losing two collections at once.
What happened
An early stretch of work had measured the cost of changes against a collection of prose excerpts, and the baseline of our gates against a set of canonical examples of the rules. Neither collection had been kept as a file. What survived was the numbers in the reports.
New changes had to be compared with those old zeros, and the denominator of the old zeros was gone. Since comparing against a different collection is not legitimate — that is the subject of the third article — the choice was between declaring the old numbers incomparable and losing the history, or reconstructing the collections exactly.
What worked
Reconstruction, not recollection. The assembly of the prose collection is deterministic: a configuration lists the sources, and a script cuts them into excerpts by fixed rules. Not all of the listed sources were present any more. The hypothesis was that the old collection had been harvested from the subset that was present at the time. Running the assembly over that subset produced exactly 7,773 excerpts — the same number the old reports carried. The match of an exact count is itself the proof of composition. The old zeros became comparable again.
Fill the collection up, never replace it. The missing sources were fetched and the collection grew to 34,470 excerpts. Both costs are now reported for every change: on the historical subset, for comparability with the record, and on the full collection, as the current baseline. Replacing the first with the second would have erased the ability to read our own history.
The rebuilt baseline was made stricter, not looser. In place of the lost set of canonical examples covering a few dozen rules, we assembled 493 examples covering 162 rules, taken from the source the rule map itself is built from. The baseline on the new set matched the old one, down to the single known red case. Widening the coverage did not break the history, which is the only evidence that the widening was done right.
Raw material does not have to live in the repository. The sources are large and can be fetched again. What lives in version control is the configuration, the assembly script and the fingerprint numbers. That is enough for anyone to rebuild the same collection.
The rules this produced
- Every measurement names its collection and its size, every time.
- A collection is added to, never replaced: the old composition stays reproducible, and an extension adds a second figure instead of overwriting the first.
- A deterministic assembler, a configuration and a fingerprint count are the minimum kit for reproducibility. Keeping the collection itself as a file is optional once the kit is complete.
What is still open
The learner side of the bench cannot be reproduced this way. Writing by learners is not copied into version control; what serves as the collection there is a set of references to places in storage, and references are more fragile than a fingerprint. A reference that has moved is indistinguishable from a reference that was never right.
How to make measurements on learner writing as reproducible as measurements on published prose is an open question, and it is the one that currently limits how far we trust our own recall numbers.
All articles
Free practice for every rule
Every grammar rule on the site has a trainer behind it. It is free and needs no sign-up.