Papers

The defect usually lives in the instrument

The most expensive lesson of the past months is also the least glamorous. When a measurement produces a strange number, the fault is more often in the thing doing the measuring than in the thing being measured. What follows is a catalogue. Every item is a real case that cost hours of searching in the wrong place.

Ways our own tools misled us

The tool repeats the product. A checking script assembled from the same pieces as the code it checks does not confirm correctness, it confirms identity: it is blind to exactly the same mistakes. The cure is to measure along a different path, with different machinery, or by reading.

The tool reads only the first match. One script took the first item that fitted and reported "one match" where there were five. A zero or a one from such a tool means nothing until it is shown that the tool reads to the end.

A guard is blind by construction. One of ours inspected files for borrowed wording, but only extracted text in quotation marks. A borrowed fragment set in brackets was invisible to it. A green guard proves the absence of the family of problems it can see, and nothing beyond that.

An empty search is not a clean result. No output means "not found", and "not found" can equally come from a typo in the query. A negative result is accepted only after the same query has found something that is known to be there.

Green before the check is not green. A report of "fixed" written before the tool has run is a statement about intention, not about state. We report state.

Our own limit reads as someone else's defect. A rate limit on our side appeared in one measurement as a failure of an external service. Before blaming the measured thing, check whether the instrument hit its own ceiling.

What worked

Take the unit from the code. The tool must take its unit of measurement from the code being measured, instead of inventing a convenient one. When we measured how far a check gets through a sentence, the partial attempts were run by the same machinery the product uses. That is why the resulting numbers could be trusted: the path was the product's path, not a sketch of it.

One change per measurement. Between two measurements exactly one thing changes. Variants are numbered, each differing from the previous one by a single edit, and only the winner is kept. Otherwise there is no way to say which edit produced the effect and which produced the cost.

A tool found wrong retracts its numbers. When an instrument turns out to have been broken, every number it produced is withdrawn in public, including the convenient ones. This is not self-flagellation; it is the only way the record stays usable.

What is still open

We have no automatic step that checks the instruments themselves, the one that would demand "this tool found something known to exist before its zero was believed". It rests on the discipline of whoever runs the bench, which means it will be skipped on a tired evening. How to build it into every run without doubling the cost of the bench is an open question.

A related limit is worth naming: an instrument always measures the state it was called from, not the state you had in mind. A measurement taken from a working copy says nothing about the published version, and a green result against stale data is a statement about yesterday.

All articles

Free practice for every rule

Every grammar rule on the site has a trainer behind it. It is free and needs no sign-up.

Report a problem

Page you are on
/en/science/papers/measure-02-instrument.html en

English is easiest for us, but write in any language you like.

The report carries only your words, the page address and the language. No screenshot, no logs, no account. We cannot reply to a report, so if you need an answer, write to [email protected]