Papers
What surface form can and cannot decide
Most of our automatic checks work on surface form: the shape of a phrase, the order of words, the form a verb takes. This is cheap, fast and explainable — when such a check fires, the reason can be stated in one sentence to the person it concerns. It also has a hard ceiling, and the useful part of the last months was learning to put a number on that ceiling instead of arguing about it.
Where form runs out
The shape of a mistake is often a legal shape of the language. Take a real learner error: a bare verb after a modal where the perfect is required, as in the invented example "he could pass the exam last year" instead of "could have passed". A check built on that shape recognises the mistake perfectly. It also fires on hundreds of perfectly correct sentences in ordinary published prose, because that same shape is everywhere in English. The mistake cannot be closed by form alone. We measured the cost, named it, and rejected the check.
Requiring the phrase to end cuts off living cases. Narrow the check so that it only fires at the end of a sentence, and it stops recognising the mistake in real writing, where something always follows. Remove that requirement and the check starts firing on correct sentences of our own, because an English word can be a noun in one place and a verb in another, as in the invented "told us help is coming", where "help" is a noun and our machinery knows it as a verb.
A handwritten word list is silent about what it does not contain. Two checks were quiet on genuine cases for no deeper reason than that the verb in question was missing from a list assembled by hand. A list written to cover the canonical examples of a rule will systematically fail to cover the vocabulary of real writing.
Neighbouring words are not syntax. Our checks compare words that stand next to each other. They do not see clause boundaries, the subject of a subordinate clause, or a displaced object. Widening the gap between two parts of a pattern once let in a clause that belonged to a different sentence inside the same sentence.
What worked
Add a requirement instead of removing one. In the case above, the fix was not to loosen the check but to demand a particular kind of word after the verb: one that occurs after a transitive verb and does not occur after a noun in the same position. The check opened up on real cases, the quiet-on-correct gates stayed green, and the measured cost on our reference collection of ordinary prose was a single false alarm, in a quoted line of direct speech.
Measure the parts before repairing the whole. Before changing a check, we run its partial versions and see which part loses the cases: "the first part recognises six out of seven, the part that consults the word list recognises none". The real culprit is named by a number before any edit is made, so the edit aims at it rather than at the first plausible theory. In that particular case the theory was wrong and the number was right.
A refusal with a stated price is a result. Where form cannot close a case, the check stays silent, with the reason and the cost of the alternatives written down. That is more honest than a check which fires on every tenth correct sentence a learner writes, and it is the decision we now reach for first.
What is still open
The debt is known by name: dozens of cases where the mistake is defined by syntax rather than by neighbouring words. For some rules, recall cannot be measured by form even in principle, because the same shape carries two different structures.
The road out is a syntactic parser alongside the present machinery. It is also a new dependency and a new family of instruments to check it with, because a parser will have its own ways of being wrong, and that catalogue is not yet written.
All articles
Free practice for every rule
Every grammar rule on the site has a trainer behind it. It is free and needs no sign-up.