[e6qu]

Two things: whether a larger model helps the model path (it doesn't), and the false flags of the model path on the second development set, with guards for 10 of the 14 false ones.

A 12B model on the same measurements

Gemma 3 12B (4-bit, 6.9 GB) fits in 16 GB with EMDYSI_LM_MMAP=0 EMDYSI_LM_CTX=1024 (from #82).

Locating by surprise (374 minimal pairs)	E2B	E4B	Gemma 3 12B
The corrected word is the most surprising	31.1%	30.0%	22.5%
... among the three most surprising	48.0%	46.6%	38.1%
AUC, as written against corrected	0.603	0.594	0.568

Choosing among versions (206 saved questions): correct originals changed (of 150), at most	E2B keeps (of 56)	12B keeps
1	32	26
6	44	34
10	46	34
26	48	51

Caveats: the questions were chosen by E2B, and the 12B model is another family. Still, nothing argues for the larger model here, so E2B stays the default.

The model path on the second development set

A sixteenth of the set (1,095 sentences) gave 18 flags, 4 of them real errors (realitime, anyeone, a overflow, allowing to disable). Guards for 10 of the 14 false ones:

False flag	Guard
So much the better to So much better	no the dropped before a comparative
coatings to date to to data	no word changed inside a fixed expression any reading takes as one item
AWS, Vsphere and to Sphere	no real-word typo in a technical term (cspell lists)
the logic … into main to from	no compound preposition replaced in a sentence the grammar accepts (none is in the real errors)
Landed to Lans	inflections and real-word neighbours only as listed in lower case (the list has LANs)
2 detailed updates to update (×4)	no plural made singular after a number other than one
rust-lang/rust#137944 is ready! to A rust	issue references are code-like; no edit touches one

Left: three flags in an Austen chapter (into a hackney coach to with, contrive to control, the thoughtless) and one unknown term (sret). Two broader guards (no preposition swap, no similar-stem word swap) were tried and dropped: they broke interested on chess to in, the case the model path is for. No threshold separates either: real gains scored 82–99%, these false flags 81–97%.

Each guard has a test that fails without it.

Measured

	Before	After
34 real-error pairs where a model can gain (E2B, first four guards)	8 caught; the model adds 4	12 caught; the model adds 5 (creating to of creating is new), none lost; 0 flags on the corrected text
Real errors without a model (374)	221	221, no pair changed
Blog posts without a model (where the issue references are)		0 flags in 244 sentences (dev1), 43 in 4,147 (dev2), 4 in 1,241 (held out); none about a reference

Also: examples/surprisal reports progress.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TxRMoAtBqwUZXAvBmzVhC4