I Published a Perfect Recall. Then I Measured Dust.

dossier-preflight answers one question: is this dossier going to be rejected at the counter. Not extraction, not document generation. It checks a set of filed documents against an external reference, and it measures what is actually PRINTED on the sheet, because an AcroForm field can carry a value that never prints while the clerk reads paper. The code and every number below are public at github.com/DylanMerigaud/dossier-preflight.

Nine thresholds, none of them picked by hand

Nine checks: a required field left empty, a box left unticked, a missing signature, an expired piece, two pieces whose data disagree, a forbidden value reappearing, a scan too coarse to read, a truncated page, an upside-down page. Each one emits a continuous score, and each threshold is read off a precision/recall curve instead of chosen by eye (commit db72954).

The rig behind that: 384 cells (6 rotation angles x 4 resolutions x 4 JPEG qualities x 4 noise levels) x 3 seeds = 1152 dossiers, 13,824 image readings, 121 minutes on 13 local workers. Inside the nominal domain of 150 dpi and up, eight of the nine checks hold a recall of 1.000 at zero false positives per target. The ninth, forbidden values, holds 0.997 at 0.00058 per target.

The rate a user actually feels is neither of those. It is the chance that an entirely clean dossier still sets something off: 4 dossiers out of 864, 0.46%, all four from the forbidden-value check. A gate that cries for nothing gets skimmed, and the rule sitting next to it gets skimmed with it.

The seed I did not look at

Choosing an operating point on a set of draws and then reporting its score on those same draws inflates it by an amount nobody can recover afterwards. The protocol against that is in LIMITS.md at github.com/DylanMerigaud/dossier-preflight, in the words it was written in:

The threshold is chosen on seeds 11 and 23, and the published figure is the one from seed 37, never looked at beforehand. Choosing a threshold and reporting its recall on the same draws always overestimates it.

That splits every table in that file in three rows: calibration on 576 positives, validation on 288, overall on 864. The required-field check reads 1.000 with a 95% interval of [0.987, 1.000] on the held-out seed alone. It is a real holdout, and it is the cheapest discipline in the whole project. It also did nothing at all for the failure that comes next.

The grid moved ink. It never added any.

Four factors vary in that grid: angle, resolution, JPEG quality, noise. All four move, blur or dirty the ink already on the page. Not one of them puts new ink where there was none. And the sensor that won the duel for "is this required field empty" is an ink sensor: it does not read anything, it measures how much darker a zone got compared to the blank form (commit 3a4e682).

sensor                                         recall  false positives per target
---------------------------------------------  ------  --------------------------
added ink, threshold 128 (retained)            1.000   0.0000
full-page plus per-zone OCR, min confidence 0  1.000   0.0003
OCR alone, min confidence 10 and up            0.000   0.0000

It won on ground built to favour it. For a while the limits file said so as a hypothesis, which is the polite way of shelving a problem.

A cliff, not a slope

A separate probe put a number on it, 528 readings, on 2026-08-21. Three shapes of foreign ink laid on the clean render before degradation, so each one goes through the same rotation, blur, noise and compression as the page: a speck, a fold shadow, a pen stroke spilling over from the neighbouring field. Target: the exact field that the empty-required-field variant leaves blank. The control with no parasite returns zero false negatives and zero false positives on all four sensors, so the bench itself is sound (commit c96c5ac).

sensor                                   false negatives  95% CI
---------------------------------------  ---------------  --------------
ink, threshold 128 (retained)            1.000            [0.975, 1.000]
union of the word sensors, confidence 0  0.272            [0.207, 0.347]
full-page OCR                            0.185            [0.132, 0.255]
per-zone OCR                             0.106            [0.066, 0.165]

Those are false negatives, the expensive side: the counter rejects the dossier and the tool said nothing. The shape is the part I did not expect. The published threshold is 0.345% added ink. The sensor fires 97 times out of 97 up to 0.323%, and 0 times out of 167 from 0.380% on. The two populations do not overlap by one single reading. That is a step, and it sits on the exact number I had published.

For that field, 15,770 canonical pixels, 0.35% is a 9 x 8 px speck at 200 dpi. That is a piece of dust on the scanner glass. A pen stroke barely spilling out of the neighbouring field already adds 0.61%.

Why the threshold did not move anyway

A probe on one field, two cells and three ink shapes shows that a choice was settled on biased ground. It is not enough to move a threshold. So the grid was replayed with parasitic ink as a fifth factor, laid on the very field a variant empties, over 27 cells and 3 seeds (commit aa7e0ed).

sensor                         clean  at 0.2% ink  at 1%  at 4%  false positives at 4%
-----------------------------  -----  -----------  -----  -----  ---------------------
ink, threshold 128 (retained)  1.000  0.333        0.000  0.000  0.000
union, confidence 0            1.000  0.778        0.679  0.519  0.296
full-page OCR                  1.000  1.000        1.000  0.778  0.556
per-zone OCR                   1.000  0.778        0.679  0.630  0.481

Past 1% of foreign ink on the damaged field, the retained sensor is completely blind. In exchange it never cries wolf, at any level, while the word sensors raise false alarms on 30 to 56% of clean pages at 4%. One of those degrades a gate. The other destroys it.

The sensor stayed, and the reason is written down rather than assumed:

The operating point is chosen on clean pages, deliberately: the four parasite levels exist in equal proportion for statistical power, and choosing a threshold on the pooled set would silently assume that three pages in four carry foreign ink.

What changed is what ships beside the number. thresholds.json now carries recall_with_ink_on_the_damaged_field in the same object as recall, so the required-field entry reads 1.0 and 0.0 next to each other and nobody can quote the first without meeting the second. If your scans come off a dirty glass, full-page OCR is the sensor to prefer, and you pay for it in false alarms. That trade is yours. The measurement is there so you can make it.

Three questions for any benchmark that returns 1.000

Read against my own, on 2026-08-27, and every one of them cost me something before it became a question.

What does the generator never produce? Mine never added ink, and the sensor it crowned was an ink sensor. The blind spot of the benchmark and the blind spot of the winner were the same shape, so the grid could not see it by construction, no matter how many cells it ran.

Was the threshold read on the same draws it is reported on? If yes, the number is optimistic by an amount nobody can recover after the fact. A held-out seed costs one extra run.

Does the failure mode ship in the same file as the score? A limits document nobody opens is not a disclosure. The number and the thing that kills it belong in one object.

Same instinct as the rulebook whose agent refutes its own findings: the work that pays is the work spent trying to break your own result, and a result is worth what the attempt to break it was worth. A recall of 1.000 is not a property of a detector. It is a property of a detector and a generator, and only one of those two is going to meet a real scanner.

Method

Repo read 2026-08-27 at github.com/DylanMerigaud/dossier-preflight, HEAD 42732f0, public, local tooling only and no remote service. Grid: grid/run.py, 384 cells (angle x dpi x JPEG quality x noise) x 3 seeds = 1152 dossiers and 13,824 image readings, 121 minutes on 13 workers; 864 of those dossiers fall inside the nominal domain of 150 dpi and up. Thresholds are written by grid/analyze.py --publish into thresholds.json and LIMITS.md, calibrated on seeds 11 and 23, with the published figure measured on seed 37, never looked at while choosing. Operating point rule: the highest recall holding a false positive rate under 0.2% per target. Two thresholds do not come from their own curve and the file says so: expiry is frozen at zero by definition, and resolution is set at the other checks' readability floor minus 1%. Parasitic ink probe: parasitic-ink-probe/experiment.py, 528 readings on a git archive of commit 3a4e682, measured 2026-08-21, two cells, six seeds, seven intensities plus a no-parasite control, target field employment / City or Town. Fifth-factor replay: grid/run.py --parasite, 27 cells x 3 seeds, commit aa7e0ed. Corpus: three blank forms exactly as their administration publishes them, W-9 (IRS) and I-9 (USCIS) in the public domain, Cerfa 14011*02 under Etalab Open Licence 2.0, with producer, source, retrieval date, licence and sha256 recorded in corpus/CORPUS.md. One fictional dossier, one invented person, one defect per check. No real scan and no document ever issued to anybody entered the measurement, so a recall of 1.000 here is that of the same defect seen 864 times, not of 864 different defects.