The forty pages that lied

We are teaching machines to read Afghan lithographs from the 1870s to the 1930s. To know whether a reader is any good you need pages where you already know the answer, and to iterate quickly you test on a slice rather than the whole set. Ours was the first forty rows of the gold file. For months, every headline number this project produced came from those forty pages.

They were the forty easiest pages in the file.

How it happened

The gold file was written sorted by alignment quality, ascending. That was a sensible thing to do when it was built, because it put the cleanest matches at the top where you could eyeball them. Nobody wrote down that the file was sorted. Later, a different piece of code took the first forty rows and called it a sample. It reads like a sample. It looks like a sample in the code. It is a best case.

The gap is not subtle. Across those forty pages the median gold character error rate was 0.232 and never went above 0.297. Across the other sixty-two it was 0.462. So the test set was not just easier, it excluded hard pages by construction.

What it cost

Every absolute accuracy figure this project published before we caught it is roughly twice as good as reality. That includes a fine-tuned model bakeoff, a comparison against a commercial model, and a cropping experiment whose headline was a 46% improvement. The 46% number is now retired. On a fair sample the same intervention wins 41 pages out of 91, which is a coin flip.

One thing did survive, and it is worth being precise about why. Every arm in those experiments ran on the same forty pages and was compared pair by pair. A shared bias that lifts both arms equally does not change which one wins. So the rankings held up and the absolute numbers did not. That is the useful distinction: a contaminated sample can still tell you which of two things is better, as long as you never quote the number itself.

The second bug, found while reading the first

While re-deriving the results we found that our significance test summed only one tail. Any arm that lost most of its pages came back with p = 1.0, which reads as "no measurable difference". Several cheap-model configurations had been recorded as indistinguishable from the expensive one on exactly that basis. They were not indistinguishable. They were losing. The fix was a proper two-sided test that also reports direction, and the raw win counts had been recorded all along, so everything could be re-derived rather than re-run.

What we changed

Both harnesses now subsample evenly across the difficulty-sorted file instead of slicing the head. We checked the new sampler against the full set before trusting it: median gold error 0.363 versus the full set at 0.364, spanning 0.070 to 0.710. That is what a sample is supposed to look like.

The general lesson is cheap to state and easy to miss. A sorted file plus a head slice equals a best case, and neither half of that sentence looks like a bug on its own. If you take a slice of anything, check the distribution of the slice against the distribution of the whole. It is two lines of code and it would have saved us months of numbers we had to withdraw.

We publish the corrected figures on the archive itself rather than only here. The Afghan Press Archive states its measured error rate, what it was measured against, and what that reference cannot tell you, on every page and in a machine-readable provenance record.