The intervention that failed usefully

The most obvious way to improve machine reading of a dense newspaper page is to stop asking the model to read the whole page at once. Detect the columns and blocks first, crop them, read each one separately. Everyone who has looked at a page of nineteenth-century Afghan newsprint has this idea within about a minute.

We built it. On a fair test it produced no accuracy gain at all, at about eleven times the cost. And we shipped it anyway. Here is why both halves of that are true.

The clean test

All 102 gold pages, whole-page against layout-first, same prompt, same model, same pages. The only variable was the cropping. Paired across the 91 pages both arms scored, the median difference was −0.0035 in favour of not cropping, layout-first won 41 of them, and the sign test came back at p = 1.0. That is a coin flip, and arguably a slight loss.

An earlier version of this experiment had reported a 46% improvement. That number came from the forty easiest pages in a difficulty-sorted file, and it is now retired.

The part the average hides

Whole-page reading produced no scoreable output at all on 8 of the 102 pages. Not a bad reading. Nothing. Layout-first scored all 102, and those eight came in at a perfectly respectable median. It failed on zero pages where whole-page succeeded.

That changes the accounting. If you score a total failure as zero rather than excluding it, which is the only honest way to look at it when you are reading a whole collection, median yield goes from 0.376 to 0.406. About eight percent of the corpus moves from nothing to a usable read. The median across the pages that both methods could handle is still a coin flip. Both facts are true at once, and only one of them shows up in the headline number.

What we actually shipped

Not the default. Cropping means roughly eleven model calls per page against one, so making it the default would have multiplied the bill by an order of magnitude to move the median by nothing. Instead it is a fallback: read the page whole, and if the output comes back empty, too short, or failing its sanity check, re-read that page with cropping. That buys the entire eight percent rescue at roughly eight percent of the cost.

The general shape is worth naming, because it comes up constantly in this kind of work. An intervention with no effect on the median can still be the difference between having a page and not having it. Averages are the wrong instrument for finding that out. Look at the failures separately, count them as zero rather than dropping them, and ask what the intervention does to the tail rather than to the middle.

It is also a reminder that "it did not work" and "do not build it" are different conclusions. This one did not work at the thing we built it for, and it is in production for something else.