Ninety percent of the bill was thinking that did not help

We had 63,000 pages of Afghan lithograph to read and a quote of about $9,000 to read them. That is the kind of number that makes you look for a cheaper model. The cheaper model turned out to be the wrong lever, and the right one was sitting in a config field we had never touched.

The test

Six configurations, each reading the same 102 gold pages once, same prompt, same rendering. Only the model and the reasoning budget varied. Cost came from each response’s own usage metadata at published list price rather than from an estimate, because estimates on this kind of job are usually wrong in the direction you want them to be.

The result nobody expected

Setting the reasoning budget to low on the same frontier model cut output tokens from 11,736 per page to 1,130. That is a tenfold reduction, and cost fell ninefold, from $0.1433 a page to $0.0161. The quality difference, measured pairwise across 93 pages, was −0.0012 with 43 wins and p = 0.53. There is no detectable difference. Roughly ninety percent of what we were paying for was reasoning that did not change what the model read off the page.

For the full job that is $1,012 instead of $9,030.

The result we were expecting, which was wrong

The obvious next move is to also drop to a cheaper model. It does not work, and the way it fails is instructive. Looking at medians alone, the cheap model looks fine: its median score of 0.461 actually edges the expensive one at 0.460. Compared page by page, the expensive model wins 73 of 94 pages with p = 7e-8. Comparing medians of two separate distributions would have picked the wrong model. Only the paired test shows it.

So we asked whether there was at least a subset of easy pages worth routing to the cheap model. There is not. The cheapest model matches or beats the expensive one on only about 21% of pages, and an oracle router with perfect foresight, which nobody can build, would save $209 out of $1,012. A real router captures a fraction of that. Worse, the pages where cheap was good enough are indistinguishable from the rest on every feature you can see before you know the answer: 1,118 output tokens versus 1,198, 633 normalised words versus 651. There is nothing to route on.

What this generalises to

Reasoning budget and model tier feel like the same dial. They are not. On this task the reasoning budget was almost pure waste and the model tier was almost pure signal, and the two moved cost by similar amounts. If you are running a large batch job against a frontier model, test the reasoning budget before you test a cheaper model. It is a config change rather than a quality change, and on our job it was worth eight thousand dollars.

The other lesson is about medians. Two distributions with nearly identical medians can still be separated at p = 7e-8 when you compare them on the same inputs. If you are choosing between models and you are not testing them pairwise on identical items, you are not measuring what you think you are measuring.

The corpus this came from is public. 69,624 pages, read at the configuration this test chose, with the measured error rate and its caveats published alongside.