The best model for reading a page depends on the script

We needed to pick a model to read 69,000 pages of Afghan print. The sensible way to do that is to benchmark the candidates on Perso-Arabic text and take the winner. We did, and it nearly cost us the whole project, because which model wins depends entirely on which script you test.

Three benchmarks, four models

We scored four Gemini models against human ground truth on three different kinds of Perso-Arabic page. Clean typeset naskh, from the OpenITI corpus. Handwritten nastaliq manuscript, from OpenITI MAKHZAN with line-level human transcription. And the thing we actually cared about, real lithographed nastaliq from the Afghanistan Digital Library.

On clean naskh print, the 2.5 generation wins by roughly three times, and 2.5-flash ties 2.5-pro at flash prices. That is a clear, comfortable result, and it points straight at the cheap model.

On nastaliq the ranking inverts. Gemini 3.x wins decisively on the manuscript benchmark and stays ahead on the real lithographs. And 2.5-flash, the model the first benchmark recommended, is the single worst reader of the four on lithograph nastaliq by a wide margin.

Why this was nearly expensive

Our pipeline had already distilled a cheap student model into 2.5-flash, on the reasoning that it matched the expensive teacher. It did match, on a benchmark where the teacher was also a 2.5 model. That comparison was circular. Independent gold showed the student was the worst available reader on the script we were actually reading. The whole distillation target was wrong, and no amount of additional training data would have fixed it, because the ceiling was the base model.

The tell we missed for months is that our accuracy work and our model choice were validated against each other rather than against anything outside the system.

What to take from it

Script is not a detail of a text-recognition benchmark, it is the axis the result turns on. Naskh and nastaliq are both Perso-Arabic and both read right to left, and a model can be three times better at one and worst-in-class at the other. If you are choosing a model to read a specific corpus, benchmark on that corpus, or on the closest public material to it, and treat a result from adjacent material as a hint rather than a decision.

The second lesson is about who grades. If your candidate and your reference come from the same model family, you have measured family resemblance rather than accuracy. Build the answer key from a different vendor, or from real human transcription where any exists.

One caveat on the numbers

The absolute error rates on the lithograph column are inflated, because the reference is a modern printed edition and part of the gap is editorial rather than a misreading. The cross-model ranking is the decision-relevant signal, and it is consistent across all three benchmarks. We would not quote the absolute figures without the caveat, and we say so on the archive itself.