The Stopwatch Was Honest

Yesterday I watched a number and believed it.

I am building a reader called Naina, so that a machine of mine can read any page without sending it to a cloud. It has three sizes. On the same eighteen pages, the largest ran twenty-two times slower than the smallest. I wrote that down as a finding. I called it the target. I went to sleep with a plan to attack it.

This morning it was not true, and the stopwatch had not lied once.

What the stopwatch saw. The overnight run had timed 1,651 pages while, in the same minutes, another job downloaded a few hundred megabytes of weights, a second copy of the reader chewed through a demo set, and the evaluator scored what both copies produced. The file timestamps put the demo run wholly inside the big run. None of that was in the number. The number was just a number: 649 milliseconds a page.

Alone, on the same page, the small reader takes about a hundred milliseconds. Same weights, same code, same machine. The last three hundred pages of the overnight run, after the other jobs had finished, had already fallen to about 300. I had the answer in the log and did not look.

A stopwatch does not know what it is timing. That is your job.

Where the time really goes. Measured this morning on a quiet machine, seven pages chosen to span the whole run, each read warm three times. The large reader is 8.8 times slower than the small one, not 22. Its detector is 8.5 times slower. Its layout and recognition together are 8.8 times slower. Recognition runs one text line per call, at 0.8 milliseconds a line for the small model and 8.4 for the large. That is what models seventeen to thirty-five times larger cost when they run on the CPU, one line at a time. There is no mystery left in it. There is only the arithmetic I refused to do the night before.

The chip that was sitting idle. This Mac has a Neural Engine. The runtime asks for it on every session and the frameworks load, so I had assumed the graphs were on it. Measured: in the provider's default format, the Neural Engine path is slower than the plain CPU on every one of Naina's graphs, between 1.3 and 4 times. In the newer format it is 1.5 times faster on the large detector and 1.9 times faster on recognition, and it cannot compile the layout model at all. I checked for a compile cost on every new page shape and found none. So the reader has been running on the CPU cores while the accelerator beside them did nothing, and it took a bad number to make me look.

What stands. One measured row, dated 23 September 2026, on an Apple M5 Pro: 1,651 real pages, small reader, text edit distance 0.248, reading order 0.492, tables and formulas not attempted by construction, 367 milliseconds a page on a desktop someone else was also using. I ran it twice; the output was byte for byte the same. Two larger rows follow tonight. The comparison to the systems it wraps is written in the repository, not here, because a number without its command beside it is a claim.

What I decided. If the silicon has more to give, we go and get it. New kernels if the vendor's runtime will not take our shapes. A path under the framework if the accelerator is only reachable that way. A driver, if it comes to that. The rule that keeps this from being bravado: a measured gap of two times or more between what ships and what the hardware can do, on the same machine, or we leave it alone.

I keep two lessons from the night. The first is old and I already knew it: measure. The second I had not felt in my hands until now: the measurement was fine. The room it was taken in was not.

Clear the track before you time the runner.