Somebody Changed The Glasses

The Mac updated itself while I slept, and in the morning my reader read differently.

Not a little. Naina, the document reader I am building so that a machine of mine can read any page without sending it anywhere, had scored 1,651 real pages the day before. I ran the same pages again with the same code and the same weights. On the smallest tier 1,595 of the 1,651 outputs changed, and its text error went from 0.248 to 0.309. The middle tier changed on 1,470. Newspapers got worse. Papers got worse. Every kind of document but two got worse. I had touched nothing.

The suspect I wanted. I had, in fact, changed something the day before: the resampler, the code that shrinks a page before the detector sees it. It was the obvious culprit and I wanted it to be guilty, because a bug of mine is a bug I can fix. So I checked out the exact tree from before the change, built it, and ran it under the new operating system. It agreed with my new code on 28 of 30 sampled pages. It agreed with its own output from the day before on none. Over all 1,651 pages the operating system had moved 1,470 and my code had moved 25, and on those 25 the words were the same; only the order of a few lines had changed. The refactor was innocent. The ground had moved.

Whose eyes. Here is what I had not understood about my own program. When the reader loads a model, the runtime underneath it asks the machine for every accelerator it has, and on a Mac that means the Neural Engine, reached through a framework that ships with the operating system. I wrote yesterday that this path was slower than the plain CPU on every one of Naina's graphs, and I left it on anyway, because it was the default and it was not hurting the numbers. It was doing something else. It was lending them. The answers belonged to that framework's version, not to my code, and when the OS moved, the answers moved with it. Somebody had changed the glasses on my reader, and I had not noticed because I had never looked at whose glasses they were.

A number you cannot reproduce after a software update was never your number. You were borrowing it.

The proof. I ran the smallest tier again with the accelerator refused, every graph on the CPU cores. Text 0.2477. Reading order 0.492. Formulas 0.799. Every figure equal to the old row to four decimals, and the 1,651 output files identical to the old run, byte for byte, every one. The same tier, the same night, with the accelerator allowed: 44 files matched. So the old rows had been CPU numbers all along. Under the old OS the accelerator had simply given the CPU's answers, and the new OS is where they parted. The CPU run was also faster, 184 milliseconds a page against 531, but the machine was doing other things at the time, so I will not publish either figure; the direction agrees with what I measured yesterday on a quiet machine. The other two tiers were rerun the same way before morning: the middle one matched its old row exactly, 1,650 of 1,651 files, and the largest came in at 0.173 with no empty pages. Their rows are in the repository, beside the command that made them, not here.

The second trap. The update had one more thing in it. It reset the licence on the developer tools, and until I accepted it the compiler refused to compile anything. My build command produced nothing and said nothing. Then my test runner ran the old binaries still on disk and reported fifteen of fifteen passing. I had edited the code and the tests were green, and for ten minutes I believed the edit was in. The file times gave it away: not one object newer than my change. Green is not a result. Green is a report, and a report can be about yesterday.

What I decided. The CPU is the reader's default now. The accelerator is there for anyone who asks for it by name, and nobody gets it by accident. Every published row says which provider ran and which operating system it ran under, and the rows from before the update are marked for what they were. If Apple's path ever earns its way back, it does so under the rule everything here lives by: twice as fast on the same machine, measured, or it stays off.

I do not know why the old framework agreed with the CPU so exactly, or what the new one does differently inside. I have not looked, and I will not pretend. What I know is what I can reproduce, and tonight that is one row, from eyes I can account for.

The reader has one pair of eyes now. They are mine.