Every night, fifteen handwritten pages are transcribed and compared against what is really on them. Over twenty-eight runs the middle page comes back at 1.5 % character error. But the average hides the finding: ordinary prose and lists are read almost perfectly, while crossed-out corrections, notes in the margin and anything drawn rather than written are read badly enough to be useless. Which backend you point at them is the other half of it — the one built into macOS, which costs nothing, reads the same pages at 15.7 %.
Character error rate per page, in percent — each cell the median of the runs recorded so far, not a single night: twenty-eight for the first three columns, over twenty-five nights of which three ran twice. Lower is better; 0.00 means every character matched. The first three columns are paid vision services, all reached the same way. The fourth is Apple's built-in recognition, which joined on 9 September 2026 and has read every page to the same figure on each of its nine nights, so its cells are that reading; its median here is over these fourteen rows, which is why it is higher than the fifteen-page figure in its own section below. The fifth is the model the plugin downloads, Qwen3-VL-8B, read hosted at 16-bit precision so that it can run every night beside the others; it joined the same night as Vision and its cells carry six to eight nights each, because its host rate-limited some pages away. A per-night figure swings enough to reorder the paid three from one night to the next, which is a fact about the measurement rather than about the models; the series is what survives that. Latest night , and every night of it is in the CSV.
| Page | Claude Sonnet 5 | Gemini 2.5 Flash | GPT-4o | Apple Vision | Qwen3-VL-8B |
|---|---|---|---|---|---|
| Clean prose | 0.00 % | 0.00 % | 0.00 % | 12.40 % | 1.40 % |
| Rushed prose | 0.00 % | 1.00 % | 2.20 % | 29.20 % | 10.10 % |
| Bulleted list | 0.51 % | 1.02 % | 0.51 % | 13.78 % | 0.51 % |
| Numbered list | 0.96 % | 0.00 % | 1.28 % | 6.09 % | 1.28 % |
| Table | 3.07 % | 3.68 % | 3.07 % | 71.78 % | 4.29 % |
| Formula | 2.38 % | 3.57 % | 25.30 % | 26.19 % | 1.79 % |
| Mixed language | 0.00 % | 0.72 % | 0.00 % | 12.89 % | 0.24 % |
| Diagram | 35.42 % | 18.75 % | 25.00 % | 56.25 % | 29.17 % |
| Nearly empty | 0.00 % | 0.00 % | 0.00 % | 40.91 % | 0.00 % |
| Dense | 0.59 % | 0.71 % | 0.71 % | 24.85 % | 4.99 % |
| Numbers | 3.48 % | 4.20 % | 3.19 % | 15.65 % | 3.48 % |
| Special characters | 2.10 % | 2.10 % | 3.08 % | 11.48 % | 4.62 % |
| Corrections | 18.58 % | 11.50 % | 14.60 % | 22.57 % | 15.49 % |
| Margin note | 18.97 % | 19.49 % | 4.62 % | 14.62 % | 6.15 % |
| Median | 1.53 % | 1.56 % | 2.63 % | 19.11 % | 3.89 % |
Four pages come back without a single wrong character. Bulleted and numbered lists arrive as lists, and the table page arrives as a table — the structure survives, on all four models. Then three pages fall off a cliff, and they fall for every model: what is crossed out, what sits in the margin, and what was drawn instead of written.
Two results are worth more than the median. Pages 01 and 02 carry the same text, written once carefully and once in a hurry — and Claude reads both at 0.00 %. Legibility, the thing everyone assumes is the variable, was not the variable here. Meanwhile GPT-4o reads the margin note four times better than the other two and the formula seven to ten times worse, which is a reminder that one number per model is already a simplification. Across the series Claude and Gemini are 1.53 % and 1.56 %: on this corpus they are not telling apart, and any ranking between them belongs to a single night rather than to the models.
The fifth column is there to be checked against something, and it has been. The same Qwen3-VL-8B weights, run on one Mac on 10 September in the 4-bit build the plugin actually downloads, read these fourteen pages to 2.49 % against the hosted column's 3.89 % (1.79 % over all fifteen, which is the figure the local page quotes). Cell by cell the two agree within a point on twelve pages and within two on the margin note; only the table parts them, 0.61 % local against 4.29 % hosted. The medians differ because a median of fourteen is the mean of the seventh and eighth values, and here those sit on the steep part of the list — not because the two read differently. So a hosted row is a fair stand-in for the download, and a day the two part company will show up in this paragraph before it shows up in the table.
Four more models joined the nightly on 9 September and have no column: over the same fourteen pages Gemini 3.1 Flash Lite reads 1.76 %, Qwen3-VL-235B 2.24 %, Gemma 4 31B 2.84 % and Qwen3.6-35B 3.59 %, each over seven or eight nights. None of them beats the column it would replace — Flash Lite is worse than Gemini 2.5 Flash and costs four times as much a night — and a sixth column is where this table stops being read. Every one of their nights is in the CSV.
Every page was written to answer one question, and the image below each heading is the picture the backend was actually handed — the same PNG, produced from the same committed file by the same code. Input, expected text and result are all public, so any row here can be reproduced or disproved by a stranger.
The baseline: unhurried handwriting on ruled lines, nothing unusual on the page. If a backend cannot read this it can read nothing, and the number here is the floor everything else is measured against.
Claude Sonnet 5 0.00 % Gemini 2.5 Flash 0.00 % GPT-4o 0.00 % Apple Vision 12.40 % Qwen3-VL-8B 1.40 %
The same words as page 01, written in a hurry. The pair exists to answer one question — how much does legibility cost? — and the answer differs by backend: the paid services barely notice, while the model you run at home reads it seven times worse.
Claude Sonnet 5 0.00 % Gemini 2.5 Flash 1.00 % GPT-4o 2.20 % Apple Vision 29.20 % Qwen3-VL-8B 10.10 %
Hand-drawn bullets, one level. Whether the dashes come back as a list or as run-on prose is the simplest test of structure surviving the trip, and it is scored separately from the characters.
Claude Sonnet 5 0.51 % Gemini 2.5 Flash 1.02 % GPT-4o 0.51 % Apple Vision 13.78 % Qwen3-VL-8B 0.51 %
Numbered items with sub-items indented under them. Two levels, so the question is not only whether the list survives but whether its shape does.
Claude Sonnet 5 0.96 % Gemini 2.5 Flash 0.00 % GPT-4o 1.28 % Apple Vision 6.09 % Qwen3-VL-8B 1.28 %
A small table drawn by hand. The hardest structure on the page set: a reader that returns every word correctly but lays the cells out in the wrong order scores catastrophically, which is why Apple's recognition reads 72 % here and 12 % on plain prose.
Claude Sonnet 5 3.07 % Gemini 2.5 Flash 3.68 % GPT-4o 3.07 % Apple Vision 71.78 % Qwen3-VL-8B 4.29 %
Mathematical notation: superscripts, fractions, symbols that are not letters. It is here to catch a backend that quietly turns notation into ordinary text — a squared sign into the digit 2, an ellipsis into three dots.
Claude Sonnet 5 2.38 % Gemini 2.5 Flash 3.57 % GPT-4o 25.30 % Apple Vision 26.19 % Qwen3-VL-8B 1.79 %
German and English in the same paragraph, the way notes are actually written. A backend that decides on one language per page will correct the other one into nonsense, fluently.
Claude Sonnet 5 0.00 % Gemini 2.5 Flash 0.72 % GPT-4o 0.00 % Apple Vision 12.89 % Qwen3-VL-8B 0.24 %
Four boxes and arrows with a word in each. Almost no text and the worst numbers on the page — everything drawn rather than written is where every backend fails, and this page is why the drawing is kept beside its transcript in the note rather than replaced by it.
Claude Sonnet 5 35.42 % Gemini 2.5 Flash 18.75 % GPT-4o 25.00 % Apple Vision 56.25 % Qwen3-VL-8B 29.17 %
One short line and nothing else. It exists to show what a percentage does to a tiny page: the paid services get every character, and the one mistake Apple makes is 41 % of a page that is 22 characters long.
Claude Sonnet 5 0.00 % Gemini 2.5 Flash 0.00 % GPT-4o 0.00 % Apple Vision 40.91 % Qwen3-VL-8B 0.00 %
Small writing, packed tight, edge to edge. The test of resolution rather than of handwriting: a backend that shrinks the image before reading it loses this page first.
Claude Sonnet 5 0.59 % Gemini 2.5 Flash 0.71 % GPT-4o 0.71 % Apple Vision 24.85 % Qwen3-VL-8B 4.99 %
Figures, dates and units. Numbers are the one thing on a page where a misreading cannot be caught by reading: a wrong word looks wrong, and a wrong digit looks exactly like a right one.
Claude Sonnet 5 3.48 % Gemini 2.5 Flash 4.20 % GPT-4o 3.19 % Apple Vision 15.65 % Qwen3-VL-8B 3.48 %
Umlauts, quotation marks, dashes of three lengths. Half of this page is about the backend and half about the comparison itself, which normalises text before scoring but deliberately never flattens a dash into a hyphen.
Claude Sonnet 5 2.10 % Gemini 2.5 Flash 2.10 % GPT-4o 3.08 % Apple Vision 11.48 % Qwen3-VL-8B 4.62 %
Words crossed out and rewritten, the ordinary contents of a working notebook. Everything reads it badly, and it is not obvious what the right answer even is — the ground truth keeps what was meant, not what was struck.
Claude Sonnet 5 18.58 % Gemini 2.5 Flash 11.50 % GPT-4o 14.60 % Apple Vision 22.57 % Qwen3-VL-8B 15.49 %
A thought squeezed into the margin beside the main text. The clearest counterexample on the page: GPT-4o reads it at 4.62 % and the open-weight model at 6.15 % where Claude and Gemini are near 19 %, which is why a single "best model" is the wrong way to read this table.
Claude Sonnet 5 18.97 % Gemini 2.5 Flash 19.49 % GPT-4o 4.62 % Apple Vision 14.62 % Qwen3-VL-8B 6.15 %
Apple Vision's figures are its reading on a pinned macOS runner, identical on every night up to ; the other four are the median of the nights recorded so far. The four backends without a column are in the CSV.
A fifteenth page joined the set on 8 September 2026: one written the way people actually write on these devices, scrolling down as the page fills, four to five screen heights in all. It is not in the table above and not in the median, and the reason is worth more than the row would have been.
| Page | Claude Sonnet 5 | Gemini 2.5 Flash | GPT-4o |
|---|---|---|---|
| Read as one image | 11.30 % | 3.65 % | 39.53 % |
| Cut at its blank bands | 6.31 % | 1.66 % | 3.99 % |
Three models that sit within a point of each other on the other fourteen pages landed at 3.65 %, 11.30 % and 39.53 % on this one — a factor of eleven between the best and the worst, on a single image. That spread was not a reading result. The page rasterises to 852 by 7469 pixels and every model shrinks it before looking: Claude received about 294 by 2576, GPT-4o about 234 by 2048. What was being measured there is how each service handles a very tall image, not how well it reads handwriting.
Which is a thing that can be fixed, and now is. Since 10 September the plugin cuts a page like this where there is no ink — at the blank bands between blocks of writing — and reads each piece on its own. Nothing is cut through a letter, and an ordinary page is untouched: only writing that runs more than three times its own width is split at all, and the widest ordinary page in this set does not come close.
The second row is the first night after that shipped. Every backend improved and none got worse; GPT-4o went from unreadable to better than its own median. It is the largest single improvement any change has made to these numbers, and it came from handing the model a better picture rather than from a better model.
The page itself, scaled to fit. At its real size it is four to five screen heights — the original is what the models were given.
Apple's built-in text recognition is what a fresh install on a Mac uses, and it was the largest gap on this page until 9 September 2026, when it joined the nightly on a pinned macOS runner. It reads the same fifteen pages at a 15.7 % median character error rate — against 1.5 % for the paid services above. Roughly ten times as many wrong characters, on the configuration that costs nothing and sends nothing anywhere.
It has nine nights behind it and has read every page to the same figure on each of them, so the runner's number is exact and says nothing about anyone else's Mac. The magnitude is not in doubt: a second Mac, running the same version of Apple's engine, read the same pages at 17.4 %. The two disagree by nearly two points and agree completely about the order.
That two-point spread is itself worth stating. Apple's model ships inside macOS and picks its own version per request; nothing in the plugin can pin it. So this figure is dated, and the operating system it was measured on is recorded beside every row in the series — a macOS update can move it with nothing changed here, which is not true of any other row on this page.
It is not a bad backend. It is free, it needs no download and no key, it reads a page in a fraction of a second, and its mistakes usually look wrong on the page rather than reading as fluent prose that says something you did not write. It is the honest cost of the default, and now it is measured rather than assumed.
Fourteen pages were written on a reMarkable on 23 August 2026 for this purpose, and a fifteenth on 8 September — real handwriting, because a generated page measures nothing about reading handwriting. One trait per page, so the error can be attributed rather than averaged away. Every text on them is invented and harmless, which is why the whole set is public, alongside its ground truth. Three words came out differently while writing and were adopted into the ground truth, so the comparison is against what is on the page rather than what was intended.
The score is character error rate: the number of single-character edits needed to turn the transcript into the ground truth, divided by its length. Both sides are normalised to Unicode NFC first, and deliberately never NFKC — NFKC rewrites a superscript into a digit and an ellipsis into three dots, which is exactly what two of these pages exist to measure. Whitespace runs collapse. What survives all that is a reading error and nothing else.
Structure is recorded separately from the score, and the reason is a result in itself: a table returned as flowing prose measures 41 % character error even when every single word was read correctly. Mixed into one number it is indistinguishable from a model that cannot read. So the table page carries a verdict of its own — ok where a table came back, lost where it did not — and its percentage should be read together with that verdict rather than on its own.
Apple's recognition is the worked example. It scores 71.78 % on the table page against Claude's 3.07 %, and its verdict is lost. What it actually returned was all eighteen cells, read down the columns instead of across the rows: every place, then every height, then every duration, with no table around them. Put those same eighteen cells in the order Vision read them in and the error is 10.4 %. Nearly the whole distance between 10 % and 72 % is the order of the cells, not the reading of them.
The score is not adjusted for that, and will not be: a measurement that gets softened when a result is unflattering has stopped being one. It is said here instead, because it is the kind of thing a single percentage cannot say. On a page with a table on it, character error rate scores agreement about layout before it scores reading — which is the same finding as the 41 % above, arrived at from the other direction.
This has run every night since 19 August 2026. Across those nights Claude's single-night median has moved between 1.0 % and 2.6 %, which is the honest width of the figure at the top of this page — and the reason that figure is the median of the nights rather than the last one. Every night of it is in the series as a CSV — one row per page per backend per night, written by the run itself, including the nights that went badly.
Because the alternative is a number with nothing behind it, and every tool in this corner of the market already has one of those. A measurement you can rerun against pages you can read is worth more than a percentage in a feature list — including to me, since the nightly is what tells me when a change made the reading worse.
It also decides how the plugin is built. The page you wrote is embedded directly above its text in the note, and that is not decoration: with a diagram page reading at 22 % to 38 %, the drawing has to stay next to whatever was made of it.
Fifteen pages of somebody else's handwriting can only tell you so much — your own notebook settles it in a minute. Start on the Mac backend that costs nothing, knowing it is the 15.7 % one, and decide from your own pages whether the difference is worth a download or a key.
Desktop only — Obsidian for Mac, Windows or Linux. Doesn't open? See it on obsidian.md, or in Obsidian: Settings → Community plugins → search Tagged Sync.