AI model evaluation
Where code stops and a model earns its place
Problem
I built a pipeline to transcribe made-up bedtime stories and label who said what, with one constraint: nothing leaves my local machine, for family privacy. In a companion piece I showed that for real names — ones I could check against a known roster — plain code beat every AI model I tried. This is the other half of the story, and it ends the opposite way.
The hard case is the names a parent invents on the spot. A train engine gets christened mid-sentence; a sidekick is named and renamed three lines later. There is no roster, because the names don't exist until they're spoken. So the only thing that can go wrong is inconsistency: the same invented name's transcription spelled several ways inside one story. That breaks the thing I'm building toward — charting a character across a story — because the tool no longer knows it's the same character. Here is one made-up train engine, named once and spelled four ways:
With no correct spelling to match against, I couldn't ask “is this name wrong?” I could only ask “is this one name wearing four name tags?” So I ran an evaluation to find the most reliable way to catch that.
Action
I built the detector in three stages, each one cleaning up after the last.
Stage one is phonetic detection. It groups every capitalized word by how it sounds, then flags any group that shows up spelled more than one way. This catches the inconsistencies, but it's loud: it also snags ordinary capitalized words that happen to rhyme, so most of what it flags is noise.
Stage two is a dictionary filter. If every spelling in a group is just an ordinary English word, drop the group — it's almost certainly not a name. This cleans out most of the noise. But it has a blind spot: invented names that are also real words. A character called Bibi or Bacchus gets thrown out with the trash, because a dictionary can't tell “Bibi the character” from “bibi the word.”
Stage three is where a small local AI model earns its place. I hand it exactly the residue the dictionary couldn't judge — the groups where every spelling is an ordinary word — with a few example lines, and ask one question: character name, or just ordinary words? Unlike the roster project, where I deliberately held the prompt fixed across all models, here I tuned the prompt to each model, because the whole point was to find the best the small models could do.
Result
I swept 4 local models over the ambiguous groups, 3 runs each. Every one of them was perfectly stable — zero crashes — for a simple reason: the model only ever judges a handful of groups per story, not the whole transcript. (In the roster project, where models read the entire transcript, several of them crashed.) Here, the work fits in memory every time, so the question is purely who judges best.
| Model | Runs on device? | Result on the ambiguous groups |
|---|---|---|
| Gemma, ~2B | yes | Deployable, but a coin-flip on the keep/drop call (0.50 precision) |
| Gemma, ~4B | yes | With a few worked examples in the prompt, a clean sweep1 (1.00 precision, 1.00 recall) |
| Qwen, ~8B | yes | Over-eager — kept almost everything, and the tuning made it worse (0.29 precision) |
| Gemma, ~12B | yes | Needed tuning just to reach a coin-flip; the examples that fixed the 4B confused it2 (0.50 precision) |
The headline writes itself: the small, easily deployable 4-billion-parameter model won outright — and beat the 12-billion one three times its size. That's the mirror of the roster project, where bigger models also failed to justify their weight. The difference this time is that a small model didn't just survive the comparison — it clearly earned its keep.
And it earns it where it counts: end to end. Stacking the model's judgment on top of the code recovers the invented names the dictionary had been quietly discarding. The share of inconsistent names the system catches rose from 46% to 86%, while 88% of what it flags is still correct. The model isn't doing the heavy lifting — the code does that — but it closes a gap the code alone never could.
The choice has honest costs, and they live in 3 cases:
The first cost is the real ceiling: a tool that finds inconsistencies is blind to names that were never inconsistent. The honest read is that 86% isn't 100% because some of the missing names were spelled right all along.
The next step is to give the model the audio and let it listen, not just read the page.
In the companion project, the honest result of the evaluation was that plain code won and the model wasn't worth its weight. This time the same kind of careful measuring landed the other way: the code carries the load, but a small local model earns its keep on the one slice the code can't resolve — and you only learn which way it falls by measuring, not by guessing in advance.
The code behind this (the clustering, the dictionary filter, the model judge, and its scorer) was written with AI assistance, reviewed and edited by me.