4. Choosing my model
The Problem
The pruning and quantising work gave me some larger models I could then test alongside the ones more suitable for my rig. Now I just had to see which was best for me.
Ground Rules
Each candidate got the same 72 GB of graphics memory to spend however suited it: a big model spends it all on weights, a small one uses the spare for higher precision, a longer working window, or helper models running alongside.
A model had to fit entirely in graphics memory with a working window of at least 64,000 tokens, or it was out. Models that spill into system memory run at a crawl and fail their longest tasks by producing nothing.
Everything ran on the same software, the same exam, and the same marking.
Casualties
The biggest project of the series didn’t fit. Step-3.7-Flash, 201 billion parameters, was pruned and quantised down to 65.2 GB - under 72 on paper. Loaded, with its working memory on top, it needed about 75 GB, and only about 70 is usable.
A full day of work failed 30 seconds into loading. Now I check fit first.
A second model only runs on a modified version of the serving software, so it sat out. That left twelve.
Results Without Tools
The first phase ran every model bare: no tools, no reference library. Full tables: Full results. Highlights below.
Among the big models, Laguna - a 117B that needed no surgery, since its maker ships ready-compressed files - took every judged subject: writing 7.2, reasoning 7.8, coding 8.0, at 36.5 tokens a second.
Among the small ones, Gemma-4-31B took the quality prizes and the booby prize at once: best writing (7.3) and coding (8.3), and the worst honesty score of the lot (0.42, with 29 invented facts). It was also the slowest, at 14.5 tokens a second.
Results With Tools
The second phase added the working environment the winner would actually live in: the reference library, web search, a checker that blocks unsupported claims, and permission to hand hard questions up to a cloud model.
Laguna was already at its limit: 65,536 tokens was its ceiling on this hardware, exactly the minimum the rules allow, with nothing spare. So this phase came down to the two Gemmas, and the deciding subjects were the practical ones: honesty, tool use, knowing what to hand up, speed.
The honesty scores moved most. The 31B went from 0.42 bare to a corrected 0.98; the 26B scored 0.96. Corrected, because the automatic marker first said 0.24, and checking every flagged invention by hand found almost none were real.
Most were correct facts the marker didn’t recognise, or refusals worded in ways it didn’t expect. One flagged answer cited a security-vulnerability number the exam’s author had invented - which turned out to be real, published three weeks earlier. Genuine inventions: one per model.
Knowing what to hand up took longest to fix. Left to decide alone, the model answered almost everything itself, scoring 0.04. Rewriting the rules fixed it - certain categories always go up, and anything not clearly simple goes up. The 31B reached 0.72; the 26B reached 1.0, with no sensitive question kept local that shouldn’t have been.
The Verdict
| Measure | Gemma-4-26B | Gemma-4-31B |
|---|---|---|
| Honesty | 0.96 | 0.98 |
| Tool use | 0.80 | 0.65 |
| Spots questions to hand up | 1.0 | 0.72 |
| Speed (tokens/sec) | 65-98 | 31-52 |
| Memory used | 44 GB | 66 GB |
The 26B won. The 31B kept the best writing and coding of the small models, but those jobs go to the cloud anyway, and the 26B led everything the day-to-day job actually involves, at twice the speed and two-thirds the memory.
One footnote: the winner was never pruned. It arrived as its maker shipped it, and every pruned model either didn’t fit or lost the exam. The pruning work still paid its way - without it the big models couldn’t have been tested at all.
What Next
The 26B now runs day to day inside a seven-service stack: the checker, the reference library, a memory layer, and the route up to the cloud. How that fits together is the next article.
Thanks for reading!