3. Writing an exam for language models

The Problem

After pruning and quantising I had a set of candidate models and no way to rank them. Public benchmarks exist, but the models have usually seen them during training, and they say little about my actual work.

My first attempt at my own test was too small. Some subjects had three questions, others nine, so a single miss moved a score by a third. It lacked fidelity.

Benchmarking

A benchmark is a fixed set of questions with known answers, put to every model under the same conditions and scored the same way. Any difference in scores is then a difference in the models.

Making one trustworthy is a matter of having enough questions, answers checked before the exam is set, and no leaking questions into anything the models were tuned on.

The Questions

The rebuilt exam came to 426 questions. The subjects: security and data-protection knowledge, coding, writing, reasoning, honesty about what the model doesn’t know, very long documents, and when to use tools.

I got claude to write the questions - six of them working in parallel, each checking its own answer key before a question was accepted. Coding questions run a reference solution. Document questions quote their source word for word. And so on.

Questions about recent events had their answers locked in from the live web at the time of writing - the World Cup final, Wimbledon, Eurovision, the Prime Minister (oops). Encyclopedia answers were checked to exist word-for-word in my own reference library, so a model using the library can genuinely find them.

Two more rules. The questions had to stay clear of the five million tokens used for the pruning, so nothing a model was shaped on appears in its exam.

And a few questions got reworded when I noticed the right answer hiding inside a wrong one: “breached” sits inside “not breached”, so an automatic marker matches both.

Marking

Written answers are scored by 17 copies of fable running in parallel.

A second pass checks the marking afterwards. One suspected marking error turned out to be four models writing exactly the same one-line fix.

Honesty is scored with “I don’t know” in mind. Admitting ignorance on an unanswerable question earns the point; a confident invention loses it.

Faults

The exam itself was wrong three times.

1. The question files were saved in one text encoding and read in another. A single character - an em-dash - crashed six of the eight subjects, at identical places for both models. The identical location gave it away: the problem was in the exam paper.
2. The coding marker turned true/false answers into crashes, silently scoring zero for every model on any yes/no question. Because it hit every candidate equally, it never stood out.
3. The answer budgets were tuned on one model, and a slower-thinking model got cut off mid-reasoning. One subject scored 0.45 when cut off and 0.77 with room to finish.

Two judgement calls remained. One vendor’s recommended settings produced ten minutes per answer - faithful to their documentation, and useless - so budgets settled at five times the originals.

And the browser subject was dropped entirely when its scores refused to settle.

What Next

With the exam trustworthy, the models could take it. That’s the next article; scores in Full results.

Thanks for reading!