2. Quantising: Selective Lobotomy

The Problem

Pruning got a 206 GB model down to 152. My cards hold 72. Still too big.

Quantising

A language model is billions of numbers, normally stored at 16 bits each. Fewer bits per number means less detail and a smaller file. Quantising stores them at around 4 to 5 bits each, so the whole model shrinks to roughly a quarter of its size. The numbers are compressed in small blocks, with each block sharing one ‘scale’ number.

Precision is lost, but the loss doesn’t have to fall evenly: some numbers barely matter to the answers, some matter a lot. The work is deciding which ones keep their precision, and shortening the ones that don’t.

Importance Survey

Before compressing, the model reads the same five million tokens of my own writing that chose the prune, and the tool records how strongly every number takes part. The compressor then protects the numbers that got flagged. The ones that weren’t used - or weren’t used much - get shortened.

So the process watches the model work through the same text twice. The first pass scores whole experts and feeds the prune. The second scores the individual numbers inside the remaining experts and feeds the compressor. The first is too rough to guide the compressor, and it describes a 128-expert model that no longer exists after the REAP, so it has to run twice. On the pilot both passes took hours; on the bigger models the second alone runs overnight.

What Stays Full-Sized

Some parts are never compressed hard. The router - the part that picks which experts answer each token - keeps full precision, along with a few small housekeeping parts: about 50 MB in a 46 GB model. Every token’s expert choice flows through the router, so compressing it risks a lot to save almost nothing.

The process also stays at 16 bits until the final cut, so the compressor works from original numbers.

Mistral

Mistral-Small-4, 119 billion parameters, shipped with no 16-bit original. Mistral released only an 8-bit version, and this caused issues with my process.

The observation pass died twice, at different points each time. The fault turned out to be in the shortcut the tool used to read the huge files, and on the way I found one block of numbers in this model is exactly 2 GiB - sitting right on a known Windows limit for reading a file in one go. I got claude to swap the shortcut for plain reads, a chunk at a time, and because results were being saved as it went, the third run’s partial output survived its own crash.

The strangest failure was a NaN - ‘not a number’, a broken value that poisons every calculation it touches. A rough working copy of the model, needed just to run the survey, produced one at layer 33. The numbers themselves were fine everywhere I looked; the NaN only appeared when the model ran. That working copy had been compressed to 4 bits, too rough for numbers that started life at 8 bits, and a 6-bit copy fixed it. The usual advice is that any working copy will do for the survey; this model said no.

Compromises

I took two shortcuts:

The survey for the 106B model ran on an 8-bit copy rather than the full-sized original, because the original no longer fits on the cards. 8 bits is close enough that the survey comes out the same.

The survey text was capped at 1,000 sections of the full 4,858, cutting one model’s measuring pass from about 21 hours to under 5. Survey quality stops improving well before 1,000, but the calendar made the decision.

What Next

The process could now take a model from download to a compressed, personalised file that fits my cards. It couldn’t say whether any of them were any good. The next article covers the exam I wrote to find out.

Thanks for reading!