1. Pruning a model on my own writing
The Problem
My set-up for running models is three GPUs, each with 24 GB of memory - 72 GB in total. The models I most want to run are large mixture-of-experts (MoE) designs, and they need two or three times that. Shrinking their numbers helps, up to a point, but past that point the model gets noticeably worse. And sadly I can’t buy bigger cards.
So one route I’ve explored is to prune the model - delete parts of it - hoping to keep the quality while making the memory manageable. If I use my own documents to calibrate the pruning, the decisions about what to delete get made by the data I actually have at hand, rather than by a bunch of strangers on the internet who chose their own sample.
Mixture-of-Experts
In the models here, each MoE layer holds 128 experts - small self-contained networks - plus a router that chooses which 8 experts run for each token you give it. Pruning deletes whole experts outright.
The method is REAP (Router-weighted Expert Activation Pruning). Each expert gets a score: how often the router picks it, multiplied by how strongly it fires when picked. The lowest scorers in each layer are deleted.
The sample text drives the whole thing. I used five million tokens of my own work - emails, reports, documents, code, and two theses - so the experts my actual work never calls on are the ones that get binned.
Pilot Run
I chose Qwen3-30B for the rehearsal. Its weights are 61 GB, which fits in my 72, so I can already run the stock model quickly in VRAM.
There were two problems with the stock tool. First, it reads only the first few hundred samples of the calibration file, and my file was ordered by domain, so the prune would have been tuned almost entirely on information security documents.
Kind of like sorting the dictionary alphabetically then only picking the first 50 words - it’s going to be all ‘A’s. I got claude to change the code to shuffle the lines according to the ratios of the whole corpus.
Second, the default tool loads the whole model into system memory: 105 GB for this 30B, but 400 GB for the 200B-class models I’d actually like to shrink. I don’t have 400 GB. So I got claude to change the code to fetch the weights from the NVMe drive one layer at a time, as needed.
Pruning only ever looks at one layer at a time, and layers only feed forwards. So each layer’s results can be saved and handed to the next, and every layer still sees exactly what it would in a normal run. Memory use fell to about 46 GB, and stays there no matter how large the model is.
The Cut
I deleted the 32 lowest scorers in each of the 48 layers - a different 32 each time, depending on my usage - leaving 96 per layer. That’s about 7.2 billion parameters gone: 61 GB down to 46.6.
To judge it, I compared the pruned model, compressed to 14.2 GB, with the stock release at 18.6. Writing came out dead even, coding identical, and its judgement about when to hand a question up to a bigger model came out slightly better. And its information security scores dropped six points - the domain most represented in the calibration text got worse. I’m probably terrible at information security.
Calibrating on your own text decides which experts survive the REAPing. Facts stored in the deleted experts are gone, and facts are not neatly sorted per-expert so the model becomes worse across the board. So expect slightly worse and much smaller, and only prune when the freed space is useful: more precision, a longer working window, or fitting at all.
Cut #2 Electric Boogaloo
GLM-4.5-Air is 106 billion parameters and 206 GB of weights, a tier above anything I can run whole. Squeezed in unpruned it would need about four bits per number, where 16 is normal, and a cramped working window. It could either be pruned at five bits with a proper working window, or run stock at the four bits it needs to fit.
The first hour produced three failures. The standard router code reports only its top 8 picks, and the tool needs all 128 scores. The tool’s own files already contained a fixed version - I just had to find it. Then the tool’s internal bookkeeping outgrew a 24 GB card, because GLM’s layers are twice as wide as the pilot’s.
The save stage assumes the whole model can sit in memory while the cut is applied, and 206 GB does not sit in 72 GB of graphics memory plus 128 of system memory. The model’s weights sit across dozens of big files, so I got claude to build a small saver that works through them one at a time: delete the marked experts, renumber the rest, write the file back out, move on. It never holds more than a few gigabytes. 1,440 experts gone, and the files down from 206 GB to 152.6.
Testing
The pruned model sat the exam against its uncut self, judged by fable as in the pilot. The stock model can’t fit even its four-bit weights on the cards, so it ran partly from system memory, and on the two longest reasoning tasks it timed out mid-thought and returned nothing. Those failures were struck from its score.
Results
- Judged writing went 5-4-1 to the pruned model.
- The pruned model led on information security, long documents, honesty and sticking to its sources. On knowing when to hand questions up it matched the original, with fewer false alarms.
- It trailed only on code - which is fine, because code is planned to go to the cloud anyway.
The pruned model runs entirely on the cards at five bits, with a 96,000-token working window, at 46 tokens a second. The file that actually serves answers is 55 GB, from a 206 GB original. Full scores: Full results.
What Next
I went on to prune three more models: a 122B, a 119B and a 201B. The next article covers quantising - how a model gets squeezed down to about four bits per number, and why the 119B made that difficult.
Thanks for reading!