Local AI - V1

The Problem

I have three graphics cards in one computer: a Nvidia 4090 and two Nvidia 3090s, with 72 gigabytes of memory between them. For most work that is ample. But for running a large language model at home it falls into an awkward gap.

The models that match the good cloud ones need hundreds of gigabytes. I did load one of the giants, by shrinking its numbers to two bits each - where they are normally up to 16 bits each - and even then it was an 81-gigabyte file that ran at about two words a second while lacking all of its intelligence. That is too slow to use. The two biggest I have tried, at 142 and 160 gigabytes, will also technically run by spilling over into system memory, at about one word a second.

A small model is the other extreme. One fits with room to spare and answers quickly, and it leaves most of the hardware idle. It also cannot be trusted on its own. It invents facts, with no sign of knowing it has done so, and no way to check itself. Filling the rest of the space with KV cache - the working memory it reads while answering - just gives it more information than it can reason over reliably.

The three cards will not combine into one large card either. They sit in slots of different widths, and splitting a single model across them is more trouble than it is worth. It’s like having one connected via a 16-lane road, one via a 4-lane road, and one via a single-lane road. Merging is a nightmare and slows everything to a crawl. So I was left in the middle: too big for one small model, and too small for one large one. But I want to make the most of my hardware.

The Idea

The way through was to stop looking for a single model and build a team of small ones instead.

If no small model can do the whole job, each can take a part of it. One plans and reasons. One finds the source material. One checks that the claims match the material. A mind, a pair of hands, and a conscience - like three wise men - each on its own card running at high speed.

The important decision was how they check one another. The obvious method is to have several models answer and take the majority, but that works less well than you’d think. Models trained on similar material tend to make the same mistakes, and a vote among them settles on the wrong answer with confidence. It turns out LLMs suffer from group-think too.

Instead, every claim in a draft is checked against the source it came from, and a claim the source does not support is dropped. This is also how claims in police reports are (or should be) made - with reference to evidence.

The Build

The team ended up as four models across the three cards. Sadly I couldn’t quite make it 1-1 to keep my three wise men. One has a split personality.

The ‘head’ is Qwen3.6-27B, on the 4090. It reads the question, decides what needs finding, and writes the final answer. At four bits it takes about 21 of the card’s 24 gigabytes - that includes a 96,000-token working window - and writes about 36 tokens a second. The ‘hands’ are Holo3-35B, a model built to drive a web browser, on one of the 3090s; it alone can click, type, and read pages back. Notably, it has vision built-in, which lets it navigate a wider range of pages without difficulty.

The last 3090 holds two smaller models of eight billion parameters each - both from the IBM Granite family. One is an answerer that writes only from text it has been handed, never from memory. It is effectively a very accurate summariser. The other is a verifier, trained for a single task: to read a claim beside its source and mark it sourced or unverified. Anything marked unverified is blocked. This dramatically reduces hallucinations - one of the biggest weaknesses of small models.

Each model is pinned to one particular card so they do not tread on one another’s memory space. The models are limited to four bits rather than the usual sixteen. At that size the ‘head’s’ numbers alone would overflow a 24-gigabyte card before it could even read a single word. So it’s limited. This is the same trick as the map coordinates in Giving a local AI a reference library - shorter numbers simply take up less space! But at the cost of accuracy.

The layout of the machine decided where each model went. The card that also draws the desktop always uses a few gigabytes of its memory to store the windows, background, any videos being streamed, and so on. So it took the two small models. The empty card took the largest model; it sits in a single-lane slot that slows a model as it loads but not as it runs, so the model that loads once and stays put belonged there.

Testing

I tested the team two ways:

• First, I took a case I had already worked through, so the ‘best answer’ was known, and set the team the same task. This involved searching the web for information about a subject. To measure success, I compared the number of findings I made the first time with the number of findings made by the new team.

• Second, I ran a blind comparison of reasoning against Claude, both working the same task bundles, marked without the grader knowing which was which. The bundles were two research cases: a small one (18 entities, 17 documents) and a bigger one (44 entities, 63 documents, about 12,000 tokens of material). Rather than comparing how diligent the model is (as it often halted prematurely), this measured how accurate it was over multiple rounds of reasoning.

I also checked the verifier on its own, feeding it claims I knew were unsupported to see whether it caught them.

Findings

The team was actually… only ok. On the known case it reached most of what I had, which is more than any of its models could have managed alone. So the team architecture definitely helped, but not enough to replace the cloud.

1. The verifier stack mattered more than the reasoning model size when it came to accuracy. The benefit comes from the arrangement of the models, more than from any one being large. A small answerer watched by a good checker is better than a larger model marking its own work. The ‘head’ was actually the cleanest citer of every model tested - about one unsupported claim across both cases - because it fails by leaving things out rather than by inventing them.

2. Claude obviously still won on reasoning. On the blind comparison it scored 81; my ‘head’ model managed 55. The best local score, 69 (nice), came from a 355-billion-parameter model that needs all three cards plus most of my system memory and runs at about one word a second - an overnight tool, not something that can sit in the loop. So the gap between Claude and the stack as it actually runs is 26 points, and buying back even half of that locally means tying up the whole machine.

3. On the small case the ‘head’ scored 87 - quite good! But on the large one it collapsed to 24, missing the important connective findings entirely. Small models can reason over a small pile of evidence; they cannot hold a real one in their heads.

4. The browser model needed much more context than I initially expected. It didn’t need more than the card could provide, but it was a bit surprising just how big web pages can be when they’re not stripped of all their hidden code.

Overall this local stack was OK. I didn’t take the chance of deploying it for real work, and instead pivoted to refining a number of workflows to a detailed specification to try to minimise the early-halting pattern and other weaknesses of the small model.

What Next

Claude is better at the hard thinking, but the reason for a local stack is that the work is private and ideally shouldn’t be sent away. Unfortunately I don’t have £100k burning a hole in my pocket to be able to have both just yet. So for the most important task Claude’s reasoning ability is still far preferable. At least I get an excuse to keep fiddling.

The next idea is a gateway that removes the identifying detail before anything leaves the machine, so Claude reasons over an anonymised version and never sees whom or what it concerns. I already have an early measure of what that masking costs: about 2 points on the small case and about 19 on the large one, because Claude loses the outside knowledge it would otherwise bring to bear on the names. Even paying that, it beat everything local. That will be the next article: how the gateway works, what is redacted, and what the anonymising costs in answer quality.

I actually did this test before I built the local reference library. Given that index runs in system RAM, it could run alongside this stack and compensate for the lack of world knowledge. I’ll want to test that again, too.

Thanks for reading!