Giving a local AI a reference library

The Problem

I like to run large language models on my own computer. They are good at a lot of things: reasoning, writing, giving advice. They are not good at precise facts, the dates and names and statistics.

Ask one for the date of a famous speech and it will offer you the right decade and the wrong year. Ask it for a well-known person’s middle name and it will often invent one. Ask it for the population of a small country and it will round to the nearest hundred million.

They are not trying to lie. They are trusting an imperfect memory formed during training, and when a precise detail was never fixed in that memory they fill the gap with something that sounds right. That is fine when it does not matter whether the answer holds up, but a lot of what I do needs accuracy and privacy.

The Idea

One thing careful people do is look a fact up when they are unsure, rather than trust memory. I can give a model the same habit: a library of reference material to check before it answers. I gave it the whole of English Wikipedia, and arranged for it to read the relevant pages before replying.

The Build

Raw English Wikipedia is about 200 gigabytes of text and markup once un-compressed, too much for any model to read in one pass. The work was turning that into something searchable in a fraction of a second.

I used the English Wikipedia snapshot from 1 July 2026. I broke every article into passages of a few hundred words, each tagged with its article and section so the context was kept. That produced more than 17 million passages.

I then gave each passage to a small dedicated model, Qwen3-Embedding-4B, whose job is to turn text into a “meaning fingerprint”: a vector of 1024 numbers arranged so that passages with similar meaning have similar vectors. This is what makes semantic search work. You pass a question through the same model to get a query vector, then compare it against all 17 million passage vectors to find the closest matches. A question about “the town where a certain author was born” finds the right passage even when it shares no words with it, because the meanings line up.

The best way to explain this is, I think, by thinking of grid coordinates on a map. They use two 7-digit strings to identify every location on a 2D map down to 1 meter of accuracy. Numbers which are numerically close together will be physically close together, and you can search without knowing exact names of places. This is the same, except there’s 1024 numbers for each passage, each a small decimal that can be positive or negative. The compressed index stores each fingerprint far more coarsely, in about 64 bytes instead of 2048. Much less precise! (SO much so, it was actually worse than not using it)

Those numbers are chosen according to a model of the English language. This is where we get the term ‘large language model’. They are approximations of natural language in the same way an architect’s miniature is a model of a building. In this case I used a model from the Qwen family, but different models emphasise different things, so may produce different ‘maps’ of meaning. Like how one architect may emphasise the materials in a miniature, and another may emphasise the interior as being more salient.

Generating all 17 million fingerprints took three days of continuous 100% load across a 4090 and two 3090s. The finished index takes a question and returns the 50 most relevant (‘closest’) passages in about a hundredth of a second. A second model, Qwen3-Reranker-4B, re-reads those 50 and picks the best eight. Those eight passages and the original question go to the main model, Qwen3.6-27B, which reads them and writes the answer. The whole thing runs on my computer, with nothing sent to the internet.

Testing

I ran SimpleQA, a public factual-accuracy test from OpenAI. It is 4,326 hard factual questions, each with a standard answer and a standard way to mark a reply correct, wrong, or not attempted. The questions are deliberately obscure, which favours looking things up over trusting memory, so it is a fair test of whether the reference library helps.

For each question I ran the main model, Qwen3.6-27B (27 billion parameters), twice: once from memory alone, and once allowed to look things up in the Wikipedia index. I then hand-checked 150 of the marked answers myself. The marking was honest, and if anything slightly generous to the memory-only answers.

Results

On the 3,449 questions that Wikipedia can answer, the difference was large.

	Correct	Wrong	Said “I don’t know”
Memory only	15%	68%	17%
With lookup	56%	3%	41%

Correct answers went from about one in seven to more than half. The bigger change is the confident wrong answers, which fell from over two-thirds to three percent. That is the figure I care about most. Looking things up did not only make the model more accurate. It made it more honest: when the answer was not in the passages it found, it said “I don’t know” instead of inventing one. A model that knows the limits of what it knows is worth more than one that always has an answer ready.

Limits

The index holds only the main text of each article. It does not include the fact boxes and tables in the sidebars, where a lot of precise dates and figures sit, so those are under-represented. Nothing stops someone extending the index to cover them, and that is what I plan to do next once I’ve emotionally recovered from 60db for four days.

The Wikipedia snapshot is fixed at 1 July 2026, so this is for settled facts, not news. And about a fifth of SimpleQA questions, roughly 880, are not covered by Wikipedia at all. The figures for the full 4,326-question set are lower than for the Wikipedia-answerable slice, though the pattern is the same.

Using it

You will need 40gb of RAM for the uncompressed index atop your other uses. The compressed one is only 1.33gb but is very inaccurate.

I have published the cleaned-up 17 million passages, all of their meaning fingerprints, and the finished search index, so anyone can reproduce this lookup ability in their own project without repeating the three-day computation. They are on Hugging Face: huggingface.co/Sherlock-Comms (three datasets: passages, embeddings, and the FAISS indices).

A language model that runs at home can be made trustworthy on facts by giving it a local reference library, and, just as usefully, it learns to admit when it does not know.