The Privacy Gateway
The Problem
Local AI - V1 ended on a poor score: Claude 81 versus my local stack at 55. Not as terrible as one might expect, but not good enough for big jobs.
The core goal remains: the most powerful stack I can manage to ensure maximum privacy. The core issue is that every prompt carries its data with it. Send Claude a question about your research and you’ve sent the names, addresses and connections inside that question too.
Since the whole point of my local setup is that those stay absolutely private, I was in effect paying 26 points of reasoning quality to keep it so. But I’d rather not continue to trust their retention policies after the unilateral change that came with fable.
Unfortunately I don’t have access to the hardware to have that choice. So we muddle on.
The Idea
Anyone who’s handled sensitive documents has seen the black redaction boxes. Connecting information rarely needs real names. “Person one directs org two, and org two shares an address with org three - what should I check next?” reasons the same whoever person one turns out to be.
The gateway applies that marker automatically. Every prompt bound for the cloud passes through a checkpoint that finds identifying details and swaps them for placeholders - [PERSON_1], [ORG_2] - before anything is sent. The map between placeholders and real names stays on my machine. When the reply comes back the placeholders are swapped back, and I read the answer with the names in place. Claude answered the question without ever holding them.
If the detector is down, or unsure, nothing is sent.
The Build
The gateway runs on a Raspberry Pi on my network, purely because there are some automations on there that make use of Claude which I also want privatised without having to turn on the big PC. Finding names in text is relatively light work, and the triple-GPU workstation would be overkill. However there’s nothing to stop it being accelerated by running locally on the workstation too, space permitting.
The service is FastAPI, the detection is Microsoft’s Presidio with spaCy, plus custom recognisers for the UK-specific formats, and the cloud call goes out through claude code. The named-entity recogniser reads for people, organisations and places by context, the way you’d read “Miroslav Havel” as a person without having heard of him. The pattern rules pick up the identifiers, which are formats rather than judgements: emails, phone numbers, card and account numbers, National Insurance numbers, postcodes, sort codes. For example, 01234 567 890 is probably a phone number. AB 123456 C is an NI number. And so on.
The system uses two lists on top of the detector: one provided by me, and one it generates itself. Mine holds the artifacts given in the spec - such as the subject of a piece of research. The generated one grows as the search progresses: each new name the system encounters is assigned a placeholder and it’s added to the list.
I can also ask the gateway to show me what the cloud would receive, without sending it. The cloud call itself is stripped. No tools, no browsing, no prior context. Question in, answer out.
Testing
The leak test seeds documents with twelve types of identifying data and looked for any that got through. All 12 were successfully redacted. If anything the pattern rules over-match. This means long runs of digits got wiped even when they were harmless. However I’d rather it did this than the opposite, so it’s acceptable.
I also measured the impact on reasoning ability over redacted vs un-redacted data. Claude scored 91.6 unredacted, and 81 redacted (as mentioned in Local AI - V1). So redaction costs about 10 points, and it costs more the bigger the case.
The reason for this is that masking names strips Claude of its world knowledge. Show it a named company and it brings everything it knows about that company; show it [ORG_2] and it can only use what the documents say. The bigger the case, the more that missing knowledge would have contributed.
Findings
Even paying the tax, gateway-Claude beat the best local model by 12 points - and that local model ties up all three cards overnight to do what Claude can do in a couple of minutes. I’m not paid by the hour so I’ll take it.
No detector catches everything. While this is better than using an LLM, an unusual name in an odd sentence will still slip through eventually. Hence the lists and the preview.
The placeholders have to persist. [PERSON_1] must mean the same person in every prompt across a job, and not be over-written, or the reasoning falls apart halfway through. That needed a persistent map per job rather than a fresh one per request/call. This just means a file kept in the project folder rather than generated at the reasoning step in the wider workflow.
What Next
The V1 stack can browse and gather with real names, since nothing it touches leaves the network, and pass up blanked fragments when something needs the better reasoner.
Local AI - V2 is in progress. I’m pruning and quantising models to see what I can really fit on here. But I know it’ll have this gateway and the reference library from Giving a local AI a reference library underneath. It increasingly seems like when it comes to local, small-scale AI, the worker really can blame their tools.
Thanks for reading!