Deploying LLMs without sending data to a public API
The most common objection we hear is not about AI quality — it is “our data cannot leave the building.” That constraint is legitimate, and it is also solvable. Here is how private deployment actually works.
01 / the problem
Why public AI APIs are a genuine blocker
For many UK organisations, sending records to a third-party AI endpoint is not a preference issue — it is a compliance position. UK GDPR obligations, client confidentiality commitments, DPIAs and third-party risk assessments all ask the same question: where does the data go, and who can touch it?
“The vendor says they do not train on it” answers a different question. Data still leaves your boundary, still sits under another legal jurisdiction, and still depends on a supplier’s retention and access policies you cannot audit. For regulated or simply careful organisations, that is enough to stop a project.
02 / what private means
What “private deployment” actually means
Open-weight models — families like Llama, Mistral and Qwen — can run entirely inside infrastructure you control. The model weights live in your Azure tenant or on your own hardware; prompts and responses never transit a public API.
“Private” is a spectrum rather than a switch: tenant-isolated managed endpoints keep data inside your cloud boundary, while fully on-premise inference keeps it inside your building. The right point on that spectrum is decided by your sensitivity classification and procurement constraints — not by a vendor’s default.
03 / the stack
The layers a private deployment needs
- The model itself — sized to the workload. Many operational tasks are well served by smaller open-weight models that run on modest hardware.
- Retrieval grounding (RAG) — the model answers from your documents through governed retrieval pipelines, not from whatever it memorised in training.
- Permission-aware access — retrieval respects the same access controls as your source systems; the model cannot read what the user cannot.
- Tooling interfaces — MCP servers expose the actions agents may take, with scoped credentials per agent rather than shared service accounts.
- Logging and evaluation — every prompt, retrieval and action recorded, so behaviour can be audited and quality measured over time.
04 / trade-offs
The honest trade-offs
Private deployment is not free of cost. You own the infrastructure — compute, patching, model updates — and frontier hosted models still lead on the hardest reasoning tasks. For many workloads that gap is irrelevant; for some it is not, and a credible assessment says which side your use case sits on.
The pattern that works in practice is layered: private models handle sensitive operational work inside the boundary, while non-sensitive tasks may still use hosted APIs where contracts permit. Our localised LLM deployment service covers the assessment, provisioning and grounding work, and integration layers connect the model to the systems it needs to act on.
Data-sensitive? That is the design constraint, not a blocker
An architecture review assesses your data sensitivity, workloads and infrastructure — and shows what a private deployment would actually involve for your estate.
Request an architecture review