On-premise LLM deployment for sensitive data

An on-premise LLM deployment runs an open language model on your own servers, so prompts and documents never go to an AI provider's API. A private LLM deployment is the wider term: the servers can also be rented, or sit with a cloud provider you choose. For the definition, see self-hosted LLM in our glossary. This page is for CTOs, heads of IT and product owners whose texts contain personal data or business secrets, in public bodies and NGOs, financial companies and energy companies.

For PSONI, the Polish Association for Persons with Intellectual Disability, we build Generator ETR on a private LLM. It is a web and mobile app that turns difficult text, such as an official letter, into easy-to-read (ETR) language, in a two-year programme that started in 2026. The texts people paste in can be personal, so a public AI API was ruled out. We host an open model ourselves on rented GPU servers and serve it with vLLM, an open-source server for running language models, so no text goes to an AI provider's API.

A private LLM deployment starts with the choice of an open model, and we make it by testing. For PSONI we ran PLLuM 12B, a Polish-language model, on the same text simplification task as Gemma 4, and went with Gemma 4 in its 26B mixture-of-experts version (a design that uses only part of the model's parameters for each request).

The connection settings are not hard-coded. An administrator changes the address, key, model name, temperature and max_tokens in the Django Admin panel, without restarting the backend containers. So the model or the GPU server can be replaced without deploying the application again. The model sits inside a standard application your team can maintain: Django and Celery in the backend, React and React Native in the apps. Our work is choosing an open model, writing prompts, evaluating the output and serving the model. We judge whether your task needs fine-tuning in a discovery workshop.

The model can run on rented GPUs, at a cloud provider in the region you need, or on your own servers with suitable GPUs. You choose the location. Moving Generator ETR to its target infrastructure depends on PSONI's decision about where the servers will be, so it stays on the rented instances until then.

The choice changes the bill, because you pay for GPU servers. The cost depends on the size of the model and on how many requests it has to handle at once. We size both in discovery against the traffic you expect, before you commit to a build.

Hosting a model yourself keeps texts away from a public AI provider. It does not make a system GDPR-compliant on its own, because what you store, for how long and who has access matter as well, and we go through those questions in discovery. In Generator ETR we anonymise personal data in texts with Microsoft Presidio and we are still developing it. We are also analysing whether to train HerBERT, a Polish language model, to anonymise Polish texts. Stored texts are encrypted and deleted under GDPR retention rules.

The public API of Generator ETR is under construction, and it is built around the same protection of stored texts. It will show a user's text history with a retention of 30 days. Following PSONI's decision of 1 October 2026 it will have four endpoints: login (token), generating an ETR text, the user's list of texts and a preview of a past text. The endpoints are designed so that they can be switched off later, and access will go through an API key generated in the administration panel. The app counts a user's requests and shows the counter in the interface. We agree the request limits (throttling) with the client before final acceptance.

A private model is only useful if you can show that its output is good enough, so we run an LLM evaluation on every iteration. A script scores each simplified text with the Gunning Fog readability metric and writes a report over a set of test samples. A reviewer trained in the ETR standard compares the output with the source text. The results go into the next iteration of the prompt and the model, and the fifth iteration is in progress. The collection of texts behind the test samples is checked by editors in a separate online service.

Guardrails get tested too. Our QA tries to pull the model away from its task, for example by asking for an apple pie recipe instead of a simplified letter, and repeats these jailbreak tests with every new iteration. For legal and medical texts the app shows a notice: the simplified version is not a legal document, and a lawyer or doctor can explain the details. We are also testing a classifier for sensitive content, such as hate speech and self-harm.

The EU AI Act sorts AI systems by risk, and where a model is hosted does not change that. Regulation (EU) 2024/1689 puts systems that assess the creditworthiness of natural persons in the high-risk category, except those used to detect financial fraud (Annex III, point 5(b)). The obligations for high-risk systems are due to apply from 2 December 2027, after the Digital Omnibus package, the EU's change to the AI Act timetable (Council of the EU, 7 May 2026). Our glossary explains the EU AI Act in more detail.

In fintech and energy we build software for regulated companies, and our private LLM work is Generator ETR for PSONI.

Every project starts with a free consultation or a discovery workshop. We look at your texts, the rules you work under and the task, and agree how the output will be tested. Discovery covers what the model reads, where it may run and who has access to the stored texts, the same questions described above. You leave with a scope and a way to test the output. Copyright to the code passes to you with each payment, within the fields of exploitation set in the contract. We reply to a message from the contact form within 48 hours on working days.