# Private LLM Deployment Source: https://ordergroup.co/ai-software-development/private-llm/ Last updated: 2026-10-22 > On-premise LLM deployment for data that can't go to a public AI API: open model choice, vLLM serving, data protection and tested output. Free consultation. # On-premise LLM deployment for sensitive data We deploy open language models on infrastructure you choose, so texts with personal or confidential data never reach a public AI API. For PSONI, a Polish NGO, we host Gemma 4 behind Generator ETR, an app that turns official letters into easy-to-read text. The same setup can run on servers you own. [Book a free consultation](https://ordergroup.co/contact-us/) Tell us what the model should read. A few details tell us which engineers to bring to the first call. - ![AASA24](https://ordergroup.co/media/images/aasa24-mono.max-208x64.format-webp.webp) - [![AP-Tech](https://ordergroup.co/media/images/ap-tech-mono_ihuJP5T_6b2rSSb.max-208x64.format-webp.webp)](https://ordergroup.co/case-studies/monster-airport-lighting-system/) - ![Avafin](https://ordergroup.co/media/images/avafin-mono.max-208x64.format-webp.webp) - [![BizBot](https://ordergroup.co/media/images/bizbot-mono_StdJIUN.max-208x64.format-webp.webp)](https://ordergroup.co/case-studies/bizbot-fintech-ux-design/) - [![Enerace](https://ordergroup.co/media/images/enerace-mono_FKuME1X_Fp3WF4B.max-208x64.format-webp.webp)](https://ordergroup.co/case-studies/enerace-energy-trading-software/) - ![Filmgrail](https://ordergroup.co/media/images/filmgrail-mono_gPdfbHO.max-208x64.format-webp.webp) - [![Kyoto Group](https://ordergroup.co/media/images/kyoto-group-mono_hlqECHG.max-208x64.format-webp.webp)](https://ordergroup.co/case-studies/kyoto-digital-twin-software/) ## On-premise LLM deployment for data that can't go to a public AI API An on-premise LLM deployment runs an open language model on your own servers, so prompts and documents never go to an AI provider's API. A private LLM deployment is the wider term: the servers can also be rented, or sit with a cloud provider you choose. For the definition, see [self-hosted LLM](https://ordergroup.co/glossary/self-hosted-llm/) in our glossary. This page is for CTOs, heads of IT and product owners whose texts contain personal data or business secrets, in public bodies and NGOs, financial companies and energy companies. For PSONI, the Polish Association for Persons with Intellectual Disability, we build Generator ETR on a private LLM. It is a web and mobile app that turns difficult text, such as an official letter, into [easy-to-read (ETR)](https://ordergroup.co/glossary/easy-to-read/) language, in a two-year programme that started in 2026. The texts people paste in can be personal, so a public AI API was ruled out. We host an open model ourselves on rented GPU servers and serve it with vLLM, an open-source server for running language models, so no text goes to an AI provider's API. ## Private LLM or public AI API: what changes How a public AI API and a private LLM differ on data, model version, cost, quality and upkeep. | Aspect | Public AI API | Private LLM | |---|---|---| | Where the data goes | Every prompt leaves your infrastructure for the provider's servers | Texts stay on servers that you or your hosting provider control | | Model version | The provider retires old versions on its own schedule | The version you tested stays for as long as you need it | | How you pay | Per token | For GPU time, also when the model is idle | | Quality | The largest API models are stronger at general tasks | Works best on a narrow task with its own evaluation | | Who keeps it running | The provider runs the model | You or your hosting provider run the GPU servers; the software around the model is ordinary code your team can maintain | Choose the model by testing it, then serve it with vLLM ## How we run a private LLM deployment We test open models on your real task and pick one by the results. The model is served with vLLM, and its connection settings change without redeploying the app. Read more - [Self-hosted LLM](https://ordergroup.co/glossary/self-hosted-llm/) ![How we run a private LLM deployment](https://ordergroup.co/media/images/photo-1555949963-aa.2e16d0ba.fill-2000x1200.format-webp.webp) A private LLM deployment starts with the choice of an open model, and we make it by testing. For PSONI we ran PLLuM 12B, a Polish-language model, on the same text simplification task as Gemma 4, and went with Gemma 4 in its 26B mixture-of-experts version (a design that uses only part of the model's parameters for each request). The connection settings are not hard-coded. An administrator changes the address, key, model name, temperature and max_tokens in the Django Admin panel, without restarting the backend containers. So the model or the GPU server can be replaced without deploying the application again. The model sits inside a standard application your team can maintain: Django and Celery in the backend, React and React Native in the apps. Our work is choosing an open model, writing prompts, evaluating the output and serving the model. We judge whether your task needs fine-tuning in a [discovery workshop](https://ordergroup.co/services/consulting-discovery/product-workshops/). Rented GPUs, your cloud or your own servers ## Where the model runs You choose the location: rented GPU servers, a cloud provider in your region or your own servers. Generator ETR runs on fixed GPU instances rented on Vast.ai. ![Where the model runs](https://ordergroup.co/media/images/photo-1544197150-b9.2e16d0ba.fill-2000x1200.format-webp.webp) ### Where a private LLM can run Where a private LLM can run, and which option Generator ETR uses today. | Option | Rented GPU servers | Cloud provider you choose | Your own servers (on-premise) | |---|---|---|---| | Where the model runs | On GPU instances rented from a provider | At a provider in the region your rules require | On hardware you control, provided it has suitable GPUs | | Status | In use for Generator ETR, on fixed GPU instances rented on Vast.ai | Possible with the same setup of open model, vLLM and application | Possible with the same setup, as an option | The model can run on rented GPUs, at a cloud provider in the region you need, or on your own servers with suitable GPUs. You choose the location. Moving Generator ETR to its target infrastructure depends on PSONI's decision about where the servers will be, so it stays on the rented instances until then. The choice changes the bill, because you pay for GPU servers. The cost depends on the size of the model and on how many requests it has to handle at once. We size both in discovery against the traffic you expect, before you commit to a build. Hosting the model yourself is one control among several ## Data protection around the model We anonymise personal data in texts with Microsoft Presidio and encrypt what is stored. Stored texts are deleted under GDPR retention rules. Read more - [Data anonymization](https://ordergroup.co/glossary/data-anonymization/) ![Data protection around the model](https://ordergroup.co/media/images/photo-1584433144859.2e16d0ba.fill-2000x1200.format-webp.webp) Hosting a model yourself keeps texts away from a public AI provider. It does not make a system GDPR-compliant on its own, because what you store, for how long and who has access matter as well, and we go through those questions in discovery. In Generator ETR we anonymise personal data in texts with Microsoft Presidio and we are still developing it. We are also analysing whether to train HerBERT, a Polish language model, to anonymise Polish texts. Stored texts are encrypted and deleted under GDPR retention rules. The public API of Generator ETR is under construction, and it is built around the same protection of stored texts. It will show a user's text history with a retention of 30 days. Following PSONI's decision of 1 October 2026 it will have four endpoints: login (token), generating an ETR text, the user's list of texts and a preview of a past text. The endpoints are designed so that they can be switched off later, and access will go through an API key generated in the administration panel. The app counts a user's requests and shows the counter in the interface. We agree the request limits (throttling) with the client before final acceptance. ## Proving the output is good enough A private model is only useful if you can show that its output is good enough, so we run an [LLM evaluation](https://ordergroup.co/glossary/llm-evaluation/) on every iteration. A script scores each simplified text with the Gunning Fog readability metric and writes a report over a set of test samples. A reviewer trained in the ETR standard compares the output with the source text. The results go into the next iteration of the prompt and the model, and the fifth iteration is in progress. The collection of texts behind the test samples is checked by editors in a separate online service. Guardrails get tested too. Our QA tries to pull the model away from its task, for example by asking for an apple pie recipe instead of a simplified letter, and repeats these jailbreak tests with every new iteration. For legal and medical texts the app shows a notice: the simplified version is not a legal document, and a lawyer or doctor can explain the details. We are also testing a classifier for sensitive content, such as hate speech and self-harm. Texts that can't go to a public AI API? Tell us the task and we will say whether a private model fits. [Book a free consultation](https://ordergroup.co/contact-us/) ## Regulated industries and the EU AI Act The EU AI Act sorts AI systems by risk, and where a model is hosted does not change that. Regulation (EU) 2024/1689 puts systems that assess the creditworthiness of natural persons in the high-risk category, except those used to detect financial fraud (Annex III, point 5(b)). The obligations for high-risk systems are due to apply from 2 December 2027, after the Digital Omnibus package, the EU's change to the AI Act timetable (Council of the EU, 7 May 2026). Our glossary explains the [EU AI Act](https://ordergroup.co/glossary/eu-ai-act/) in more detail. In [fintech](https://ordergroup.co/fintech-software-development/) and [energy](https://ordergroup.co/energy-hub/) we build software for regulated companies, and our private LLM work is Generator ETR for PSONI. ## How a project starts and what you own Every project starts with a free consultation or a discovery workshop. We look at your texts, the rules you work under and the task, and agree how the output will be tested. Discovery covers what the model reads, where it may run and who has access to the stored texts, the same questions described above. You leave with a scope and a way to test the output. Copyright to the code passes to you with each payment, within the fields of exploitation set in the contract. We reply to a message from the contact form within 48 hours on working days. One team from model choice to the apps ## Why build a private LLM with us The same team chooses and serves the model, evaluates its output and builds the web and mobile apps around it. PSONI owns the code of Generator ETR. Read more - [AI Software Development](https://ordergroup.co/ai-software-development/) ![Why build a private LLM with us](https://ordergroup.co/media/images/photo-1694903110330.2e16d0ba.fill-2000x1200.format-webp.webp) Software projects **160+** delivered since 2013, across all industries Team **40+** full-time equivalents on our team App users **40,000+** people using apps we built, across all our projects Do you have questions? ## On-premise LLM deployment: frequently asked questions ![Łukasz Gajownik](https://ordergroup.co/media/images/T02DHCC1Z-UT420UBA7-d74022d27c3c-512.format-webp.webp) Łukasz Gajownik Head of AI & Frontend [Book a free consultation](https://ordergroup.co/contact-us/) ### What is an on-premise LLM deployment? It is an open language model running on your own GPU servers, so texts never go to an AI provider's API. When the servers are rented or sit with a cloud provider you choose, we call it a private LLM deployment. We build the application around either. Our glossary explains how a [self-hosted LLM](https://ordergroup.co/glossary/self-hosted-llm/) differs from an API model. ### Can you deploy an LLM on our own servers? Yes, as an option, provided the servers have suitable GPUs. The setup is the one we use for PSONI: an open model served with vLLM and an application around it. You choose the location, whether that is your hardware or a cloud provider in the region your rules require. ### Which open model should we use? The one that wins a test on your real task. For PSONI we compared PLLuM 12B, a Polish-language model, with Gemma 4 on one simplification task and chose Gemma 4. We repeat that kind of comparison on your own texts before committing to a model, so the choice rests on results. ### What does it cost to run a private LLM? You pay for GPU time, including the hours when the model is idle, and the bill depends on the size of the model and the number of simultaneous requests. We size the model and the servers against the traffic you expect during discovery, before you commit to a build. ### Is a private LLM GDPR-compliant by default? No. Hosting a model yourself keeps texts away from a public AI provider, but GDPR also covers what you store, how long you keep it and who has access. We go through retention, encryption, access and anonymisation in discovery, and we do not promise compliance in advance. ### How do you check the output of a private model? A script scores the output with the Gunning Fog readability metric on a set of samples, a trained reviewer compares it with the source text, and QA runs jailbreak tests that try to pull the model off its task. The results feed the next iteration. ### Does the EU AI Act apply to a private LLM? The AI Act classifies a system and its use by risk, so a private LLM can fall under it. Credit scoring of individuals is one high-risk use. Our glossary entry on the [EU AI Act](https://ordergroup.co/glossary/eu-ai-act/) has the details, and this answer is not legal advice. ### Do you fine-tune models? Yes, if your task needs it. We start by choosing and configuring an open model, writing prompts and evaluating the output, and we assess the need for fine-tuning in a free discovery workshop before the build starts. ![Developer with headphones coding at desktop monitor in an open office](https://ordergroup.co/media/images/Image_1_3kejLC8_Y69T2Sz.format-webp.width-2880.webp) ## Open models on infrastructure you choose. Your message goes to ![Filip Szamborski](https://ordergroup.co/media/images/Filip_dTWHyBo.2e16d0ba.fill-432x432.format-webp.webp) Filip Szamborski Co-founder & CEO [](https://www.linkedin.com/in/filip-szamborski/) SO FAR WE HAVE WORKED WITH BRANDS LIKE: - [![RAW Cyber](https://ordergroup.co/media/images/raw-cyber-mono_M2DWvZf_pHEQtjP.max-208x64.format-webp.webp)](https://ordergroup.co/case-studies/raw-cyber-custom-android-os/) - [![Skyfri](https://ordergroup.co/media/images/skyfri-mono_KGicBgp.max-208x64.format-webp.webp)](https://ordergroup.co/case-studies/skyfri-solar-asset-management-platform/) - ![Spacestor](https://ordergroup.co/media/images/spacestor-mono_woFaKhf.max-208x64.format-webp.webp) - [![Zeronest](https://ordergroup.co/media/images/zeronest-mono_BqSPQJQ_2nGxsSc.max-208x64.format-webp.webp)](https://ordergroup.co/case-studies/zeronest-solar-panel-monitoring-app/) - ![AASA24](https://ordergroup.co/media/images/aasa24-mono.max-208x64.format-webp.webp) Thank you for your message. We read every message ourselves and get back to you within 48 hrs on business days. Something urgent? [hello@ordergroup.co](mailto:hello@ordergroup.co)