Self-Hosting an LLM: What It Costs, What It Takes — and When It Pays Off

“We run our own LLM” sounds like independence. Technically, getting started is easy today: a machine with enough graphics memory, a tool like Ollama, thirty minutes. The hard question comes after. Anyone self-hosting an LLM buys more than compute — they buy an operating responsibility. This article does the math on what that costs — and shows why for many companies the math is not the deciding question.

Contents

• What self-hosting an LLM really means
• Hardware: how much GPU a model needs
• European alternatives to buying hardware
• Tools: Ollama, Open WebUI and vLLM
• The math: self-hosted vs. API
• When the math is not what decides
• Not all or nothing: separate the processes
• Conclusion: the data question first, then the math
• Frequently asked questions
• Sources

What self-hosting an LLM really means

Self-hosted AI means the model runs on hardware the company controls. No request leaves its own infrastructure. That is one half of the story.

The other half is operations. Self-hosting a model means taking on permanent responsibility — for availability, updates, access rights, logging and security. Starting a model takes minutes. Keeping it in production is an ongoing job.

Three things get underestimated regularly. First, utilization: a GPU costs the same whether it computes or waits. Second, pace: open models ship weekly, and every update needs testing. Third, the interface — an API endpoint alone is not a working tool for employees.

Where a company stands on AI today can be mapped in a few minutes with the free Self-Check.

Hardware: how much GPU a model needs

The decisive figure is graphics memory, or VRAM. A reliable rule of thumb: parameters times bits per weight, divided by eight, gives the memory requirement in gigabytes. A model with 8 billion parameters at 4-bit quantization therefore needs roughly 4 GB — plus headroom for context and cache.

In practice, four tiers emerge:

• 8–12 GB VRAM: models with 7–8 billion parameters, quantized. Enough for text tasks, summaries and simple classification.
• 24 GB VRAM: models up to around 30 billion parameters, quantized. Noticeably better quality and more context.
• 48 GB VRAM: 70-billion-parameter models when quantized, or several smaller models in parallel.
• 80–96 GB VRAM: production use with large models and multiple concurrent users.

Quantization is the biggest lever here. It reduces the precision of model weights and with it the memory footprint — at a moderate cost in quality. A 70B model that occupies two specialist GPUs at full precision runs on one when quantized.

The difference between “it runs” and “it holds up” matters. A model on a laptop answers one person. Fifty concurrent requests are a different problem. That takes more than VRAM — it takes throughput, which is the real discipline of LLM inference.

The question is never whether a model runs. The question is whether the operation holds up.

European alternatives to buying hardware

Companies that do not want to buy hardware can rent it. European providers make that possible without routing through US data centers — a practical building block for digital sovereignty.

Hetzner offers the GEX131, a dedicated GPU server with 96 GB of graphics memory, from €889 per month, with no setup fee and in ISO 27001-certified data centers in Germany and Finland (Source: Hetzner, 2025). OVHcloud provides GPU instances by the hour; an L40S with 48 GB runs at around €1.40 per hour (Source: OVHcloud Public Cloud GPU, 2026) — roughly €1,000 per month if run continuously, but cheap for time-boxed workloads.

The three realistic tiers:

Scenario	Hardware / cloud	Setup effort	Monthly cost	Suitable for
Small	Existing workstation, 12–24 GB VRAM, Ollama	½–1 day	Power and maintenance, no license cost	Tests, prototypes, single departments
Medium	Dedicated GPU server, 96 GB VRAM (e.g. Hetzner GEX131)	2–5 days	from €889	Steady internal load, sensitive data
Large	Multiple GPU servers or cloud GPUs with redundancy	Several weeks	from around €2,000 plus staffing	High load, failover, SLA

The largest cost item is missing from that table: internal staff time. Setup, hardening, monitoring and updates consume capacity — permanently, not once.

Tools: Ollama, Open WebUI and vLLM

Three tools cover almost every case. They solve different problems and complement each other.

Tool	Job	Strength	Limit
Ollama	Load and run models locally	Fastest entry point, very simple to use	Not built for throughput across many users
Open WebUI	Chat interface with user management	Makes the model usable for employees	Does not replace governance
vLLM	High-performance inference server	High throughput under concurrent load	More setup and operating effort

The typical path: Ollama for proof of concept, Open WebUI as the working surface, vLLM once several teams access it at the same time. All three expose an OpenAI-compatible interface. Swapping the server therefore does not force a rewrite of applications — and that is exactly what reduces dependency.

Which models are worth considering in the first place is covered in the open source LLM overview.

The math: self-hosted vs. API

Now the numbers. A GPU server costs a fixed amount per month regardless of usage. An API costs per token — little at low volume, a lot at high volume. Somewhere the two lines cross.

The calculation with documented prices: a dedicated GPU server costs €889 per month (Source: Hetzner, 2025). A frontier API such as Claude Sonnet costs $3 per million input and $15 per million output tokens at list price (Source: Anthropic, 2026) — around $6 per million tokens blended at a three-to-one ratio. A hosted open-model API sits far below that: Qwen3 235B costs $0.20 input and $0.60 output per million tokens at Together AI (Source: Together AI, 2026), roughly $0.30 blended.

A note on how long such calculations hold: until 31 August 2026, Sonnet ran on an introductory price of $2 and $10, which put break-even at around 222 million tokens. The move to list price cut that by a third. Anyone building this kind of calculation is better off recording the date than the number.

For presentations:SVGPNG

The result is uncomfortable. Break-even against the frontier API sits at around 148 million tokens per month — roughly 4.9 million tokens per day, or about 2,500 substantial requests daily. Few mid-market companies reach that. Against a hosted open-model API, an in-house server practically never pays off, because providers spread GPU utilization across many customers.

Not included: operating effort, on-call coverage and failover. Price those honestly and break-even moves further right. How to reduce token cost in the first place is covered in the article on token-smart architecture.

The conclusion: self-hosting is rarely a cost decision. It is a control decision.

When the math is not what decides

So far this reads like an argument against the in-house server. For a substantial share of companies, though, the math is not the deciding question at all.

The German Federal Statistical Office asked companies that use no AI why not. 58% cite unclear legal consequences, 53% cite data privacy concerns (Source: Destatis, 2024). High cost trails far behind at 28%. The bottleneck is not price. The bottleneck is trust.

Three very real situations sit behind that:

• Regulation. Banks, insurers, healthcare, defence suppliers. Where a supervisor reads along, the data path is part of the approval — not a matter of convenience.
• Valuable engineering data. In mechanical engineering, the edge over competitors sits in design data, measurement series and failure histories. Handing that to an opaque cloud is a strategic decision, not an IT decision.
• Owner intent. In owner-led companies the decision is often simply that the data does not leave the building. Whether the risk is objective or perceived changes nothing about the effect.

For these companies the alternative is not “in-house LLM or API”. It is in-house LLM or no AI at all. And then a break-even at 148 million tokens is the wrong metric. The right question is what it costs to keep doing nothing.

Rule out the cloud and you are not comparing server against API. You are comparing in-house operations against standing still.

The math above keeps its value regardless. It puts a number on what control costs. That is a fair basis for a deliberate decision — and far more defensible than a gut feeling that self-hosting is “probably more expensive anyway”.

Not all or nothing: separate the processes

The second false assumption is the all-or-nothing logic. Hardly any company works exclusively with data that needs protecting.

A company that will not release design data can still run job ads, marketing copy, translations or meeting notes through an external model. Those items contain no trade secrets. Tying them to an in-house server gives away quality and money — the large models are simply better and cheaper here.

The market is moving in exactly that direction. In December 2025 Deloitte surveyed 515 leaders at companies above $500 million in revenue. The result is not either-or but a three-tier model: public cloud for training and experimentation, private infrastructure for predictable high-volume inference, edge for time-critical decisions (Source: Deloitte, 2026). There, the binary cloud-versus-on-premises question counts as obsolete.

In practice: sort the data by protection requirement, not the technology by worldview. Three classes are usually enough.

Class	Examples	Sensible route
Strictly confidential	design data, patient records, contracts, pricing	in-house model, or a controlled EU environment
Internal	meeting notes, reports, internal communication	EU-hosted model with a DPA, no training consent
Non-critical	marketing copy, translations, research, job ads	frontier API, wherever quality and price are best

This sorting is work — and it is the actual lever. It requires someone to define data classes, assign workflows and enforce the rule when someone pastes something into the wrong window. Without that layer you get exactly what bans produce: shadow AI.

This is not a one-off task but continuous operations. Anyone running several models in parallel needs a layer above them that decides which task goes to which model and makes consumption visible. That is what we build for clients as the Operations Layer — it turns the sorting above into a rule inside the system instead of a request to the workforce.

Conclusion: the data question first, then the math

The order decides. Start with the cost calculation and you almost always land on the API — and miss that for some companies the cloud is not on the table at all.

Three questions, in this order:

1. Which data must not leave the building? This comes first and is answered by data class, not across the board. It determines how much of the work is even eligible for an external API.
2. Are there people to run it? Without named ownership and a time budget, an in-house server becomes a project nobody maintains. This question has ended more initiatives than any cost calculation.
3. Is the load high and steady? Only now does the math above matter. It puts a number on what control costs — and whether the in-house server also carries commercially.

The pragmatic path starts small: sort the data into three classes, put a strong external model to work on the non-critical class right away, and test an open model on existing hardware for the protected class. After three months of measured usage, the math rests on data instead of assumptions.

That is where this question belongs — inside continuous AI Operations, not as a one-off matter of principle.

Frequently asked questions

What does it cost to run your own LLM?

Initial tests on existing hardware cost little beyond electricity and time. A dedicated GPU server with 96 GB of graphics memory starts at €889 per month (Source: Hetzner, 2025). On top of that comes internal operating effort, which is missing from almost every calculation.

What is the minimum hardware an LLM needs?

Quantized models with 7 to 8 billion parameters run on 8 to 12 GB of VRAM. Quantized 70-billion-parameter models need around 48 GB. Memory requirements follow roughly from parameters times bits per weight, divided by eight.

Is Ollama suitable for production?

For individual users and prototypes, yes. Under many concurrent requests Ollama hits its limits — that is where vLLM fits. Both expose an OpenAI-compatible interface, so switching is manageable.

Is self-hosting cheaper than an API?

In most cases, no. Break-even against a frontier API sits at around 148 million tokens per month, and against low-cost open-model APIs it usually never arrives. The reason to self-host is data control, not price.

We cannot put our data in the cloud. Is an in-house server our only option?

For the data that needs protecting, yes. For the rest, no. Hardly any company works exclusively with confidential content — marketing copy, translations or research can usually be processed externally without concern. The sensible move is to sort by data class rather than decide wholesale for one side.

Why do so many companies use no AI at all?

Not because of cost. Among companies without AI, 58% cite unclear legal consequences and 53% cite data privacy concerns — high cost trails far behind at 28% (Source: Destatis, 2024). For these companies, controlled in-house operations are often the only way to start at all.

How do you prepare the decision properly?

With measurement instead of estimation: actual token volume, data protection requirements and available operating capacity. We map that out in a free diagnosis call.

Sources

• Destatis: Reasons against using artificial intelligence technologies, 2024
• Deloitte: Enterprise AI Infrastructure Survey — A 2028 Outlook, 2026
• Hetzner: GEX131 GPU Server with NVIDIA RTX PRO 6000 Blackwell Max-Q, 2025
• Hetzner: Dedicated GPU-Line, 2026
• OVHcloud: L40S Cloud GPU Instance, 2026
• Anthropic: Claude API Pricing, 2026
• Together AI: Pricing for Open Models, 2026
• Ollama: Documentation and Model Library, 2026
• Open WebUI: Project Documentation, 2026
• vLLM: Project Documentation, 2026