Ask most teams why they gave up on hosting a model themselves and the answer is rarely that the model was not good enough. It is that the memory ran out. While a model writes, it has to hold a key-value cache for every request in video memory, and the classic engines reserve that space in one piece, sized to the longest output a request might ever produce. Whatever a short request does not use simply sits there empty, and fragmentation does the rest. The open-source project vLLM, which came out of a university research group in 2023, took a different line: it moved the paging scheme that operating systems use for ordinary memory onto the graphics card. The technique, PagedAttention, has since become one of the most cited ideas in model serving, the project has passed 91,000 stars, and the current stable release appeared in September 2026.

The mechanism is easy to follow. The cache is no longer reserved per request but cut into blocks of a fixed size, and a block table records where each request's blocks sit, so space is handed out as it is used. Fragmentation nearly disappears; by the project's own figures, memory utilisation rises from the thirty or forty percent of the old approach to above ninety, and the same card can serve two to four times as many requests at once. Continuous batching sits on top of that: a request no longer waits for a batch to fill, joining whenever it arrives and leaving the moment it is finished, which widens the difference further when long and short jobs share the machine. In several independent measurements the aggregate throughput on one card reached two to five times the comparison, and concurrency that once needed several cards now fits on one.

For engineering teams the other decisive choice is the interface. A server that speaks the request format of the large hosted providers is built in, so code already written in that style switches over to a self-hosted model by changing one address - not a line of business logic moves. The rest of what production needs mostly comes in the box. Tensor and pipeline parallelism spread a large model across several cards; one service can carry several LoRA adapters and switch between them per request, so a single deployment covers several narrow domains; automatic prefix caching lets requests that share a system prompt reuse blocks already computed, which shortens the wait before the first token in conversational applications; quantisation covers AWQ, GPTQ and FP8; and the figures are published in a format that existing monitoring already understands.

Its place needs stating just as plainly. vLLM is a serving framework for server-grade cards with no graphical interface, and its home ground is turning open models in the 7B to 70B range into an online service that holds under load. Lightweight chat on a personal computer is not what it is for; the desktop tools suit that better. The licence is permissive, so commercial use and derivative work raise no obstacle, and the large infrastructure companies commonly run it as the engine underneath. If the friendlier tools lowered the bar to getting a model to run, vLLM answers the other question - whether it holds. Once self-hosting moves from experiment to production, this layer is hard to avoid.

Two kinds of people end up here: the platform engineer who has been handed a model and a bill for cards, and the team that has to keep the whole thing inside its own walls. Neither of them wants a chat window. They want a service that does not fall over, and they are willing to read a configuration file to get one.