A ledger of AI tools and building sites for overseas traffic.

Home / AI tools / vLLM, llama.cpp, Ollama: Where Each Inference Stack Fits

vLLM, llama.cpp, Ollama: Where Each Inference Stack Fits

Throughput, ease of use and quantization support: you can't have all three. Pick by use case, not by benchmark scores.

Aug 21, 2026 3 min read naket.top Updated 2026-08-27
vLLM, llama.cpp, Ollama: Where Each Inference Stack Fits

These three frameworks are often compared side by side on "which one is fast", but their target scenarios are fundamentally different.

One table says it all

vLLMllama.cppOllama
GoalHigh-throughput servingMaximum control on a single machineWorks out of the box
ConcurrencyStrong (PagedAttention)Weak (manual slots)Weak
QuantizationLimited support (AWQ/GPTQ)Most complete (all GGUF levels)GGUF
HardwareMainly NVIDIACPU/GPU bothBroad
Getting startedMediumMediumMinimal

When to use which

  • Serving many people / many requests → vLLM. Its throughput advantage only shows up under concurrency; with a single request it has no edge at all.
  • Squeezing maximum performance out of one machine, controlling every parameter → llama.cpp. GGUF quantization levels are fine-grained, and CPU/GPU hybrid inference is flexible.
  • Just wanting to try a model quickly → Ollama. Its value is "you can chat in five minutes", not "you can go to production".

A common misjudgment

Run a single request on Ollama to measure speed, then use that number to estimate vLLM's concurrency capability. The two numbers aren't comparable: Ollama is single-stream, vLLM's strength is batching.

The right way to compare: fix the concurrency and context length, then measure total tokens per second. A single-stream speed test only answers "is it fast when I'm the only user".

The trade-off between quantization and precision

  • GGUF Q4 is nearly imperceptible on most chat tasks; coding tasks lose a little
  • AWQ/GPTQ 4bit works well with vLLM, but few models are available
  • When you need strictly reproducible results, don't use quantization

My actual setup

For single-machine experiments I use llama.cpp (controllable, full quantization coverage); when I need to serve, I go with vLLM. Ollama is only for "I want to try a new model" — delete it once I'm done.

The criterion for choosing a tool isn't "the strongest", it's "matches how you use it".

A metric people overlook

Everyone compares tokens/s when choosing, but the real experience is determined by time to first token (TTFT). With long context, prefill is slow and TTFT can climb from 0.3s to over 5s — users will think it's "stuck".

Measure them separately: TTFT on a short prompt, TTFT on a long prompt, and steady-state generation rate. Look at all three numbers together.

Differences in deployment form

Single processLong-running serviceContainer
Startup costLoad the model each timeOnceOnce + image layer
Memory usageReleased when doneResidentResident
Suited forExperimentsLong-term useSwitching between environments

On a machine with 24G of VRAM, keeping a 32B Q4 resident fills the card completely — forget running anything else. Experimental usage is better served by starting on demand.

In one sentence

First decide "who's using it, how many at once, and how long the context needs to be", then pick the framework. Do it the other way around, picking by benchmark scores, and you'll finish installing only to find you don't need it.

One more thing: turn those three numbers (short-prompt TTFT, long-prompt TTFT, generation rate) into a fixed script and run the same one whenever you switch frameworks — only then is the comparison meaningful.

inferencevLLMllama.cppOllama

Further reading

vLLM documentation ↗ Ollama documentation ↗