These three frameworks are often compared side by side on "which one is fast", but their target scenarios are fundamentally different.
One table says it all
| vLLM | llama.cpp | Ollama | |
|---|---|---|---|
| Goal | High-throughput serving | Maximum control on a single machine | Works out of the box |
| Concurrency | Strong (PagedAttention) | Weak (manual slots) | Weak |
| Quantization | Limited support (AWQ/GPTQ) | Most complete (all GGUF levels) | GGUF |
| Hardware | Mainly NVIDIA | CPU/GPU both | Broad |
| Getting started | Medium | Medium | Minimal |
When to use which
- Serving many people / many requests → vLLM. Its throughput advantage only shows up under concurrency; with a single request it has no edge at all.
- Squeezing maximum performance out of one machine, controlling every parameter → llama.cpp. GGUF quantization levels are fine-grained, and CPU/GPU hybrid inference is flexible.
- Just wanting to try a model quickly → Ollama. Its value is "you can chat in five minutes", not "you can go to production".
A common misjudgment
Run a single request on Ollama to measure speed, then use that number to estimate vLLM's concurrency capability. The two numbers aren't comparable: Ollama is single-stream, vLLM's strength is batching.
The right way to compare: fix the concurrency and context length, then measure total tokens per second. A single-stream speed test only answers "is it fast when I'm the only user".
The trade-off between quantization and precision
- GGUF Q4 is nearly imperceptible on most chat tasks; coding tasks lose a little
- AWQ/GPTQ 4bit works well with vLLM, but few models are available
- When you need strictly reproducible results, don't use quantization
My actual setup
For single-machine experiments I use llama.cpp (controllable, full quantization coverage); when I need to serve, I go with vLLM. Ollama is only for "I want to try a new model" — delete it once I'm done.
The criterion for choosing a tool isn't "the strongest", it's "matches how you use it".
A metric people overlook
Everyone compares tokens/s when choosing, but the real experience is determined by time to first token (TTFT). With long context, prefill is slow and TTFT can climb from 0.3s to over 5s — users will think it's "stuck".
Measure them separately: TTFT on a short prompt, TTFT on a long prompt, and steady-state generation rate. Look at all three numbers together.
Differences in deployment form
| Single process | Long-running service | Container | |
|---|---|---|---|
| Startup cost | Load the model each time | Once | Once + image layer |
| Memory usage | Released when done | Resident | Resident |
| Suited for | Experiments | Long-term use | Switching between environments |
On a machine with 24G of VRAM, keeping a 32B Q4 resident fills the card completely — forget running anything else. Experimental usage is better served by starting on demand.
In one sentence
First decide "who's using it, how many at once, and how long the context needs to be", then pick the framework. Do it the other way around, picking by benchmark scores, and you'll finish installing only to find you don't need it.
One more thing: turn those three numbers (short-prompt TTFT, long-prompt TTFT, generation rate) into a fixed script and run the same one whenever you switch frameworks — only then is the comparison meaningful.