24 GB of VRAM sounds like a lot, but it first gets split three ways: weights, KV cache, and framework overhead. It's not that a 70B model can't run on a 4090 — it's that it can't run long contexts.
Start with the weights
| Quantization | Bits per parameter | 70B weights | 32B weights |
|---|---|---|---|
| FP16 | 16 | ~140GB | ~64GB |
| Q8 | 8 | ~70GB | ~32GB |
| Q4 | 4 | ~35GB | ~16GB |
| Q3 | 3 | ~26GB | ~12GB |
With 24 GB on a single card, only Q3 at 70B can barely "fit part of it in"; in practice you have to load layers into system RAM. Conclusion: on a single 4090, 32B at Q4 is the comfortable sweet spot.
The KV cache is the hidden cost
The longer the context, the bigger the KV cache. Rough estimate: KV overhead per token ≈ layers × hidden dimension × 2 × bytes.
Practical rules of thumb (Q4, 32B):
| Context | Extra VRAM |
|---|---|
| 4K | ~1G |
| 16K | ~4G |
| 32K | ~8G |
So "32B Q4 + 32K context" is very tight on 24 GB; in practice 16K is the stable point.
Concurrency doesn't scale linearly
llama.cpp is serial by default for a single request; for concurrency you need --parallel, but each slot needs its own KV cache. 4 concurrent requests at 16K means 16 GB of cache — instant blowup.
When to change your approach
- Need 70B or larger: rent a GPU or go through an API, don't force it
- Need to serve multiple concurrent users: vLLM plus more VRAM
- Just experimenting on one machine: 4090 + 32B Q4 is enough
What it actually feels like
With 32B Q4 on a 4090, short-context generation runs about 30–40 token/s, drops to around 20 at 16K context, and the first prefill on long context is fairly slow (a few seconds to a dozen or so).
That speed is plenty for "my own use" and not enough for "serving several people." Figure out which one you're doing first.
A practical selection table
| Use case | Model size | Quantization | Context |
|---|---|---|---|
| Everyday Q&A, rewriting | 7–14B | Q5/Q4 | 8K |
| Code completion, explanation | 14–32B | Q4 | 16K |
| Long document summarization | 14B | Q4 | 32K (via external retrieval) |
| Multi-turn agent tasks | 32B | Q4 | 16K |
The key judgement: context eats more VRAM than parameter count does. With the same 24 GB, 32B Q4 with only 4K context runs smoothly; give it 32K and it starts swapping.
An overlooked cost
Loading a model the first time means reading tens of GB of weight files. On a spinning disk, a cold start takes one or two minutes; on NVMe, a dozen seconds or so. If your usage pattern is "unload it when you're done," disk speed matters more than the GPU.
One lesson
Get the whole pipeline working with a 7B first (prompts, retrieval, output format), then swap in a bigger model. Do it the other way around and when something breaks you can't tell whether the model is inadequate or the pipeline is broken.
One more observation: the bar for a local model being "usable" isn't whether it can produce a correct answer — it's whether it can produce correctly formatted answers ten times in a row. When stability falls short, wiring it into automation costs far more than calling an API.
And one more: estimate whether VRAM is sufficient with "model size × 1.2 + context usage." Leave 20% headroom, or a single concurrent request will OOM — that's where the most common local-inference failure, "it works until you put any load on it," comes from.