A ledger of AI tools and building sites for overseas traffic.

Home / AI tools / Running LLMs on a 4090: The Real Limits of 24 GB of VRAM

Running LLMs on a 4090: The Real Limits of 24 GB of VRAM

Whether a 70B fits, how far to quantize, how much concurrency breaks it — judged by the VRAM budget, not the parameter count.

Aug 10, 2026 4 min read naket.top Updated 2026-08-16
Running LLMs on a 4090: The Real Limits of 24 GB of VRAM

24 GB of VRAM sounds like a lot, but it first gets split three ways: weights, KV cache, and framework overhead. It's not that a 70B model can't run on a 4090 — it's that it can't run long contexts.

Start with the weights

QuantizationBits per parameter70B weights32B weights
FP1616~140GB~64GB
Q88~70GB~32GB
Q44~35GB~16GB
Q33~26GB~12GB

With 24 GB on a single card, only Q3 at 70B can barely "fit part of it in"; in practice you have to load layers into system RAM. Conclusion: on a single 4090, 32B at Q4 is the comfortable sweet spot.

The KV cache is the hidden cost

The longer the context, the bigger the KV cache. Rough estimate: KV overhead per token ≈ layers × hidden dimension × 2 × bytes.

Practical rules of thumb (Q4, 32B):

ContextExtra VRAM
4K~1G
16K~4G
32K~8G

So "32B Q4 + 32K context" is very tight on 24 GB; in practice 16K is the stable point.

Concurrency doesn't scale linearly

llama.cpp is serial by default for a single request; for concurrency you need --parallel, but each slot needs its own KV cache. 4 concurrent requests at 16K means 16 GB of cache — instant blowup.

When to change your approach

  • Need 70B or larger: rent a GPU or go through an API, don't force it
  • Need to serve multiple concurrent users: vLLM plus more VRAM
  • Just experimenting on one machine: 4090 + 32B Q4 is enough

What it actually feels like

With 32B Q4 on a 4090, short-context generation runs about 30–40 token/s, drops to around 20 at 16K context, and the first prefill on long context is fairly slow (a few seconds to a dozen or so).

That speed is plenty for "my own use" and not enough for "serving several people." Figure out which one you're doing first.

A practical selection table

Use caseModel sizeQuantizationContext
Everyday Q&A, rewriting7–14BQ5/Q48K
Code completion, explanation14–32BQ416K
Long document summarization14BQ432K (via external retrieval)
Multi-turn agent tasks32BQ416K

The key judgement: context eats more VRAM than parameter count does. With the same 24 GB, 32B Q4 with only 4K context runs smoothly; give it 32K and it starts swapping.

An overlooked cost

Loading a model the first time means reading tens of GB of weight files. On a spinning disk, a cold start takes one or two minutes; on NVMe, a dozen seconds or so. If your usage pattern is "unload it when you're done," disk speed matters more than the GPU.

One lesson

Get the whole pipeline working with a 7B first (prompts, retrieval, output format), then swap in a bigger model. Do it the other way around and when something breaks you can't tell whether the model is inadequate or the pipeline is broken.

One more observation: the bar for a local model being "usable" isn't whether it can produce a correct answer — it's whether it can produce correctly formatted answers ten times in a row. When stability falls short, wiring it into automation costs far more than calling an API.

And one more: estimate whether VRAM is sufficient with "model size × 1.2 + context usage." Leave 20% headroom, or a single concurrent request will OOM — that's where the most common local-inference failure, "it works until you put any load on it," comes from.

local deployment4090quantizationinference

Further reading

llama.cpp docs ↗ vLLM docs ↗