Home / Costs & subscriptions / The Real Cost of Long Context: How to Do the Token Math
The Real Cost of Long Context: How to Do the Token Math
Context isn't free. The same conversation, placed at the start instead of the end, can cost more than twice as much.
Every service that bills by token has a multiplier that's easy to overlook: every request resends the entire context. The longer the context, the higher the per-request cost — and you pay for it over and over.
A simple model
Let C be the context length, O the output length, and N the number of requests. Total cost ≈ N × (C × input price + O × output price).
The key point is that C is not a constant: in a multi-turn conversation it keeps growing. The request for turn 10 contains everything from the previous 9 turns.
A feel for the numbers
| Scenario | Context | Cost per request (order of magnitude) |
|---|---|---|
| Short Q&A | 1K | Very low |
| Reading one long article | 30K | 30× short Q&A |
| Turn 20 of a long conversation | 60K | 60× short Q&A |
| Stuffing in an entire codebase | 200K | 200× short Q&A |
So the real spend is "context size × number of turns". Fill 200K once and then chat for 20 turns, and the bill will be frightening.
Three ways to cut cost
- Summaries instead of the original text: summarize a long document down to 2K first, then have the conversation with the summary in hand
- Retrieve on demand: don't hand over everything at once; let the model look things up as needed (this is exactly what MCP-style tools are for)
- Control the number of turns: start a new session when a task is done; don't do ten unrelated things in one very long session
The effect of caching
Most services discount requests that share a common prefix (prompt caching). That means: put the unchanging content (system prompt, rules) at the very front and the changing content after it, and you cut costs significantly.
Do it the other way around — insert new content at the front every time — and the cache is invalidated entirely, putting cost back at full price.
One habit
Before you "paste a big chunk of stuff in", ask yourself: what proportion of this content does the model actually need? Usually less than 20%.
What you save isn't just tokens — it's context left over for the parts that actually matter.
Three common misjudgments
- Dumping the whole document in: tokens spent on irrelevant paragraphs, attention diluted, answer quality down.
- Retrieving without reranking: 20 chunks recalled with uneven relevance; the model easily latches onto the wrong one.
- Chunking by character count: a flat 500 characters will cut tables and code in half, fragmenting the semantics.
A chunking strategy that works
- Split by heading level, no chunk larger than 800 characters
- Keep code blocks and tables whole, don't split them
- Prefix each chunk with "source title + section name" so the model knows the context when a chunk is retrieved
- 10–15% overlap, so information isn't broken across chunks
Cost comparison
| Approach | Input tokens for a 10K-character document | Answer quality |
|---|---|---|
| Paste the whole thing | ~15K | Medium (easy to get lost) |
| Retrieve top 5 | ~2500 | High (focused) |
| Retrieve + summarize | ~800 | Medium-high (lossy) |
In one sentence
A bigger context window doesn't mean you should stuff more into it. Being able to locate things matters more than being able to hold them.
The cost formula
Cost per request ≈ input tokens × unit price + output tokens × unit price. What's really expensive with long context is the input, and every turn resends the history all over again.
So in multi-turn scenarios the cost grows quadratically: 10 turns × 5K input per turn = 50K input tokens, not 5K. Do the total math first, then decide how big a window to open.
One more practical tip: fix the "system prompt + tool definitions" in place and put them up front for caching. This part is sent every turn, but its content doesn't change; once the cache hits, cost drops by an order of magnitude.
To close in one line: the bigger the window, the more restraint you need; the tokens you save are, essentially, the attention you save.