A ledger of AI tools and building sites for overseas traffic.

Home / Costs & subscriptions / The Real Cost of Long Context: How to Do the Token Math

The Real Cost of Long Context: How to Do the Token Math

Context isn't free. The same conversation, placed at the start instead of the end, can cost more than twice as much.

Nov 19, 2026 4 min read naket.top
The Real Cost of Long Context: How to Do the Token Math

Every service that bills by token has a multiplier that's easy to overlook: every request resends the entire context. The longer the context, the higher the per-request cost — and you pay for it over and over.

A simple model

Let C be the context length, O the output length, and N the number of requests. Total cost ≈ N × (C × input price + O × output price).

The key point is that C is not a constant: in a multi-turn conversation it keeps growing. The request for turn 10 contains everything from the previous 9 turns.

A feel for the numbers

ScenarioContextCost per request (order of magnitude)
Short Q&A1KVery low
Reading one long article30K30× short Q&A
Turn 20 of a long conversation60K60× short Q&A
Stuffing in an entire codebase200K200× short Q&A

So the real spend is "context size × number of turns". Fill 200K once and then chat for 20 turns, and the bill will be frightening.

Three ways to cut cost

  1. Summaries instead of the original text: summarize a long document down to 2K first, then have the conversation with the summary in hand
  2. Retrieve on demand: don't hand over everything at once; let the model look things up as needed (this is exactly what MCP-style tools are for)
  3. Control the number of turns: start a new session when a task is done; don't do ten unrelated things in one very long session

The effect of caching

Most services discount requests that share a common prefix (prompt caching). That means: put the unchanging content (system prompt, rules) at the very front and the changing content after it, and you cut costs significantly.

Do it the other way around — insert new content at the front every time — and the cache is invalidated entirely, putting cost back at full price.

One habit

Before you "paste a big chunk of stuff in", ask yourself: what proportion of this content does the model actually need? Usually less than 20%.

What you save isn't just tokens — it's context left over for the parts that actually matter.

Three common misjudgments

  1. Dumping the whole document in: tokens spent on irrelevant paragraphs, attention diluted, answer quality down.
  2. Retrieving without reranking: 20 chunks recalled with uneven relevance; the model easily latches onto the wrong one.
  3. Chunking by character count: a flat 500 characters will cut tables and code in half, fragmenting the semantics.

A chunking strategy that works

  • Split by heading level, no chunk larger than 800 characters
  • Keep code blocks and tables whole, don't split them
  • Prefix each chunk with "source title + section name" so the model knows the context when a chunk is retrieved
  • 10–15% overlap, so information isn't broken across chunks

Cost comparison

ApproachInput tokens for a 10K-character documentAnswer quality
Paste the whole thing~15KMedium (easy to get lost)
Retrieve top 5~2500High (focused)
Retrieve + summarize~800Medium-high (lossy)

In one sentence

A bigger context window doesn't mean you should stuff more into it. Being able to locate things matters more than being able to hold them.

The cost formula

Cost per request ≈ input tokens × unit price + output tokens × unit price. What's really expensive with long context is the input, and every turn resends the history all over again.

So in multi-turn scenarios the cost grows quadratically: 10 turns × 5K input per turn = 50K input tokens, not 5K. Do the total math first, then decide how big a window to open.

One more practical tip: fix the "system prompt + tool definitions" in place and put them up front for caching. This part is sent every turn, but its content doesn't change; once the cache hits, cost drops by an order of magnitude.

To close in one line: the bigger the window, the more restraint you need; the tokens you save are, essentially, the attention you save.

tokencontextcostprompt caching

Further reading

OpenAI pricing ↗ Anthropic pricing ↗