Every few months a model ships with a bigger context window and the same question shows up: why not just make it infinite? Feed the model your whole codebase, your whole inbox, every document you own, and never think about context again.
The honest answer is that context length isn't a dial someone is being lazy about. It's the most expensive number in the model. Every additional token you allow into context costs you in three separate, compounding ways — and none of them go away with a bigger GPU. They just move the ceiling a bit higher.
Wall 1: attention math grows quadratically, not linearly
Transformers work by having every token attend to every other token — each token asks "how relevant is every other token to me?" and gets a weighted answer. That's the mechanism that makes them so good at using context in the first place. It's also the mechanism that makes long context expensive.
If you have n tokens, computing attention means comparing every token against every other token: n × n pairwise scores. That's O(n²) — quadratic, not linear. Double your context and you don't double the work. You quadruple it.
def attention_pairs(n_tokens: int) -> int:
# every token scores against every other token
return n_tokens * n_tokens
attention_pairs(4_000) # 16,000,000 score pairs
attention_pairs(8_000) # 64,000,000 score pairs — 4x, not 2x
Self-attention cost vs. context length — the gap between "linear" and "actual" is the whole problem.
This is why a request with a 100K-token prompt doesn't just take "a bit longer" than a 10K-token one — it can take on the order of 100x the attention compute, not 10x. Techniques like sliding-window attention, sparse attention, and linear-attention variants (state-space models like Mamba are the most aggressive version of this) all exist specifically to dodge this curve, usually by giving up the "every token sees every other token" guarantee in exchange for staying linear. That trade-off — less global visibility for a lot less compute — is itself a big reason context windows aren't unlimited even when engineers know exactly how to build a bigger one.
Wall 2: the KV cache is memory you can't give back mid-conversation
Quadratic compute explains why long prompts are slow. It doesn't fully explain why they're expensive to hold open. That's a separate problem: the KV cache.
To avoid recomputing attention from scratch for every new token during generation, the model caches the key and value vectors for every token it has already seen — the "KV cache." That cache has to sit in GPU memory for the entire lifetime of the request. It's not optional and it's not compressible for free: it's roughly 2 × layers × attention_heads × head_dim × bytes_per_value per token, and it only grows as the conversation continues.
def kv_cache_bytes(tokens: int, layers: int, heads: int, head_dim: int, dtype_bytes: int = 2) -> int:
# the "2" is for storing both Keys and Values
return 2 * layers * heads * head_dim * dtype_bytes * tokens
# a modest 32-layer, 32-head, 128-dim model, fp16, at 100K tokens of context:
kv_cache_bytes(100_000, layers=32, heads=32, head_dim=128, dtype_bytes=2)
# ≈ 52.4 GB — for the cache alone, before the model weights or the batch
The KV cache only grows. There's no garbage collection for "tokens the model already read."
This is why long-context requests are often memory-bound, not compute-bound, in production — a GPU can run out of room for the cache long before it runs out of patience for the math. It's also why serving a handful of very-long-context requests concurrently is disproportionately expensive compared to serving many short ones: each one is quietly reserving tens of gigabytes for the length of the conversation.
Wall 3: more context doesn't mean the model uses it evenly
Even setting compute and memory aside — suppose you had infinite GPUs and infinite VRAM — a longer context window wouldn't be a pure win. There's a well-documented effect usually called "lost in the middle": models are noticeably better at recalling information placed at the very start or very end of a long prompt than information buried in the middle of it.
Recall by position in a long prompt. The window can hold the fact. Whether the model actually leans on it is a different question.
This matters because "fits in context" and "the model will actually use it correctly" are two different claims, and only the first one is guaranteed by a bigger context window. Stuffing a million tokens into a prompt doesn't mean the million-and-first fact you cared about gets equal weight. Past a certain length, you're not just paying more for the same reliability — you can be paying more for less reliability, per fact, unless the model and its training were specifically built to counter the effect.
Putting the three together
None of these three walls is a bug someone forgot to patch. They're structural: quadratic attention is a direct consequence of the mechanism that makes transformers good at using context; the KV cache is the direct consequence of not wanting to recompute the past on every token; and attention dilution is a direct consequence of training a fixed-capacity model to weigh a variable, unbounded amount of information.
Three independent costs, one shared conclusion: the limit is a design decision, not a missing feature.
So what are labs actually doing instead of "just making it bigger"
Nobody serious is trying to brute-force their way to infinite context — the actual engineering effort goes into making the existing window cheaper and more reliable, and into working around the window rather than removing it:
- Sparse / sliding-window / linear attention trade full O(n²) visibility for a cheaper approximation — a token attends to a local window or a compressed summary of the past instead of literally every prior token.
- KV cache tricks — quantizing the cache to fewer bits, evicting low-importance entries (H2O-style eviction), or streaming it (StreamingLLM) — shrink wall 2 without shrinking the window itself.
- Positional scaling tricks (RoPE interpolation, YaRN, and similar) let a model trained at one context length generalize to a longer one at inference time, at some quality cost.
- Retrieval-augmented generation sidesteps the problem instead of solving it: rather than putting everything in context, fetch only the relevant slice for this specific query. It's usually the right call — most tasks don't actually need the whole document held in the model's attention at once, just the right paragraph of it.
Every one of these is a real, useful technique — and every one of them is a workaround for a limit that isn't going away, not a way of making context free.
The actual question isn't "how do we make it infinite"
It's "how much context does this task actually need, and what's the cheapest way to get the model the right context, not the maximum context." A bigger window is a bigger bill, in dollars, latency, and — past a certain point — accuracy. Treating context length as a scarce, expensive resource to spend deliberately, rather than a knob to max out, is the more useful mental model, and it's the one every serious long-context system is quietly built around.