Long contexts aren’t that good. See KV caches aren’t all that > long contexts aren’t actually that good for a starter.
Super long contexts get bad:
- [2307.03172] Lost in the Middle: How Language Models Use Long Contexts
- [2410.18745v1] Why Does the Effective Context Length of LLMs Fall Short? - Llama-3 can only effective use half of its 128K context (lol)
- Context Rot: How Increasing Input Tokens Impacts LLM Performance | Chroma Research - “In this report, we evaluate 18 LLMs, including the state-of-the-art GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 models. Our results reveal that models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows.”
There are also fundamental limitations about how much information can be carried forward by each token in long contexts, and some prompts are BAPO-Hard Problems: Limits & LLM Failures. This is why global information questions suffer for very long contexts:
- “Based on everything we’ve discussed, what is the single most important theme connecting all the topics we covered?”
- “Given all of this, what should I prioritize working on first?”
- “What assumptions am I making that I haven’t explicitly stated?”
These will do poorly with longer prefixes because they are inferential and distributed across the entire context.
So what’s better? Agents.
- Building Fast & Accurate Agents with Prime-RL Post Training for an example of a 35B parameter model that outperforms Opus 4.7 on a very specific task.