what shares the budget

context_window = system_prompt + conversation_history + retrieved_documents
               + current_user_message + model_output
# all of it counts against the same limit -- not just what you type
In a multi-turn chat app, this means the budget shrinks every turn unless older history is summarized, truncated, or dropped — a common silent failure mode is a chatbot that "forgets" early instructions purely because they scrolled out of the window, not because the model chose to ignore them.

the "lost in the middle" effect

Multiple studies on long-context retrieval found that models are most reliable at using information placed at the very start or very end of the context, and measurably worse at retrieving/using information buried in the middle of a long input — even when the model's stated context window comfortably fits everything. This isn't a hard limit like running out of tokens; it's a quality-of-attention gradient across position.

Practical implication: when you control document order (e.g. in RAG), put the most important retrieved passage first or last, not third of seven. See RAG Architecture.

techniques for working within the limit

TechniqueTrade-off
Summarize old conversation turns instead of keeping them verbatimLoses detail; cheap and usually sufficient for casual chat continuity
Retrieve only the relevant slice of a large corpus (RAG) instead of pasting it all inRequires a retrieval pipeline; scales to corpora far larger than any context window
Prompt caching (reuse a cached prefix across calls)Doesn't reduce the logical context used, but cuts latency/cost when a large prefix — e.g. a system prompt or document — repeats across calls
Sliding window (drop the oldest turns)Simple, but loses arbitrarily old information with no summarization

where to go from here

RAG Architecture — retrieval instead of stuffing everything into context.
Chunking & Retrieval Strategies — how documents get split before retrieval.
Tokenization — why token counts don't map cleanly to word counts.