Sampling, context and limits
How LLMs pick each token, what temperature and top-p do, how context windows and cutoffs bound them, and why they hallucinate.
13 cards · 7 quiz questions · 9 min read
A trained language model is a function that, given some tokens, outputs probabilities for the next one. Everything you experience in a chat, the flowing answer, the occasional variety, the forgotten detail from an hour ago, the invented reference, comes from how that function is used and where its limits lie. This article covers generation, sampling settings, context windows, knowledge cutoffs and hallucination.
Generation, one token at a time
When you send a prompt, the model runs over it and produces a score for every token in its vocabulary. Those scores are turned into probabilities. A decoding strategy picks one token, it is appended to the text, and the whole step repeats. Generation stops at a special end token or a length limit. This loop is why responses stream in word by word and why long answers take longer and cost more.
The simplest strategy, greedy decoding, always takes the most probable token. It is predictable, but it can produce flat or repetitive text, because the locally best token is not always part of the best overall continuation. Most applications therefore sample from the distribution instead, shaped by a few settings.
Temperature and top-p
Temperature rescales the scores before they become probabilities:
- Low temperature (towards 0) sharpens the distribution, concentrating probability on the top candidates. Output becomes focused and consistent, close to greedy.
- High temperature flattens it, giving less likely tokens more chance. Output becomes more varied and more prone to drifting off course.
Top-p, or nucleus sampling, introduced by Holtzman and colleagues in 2019, trims the candidate list. Sort tokens by probability and keep the smallest set whose probabilities add up to at least p, say 0.9; sample only from those. When the model is confident, the set is small; when many options are plausible, it widens. This cuts off the long tail of bizarre tokens that can derail text. Related settings such as top-k (keep a fixed number of candidates) exist, and providers differ in which they expose and their defaults.
A practical rule: use low temperature for extraction, classification and code where consistency matters, and moderate settings for brainstorming or creative writing. Adjusting temperature and top-p aggressively at the same time can make behaviour hard to reason about.
Lowering temperature makes a model more consistent, not more correct. A confidently wrong answer at temperature 0 is still wrong.
Why outputs vary
With sampling, variation is by design. Less obviously, even temperature 0 through a hosted API is not a guarantee of identical output. How requests are batched together and small floating-point differences on the hardware can shift scores, and when two tokens are nearly tied, a tiny change flips the choice and the rest of the answer diverges. Providers may also update the model behind a name. If reproducibility matters, pin model versions, log prompts and outputs, and design evaluations that tolerate some variance.
Context windows
A model’s context window is the maximum number of tokens it can consider in one request: system instructions, conversation history, pasted documents, tool results and the response it is writing. Windows have grown dramatically over the years, with some models now accepting hundreds of thousands of tokens or more.
Key points about context:
- The model has no memory of its own. Its weights do not change as you chat. It “remembers” earlier turns only because the application sends them again each time.
- When the window fills, something must go. Applications truncate old messages, summarize history, or use retrieval to include only relevant pieces. Anything left out is invisible to the model.
- Bigger is not automatically better. The 2023 “Lost in the Middle” study found models used information at the start and end of long inputs more reliably than material buried in the middle. Newer models handle long contexts better, but it is still worth testing, and long prompts cost more and run slower.
Knowledge cutoffs
A model knows about the world only through its training data, which ends at some knowledge cutoff. Events after that are simply absent, and coverage of the months just before it is often thin. Models are not always reliable about their own cutoff date either. Tools such as web search or retrieval over your documents are the standard fix: current information goes into the context, where the model can use it.
Hallucination
A hallucination is fluent, plausible output that is false or unsupported: a nonexistent paper, an API function that was never written, a precise-looking statistic with no source. They are unsettling because they come in the same confident tone as correct answers.
Why does this happen? Pretraining optimizes for producing likely text, and a plausible fabrication can be statistically very similar to a true statement. The model has no built-in fact-checker and no reliable internal signal it can always surface saying “I do not actually know this”. Post-training helps but introduces its own pressures; a 2025 paper from OpenAI researchers argued that common training and benchmark scoring reward confident guessing over admitting uncertainty, much as a multiple-choice exam rewards guessing. How models represent their own uncertainty internally is still an active research area, and no current technique eliminates hallucination.
Practical mitigations
- Ground the model: supply relevant documents or retrieved passages and ask it to answer from them, citing where each claim comes from.
- Permit abstention: explicitly allow “I don’t know” and do not penalize it in your evaluations.
- Use tools: calculators, code execution, search and databases for facts and numbers.
- Verify automatically where possible: check that citations exist, run generated code against tests, validate structured output.
- Keep humans in the loop for high-stakes decisions, and calibrate trust per task through evaluation, not impressions.
Key takeaways
- Text is generated token by token by sampling from a probability distribution.
- Temperature and top-p trade consistency against variety; neither buys correctness.
- Context windows bound what a model sees; it has no memory beyond what is sent.
- Knowledge cutoffs and hallucination follow from how models are trained, so design systems that ground, check and verify.
You want consistent, focused answers for data extraction. Which setting helps most?
Sources
- Ari Holtzman, Jan Buys, Li Du, Leo Fournier & Yejin Choi — The Curious Case of Neural Text Degeneration paper Introduced nucleus (top-p) sampling.
- Nelson F. Liu et al. — Lost in the Middle: How Language Models Use Long Contexts paper Showed models used information at the start and end of long contexts better than the middle.
- Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala & Edwin Zhang — Why Language Models Hallucinate paper Argues training and evaluation reward guessing over admitting uncertainty.
- Chip Huyen — AI Engineering: Building Applications with Foundation Models book O'Reilly, 2025; covers sampling settings and hallucination mitigation in practice.