How LLMs are trained

Nobody programs a large language model’s knowledge or manners line by line. Its behaviour emerges from training: billions of small numerical adjustments that make some outputs more likely than others. Modern chat models go through several distinct stages, each with a different goal. Understanding those stages explains why models know so much, why they behave like assistants, and why “the model learned X” deserves careful reading.

Stage 1: pretraining

Pretraining is where the bulk of the work happens. A freshly initialized transformer is shown an enormous corpus, typically trillions of tokens of filtered web pages, books, code, scientific papers and more, and trained on one objective: predict the next token. After each batch, gradient descent nudges every parameter slightly to make the actual next tokens a bit more probable.

Repeated at vast scale, this produces a base model with broad knowledge of language, facts, code and styles. But a base model is a document continuer, not an assistant. Ask it a question and it might answer, or it might produce three more questions, as if continuing an exam paper. Data curation (deduplication, quality filtering, the mix of sources) matters enormously and is one of the least publicly documented parts of the process.

Scaling laws

In 2020, Kaplan and colleagues at OpenAI reported that a language model’s test loss falls smoothly and predictably, as a power law, as you increase model size, data and compute. That predictability was a big deal: labs could forecast the payoff of a larger training run before paying for it. Their analysis suggested putting most extra compute into bigger models.

In 2022, DeepMind’s Chinchilla paper by Hoffmann and colleagues revised this. With more careful experiments, they found that for a fixed compute budget, parameters and training tokens should grow roughly in equal proportion, about 20 tokens per parameter. Their 70-billion-parameter Chinchilla, trained on around 1.4 trillion tokens, outperformed much larger models trained on less data. Many earlier models had been undertrained.

Two caveats keep this in perspective. First, “compute-optimal” concerns training cost only. Because a smaller model is cheaper to run, many deployed models are now deliberately trained on far more tokens than Chinchilla would recommend. Second, scaling laws predict loss, not specific abilities; which skills appear at which scale is much harder to forecast.

Stage 2: supervised fine-tuning

To turn a base model into an assistant, labs continue training on curated examples of prompts paired with high-quality responses. This is supervised fine-tuning (SFT), also called instruction tuning. The objective is still next-token prediction, but now on a much smaller dataset that demonstrates the desired format: answer the question, follow instructions, decline certain requests, keep a consistent tone. SFT mostly shapes how existing knowledge is expressed rather than adding large amounts of new knowledge.

Stage 3: preference learning

Writing ideal responses is slow, but people can easily say which of two responses is better. Reinforcement learning from human feedback (RLHF), popularized by OpenAI’s 2022 InstructGPT paper, exploits this:

1. Sample several responses to a prompt and have people rank them.
2. Train a reward model to predict those preferences.
3. Optimize the LLM with reinforcement learning to score highly on the reward model, with a penalty for drifting too far from its previous behaviour.

The results were striking: labelers preferred outputs from a 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3. The risk is reward hacking: a learned reward model is an imperfect proxy, and models can learn to exploit it, for example by becoming verbose or telling people what they want to hear.

Anthropic’s Constitutional AI (Bai and colleagues, 2022) replaced most human harmlessness labels with AI feedback. The model critiques and revises its own responses against a written set of principles, and an AI judge picks preferred responses for the RL stage. This makes the intended values more explicit and cheaper to apply at scale, though the principles still have to be written, and interpreted, well. Related methods that learn directly from preference pairs, without a separate RL loop, are also widely used.

Stage 4: reinforcement learning on verifiable tasks

A more recent shift uses reinforcement learning where rewards come from automatic checks rather than preferences: is the final maths answer correct, does the code pass its tests, is the output in the required format? OpenAI’s o1 models, announced in September 2024, were described as trained with large-scale RL to reason before answering, though details were not published. DeepSeek’s R1 report in January 2025 documented an approach openly: rule-based rewards for correctness and format led a model to produce long chains of thought, including checking and revising its own work, and one variant developed this behaviour from RL on a base model without an initial supervised step.

This “reasoning” training markedly improved performance on maths, coding and similar benchmarks. It works best where answers can be verified; extending it to open-ended tasks remains harder. Whether the written chain of thought faithfully reflects how the model actually reached its answer is an open research question.

Each stage uses the same basic tool, adjusting parameters to make some outputs more likely. What changes is the data and the signal about what “better” means.

What “the model learned X” means

It is natural to say a model “learned French” or “learned to be honest”. What that really means is narrower: training changed parameters so that behaviour consistent with X became more probable on the kinds of inputs seen. It does not mean:

• There is a stored, inspectable rule or fact that can be looked up or edited cleanly.
• The behaviour will hold on unfamiliar inputs, phrasings or adversarial prompts.
• The model understands X in the way a person does, or has the goals the training intended.

Interpretability research is making real progress in locating features and circuits inside models, but nobody can yet read off a full account of what a large model has learned. Treat learned behaviours as strong tendencies to be tested, not guarantees.

Key takeaways

• Pretraining on huge corpora with next-token prediction builds a knowledgeable base model.
• Scaling laws make loss predictable; Chinchilla showed data should scale alongside parameters.
• SFT and preference learning (RLHF, AI feedback) shape a base model into an assistant.
• RL on verifiable rewards has driven recent gains in step-by-step reasoning on checkable tasks.
• “Learned” means “made more likely”, not “understood” or “guaranteed”.