Transformers and attention

Almost every large language model you have heard of is built on the transformer, an architecture introduced in the 2017 paper “Attention Is All You Need”. The name of the paper points at the key idea: a mechanism called self-attention that lets every token look at every other token and decide what matters. This article builds an intuition for that mechanism, why it won, and what it costs.

The job: predict the next token

Strip away the chat interface and a GPT-style model does one thing. Given a sequence of tokens, it outputs a probability for every token in its vocabulary being the next one. To write a paragraph, the system picks a token from that distribution, appends it, and runs the model again.

That sounds too simple to produce anything impressive. But predicting text well across trillions of tokens of books, code, forums and papers rewards the model for capturing grammar, facts, conventions, and patterns that look a lot like reasoning. Exactly what internal machinery forms, and how general it is, remains an open research question. “It only predicts the next token” is true about the interface and misleading about the capability.

Why context is the hard part

To predict what follows “The trophy didn’t fit in the suitcase because it was too”, a model needs to link “it” to the right noun, which may be many tokens back. Earlier recurrent neural networks (RNNs) read text one token at a time, updating a running memory state. That design had two serious problems:

No parallelism: step 500 cannot be computed before step 499, so training on long texts was slow on GPUs, which excel at doing many things at once.
Fading memory: everything seen so far is squeezed through a fixed-size state, so information from far back tends to get diluted. Gated variants like LSTMs helped, but long-range dependencies stayed difficult.

Attention mechanisms were already used alongside RNNs. The transformer’s bold move was to drop recurrence entirely.

Self-attention: a soft lookup

Think of each token as asking a question of the rest of the sequence. For every token, the model computes three vectors using learned weights:

A query: what this token is looking for.
A key: what this token offers, for others to match against.
A value: the information this token hands over if selected.

For a given token, its query is compared with every key using a dot product, a measure of how well they align. The scores are scaled and passed through softmax, which turns them into positive weights that sum to 1. The token’s new representation is the weighted blend of all the values.

It is like a fuzzy dictionary lookup: instead of retrieving one exact match, you retrieve a mixture, weighted by relevance. In the trophy example, a well-trained model can learn to put high weight from “it” onto “trophy”, pulling that meaning into the representation of “it”.

A few refinements make this work in practice:

Multiple heads: several attention computations run side by side with their own weights, letting different heads specialize in different relationships.
Causal masking: in a language model each position may only attend to itself and earlier tokens. Otherwise the model could simply read the answer.
Positional information: attention on its own ignores order, treating inputs like a bag. Positional encodings inject where each token sits, so “dog bites man” differs from “man bites dog”.

Attention decides where information flows. It does not, by itself, decide what to do with that information.

Layers: attention plus thinking time

A transformer block has two main parts. Self-attention mixes information across positions. Then a feed-forward network processes each position separately, transforming what was gathered; much of the model’s parameter count sits in these sublayers. Residual connections and normalization wrap each part to keep training stable.

Models stack dozens of these blocks. A useful, if simplified, picture: early layers handle local, surface-level patterns and later layers build more abstract, context-rich representations. Interpretability research has found specific circuits, such as heads that copy earlier patterns, but a full account of how a large model computes its answers is not yet available.

The original transformer had an encoder and decoder for translation. Most generative LLMs today use a decoder-only stack: just masked self-attention and feed-forward blocks, trained on next-token prediction.

Why attention won

Parallel training: during training, all positions in a sequence are processed at once, a perfect match for GPUs. This made it practical to train on vastly more data.
Direct long-range links: any token can attend to any other in a single step, rather than relaying information through hundreds of recurrent updates.
It scales: performance kept improving predictably as models and datasets grew, which drove the investment behind today’s models.

The cost of long contexts

The same all-pairs comparison that makes attention powerful makes it expensive. With standard full attention, every token is scored against every other, so the work grows with the square of the sequence length. Doubling a prompt roughly quadruples the attention computation.

Generation adds a memory cost. To avoid recomputing everything for each new token, systems keep a KV cache of keys and values for earlier tokens. Each step becomes cheap, but the cache grows with context length and can dominate memory for long conversations.

Engineers attack these costs with optimized attention kernels, attention restricted to local windows in some layers, sharing keys and values across heads, and alternative architectures. These make very long context windows practical, but they are engineering trade-offs, not a free lunch: long prompts still cost more, and models do not always use distant context reliably.

Key takeaways

An LLM’s core task is predicting a probability distribution over the next token.
Self-attention lets each token blend information from others, weighted by query-key matches.
Transformers replaced RNNs by training in parallel and linking distant tokens directly.
Full attention scales quadratically with length, and the KV cache grows with context, so long contexts are costly.