Tokens and embeddings

A large language model never sees the words you type. Before anything clever happens, your text is chopped into tokens and each token is turned into a long list of numbers called an embedding. These two steps are unglamorous, but they explain a surprising amount: what you pay, how much fits in a prompt, why models fumble some trivially easy questions, and how “semantic search” works at all.

From text to tokens

A neural network needs a fixed vocabulary of symbols, each with an integer ID. The obvious choices both have problems:

Whole words: the vocabulary becomes enormous, and any word not seen in training (a new product name, a typo, a rare surname) has no ID.
Single characters: the vocabulary is tiny and nothing is unknown, but sequences become very long, which makes models slower and harder to train.

Modern LLMs use subword tokenization, a middle path. Common words get a single token; rarer words are assembled from reusable fragments. A frequent word like “the” is one token, while an unusual word might be split into three or four pieces.

Byte pair encoding

The most influential recipe is byte pair encoding (BPE), adapted for neural translation by Sennrich, Haddow and Birch in 2016. Training a BPE tokenizer is simple:

Start with every text in a corpus split into characters (or raw bytes).
Count every adjacent pair of symbols and find the most frequent.
Merge that pair into a new symbol and add it to the vocabulary.
Repeat until the vocabulary reaches a chosen size.

Frequent sequences such as “ing” or ” the” quickly become single tokens. To tokenize new text you replay the learned merges. Many current tokenizers work at the byte level, so any string, in any script or even binary junk, can be encoded without an “unknown” token. Vocabulary sizes in practice range from tens of thousands to a few hundred thousand entries, and the details differ between model families.

A tokenizer is learned statistics about a training corpus, not linguistics. It has no idea what a syllable or a morpheme is.

Why tokens explain strange failures

Because the model receives token IDs, the individual letters inside a token are not directly visible to it. Ask how many r’s are in “strawberry” and the model may be looking at something like two or three opaque chunks. It can only answer by having absorbed, indirectly, which letters those chunks contain. That is why character-level tasks (counting letters, reversing strings, some arithmetic on long numbers, rhymes) have been notoriously brittle, even in models that handle far harder reasoning. Newer models often do better, sometimes by spelling a word out token by token before counting, but the underlying quirk has not gone away.

Why tokens drive cost and context

Tokens are the unit of work for an LLM:

Pricing: most APIs bill per input and per output token.
Context windows: the maximum prompt plus response length is measured in tokens.
Latency: output is generated one token at a time, so longer answers take longer.

For ordinary English, a rough rule of thumb is about four characters, or three-quarters of a word, per token. Treat that as a ballpark only. Code, numbers, unusual formatting and many non-English languages take more tokens per word, because the tokenizer’s merges were learned mostly from whatever text dominated its training data. The same message can therefore cost noticeably more in one language than another. When it matters, count with the provider’s actual tokenizer rather than guessing.

Embeddings: meaning as geometry

Once text is a sequence of IDs, the first layer of the model looks each ID up in a big learned table and returns a vector of hundreds or thousands of numbers. This is the token’s embedding. Nobody hand-designs these numbers; they are adjusted during training like every other parameter. The result is a space where tokens used in similar contexts end up near each other.

Mikolov and colleagues’ word2vec work in 2013 made this vivid. Their word vectors supported arithmetic such as king minus man plus woman landing near queen, suggesting that directions in the space can capture relationships. Such analogies are real but cherry-picked; the geometry is useful, not a tidy dictionary of concepts.

Inside a transformer, the initial token embedding is only a starting point. Each layer mixes in information from surrounding tokens, so the representation of “bank” drifts towards finance or rivers depending on context.

Measuring similarity

The standard way to compare two embeddings is cosine similarity: the cosine of the angle between the vectors. A value near 1 means they point the same way, near 0 means unrelated directions. It ignores vector length, which is often a nuisance factor.

What embeddings are used for

Dedicated embedding models turn a whole sentence or document into a single vector. Those vectors power:

Semantic search and retrieval, including retrieval-augmented generation, where relevant passages are found by nearest-neighbour search and pasted into a prompt.
Clustering and deduplication of large document collections.
Classification and recommendation, by comparing items to labelled examples or to a user’s history.

One practical warning: every embedding model learns its own coordinate system. Vectors from different models, or even different versions of the same model, cannot be meaningfully compared, so re-embed everything when you switch.

Key takeaways

Models read and write tokens, usually subwords learned by algorithms like BPE.
Letters inside tokens are hidden, which explains classic spelling and counting failures.
Tokens are the currency of cost, context length and latency, and languages are not tokenized equally.
Embeddings place text in a vector space where closeness tracks similarity of use, measured with cosine similarity.