Same model as before, now with every number shown. We run a tiny transformer on the input "the cat sat" and ask it to predict the next word. Every matrix below is computed live from the ones before it. Hover or tap any colored output cell to see exactly which numbers produced it.
A real LLM uses the same formulas with bigger numbers: d ≈ 4,000 to 16,000, around 100 heads, a vocabulary of around 100,000, and thousands of tokens. Color: ■ positive ■ negative, stronger color = bigger value. The weights here are random, as in a model before training.
Each word is a one-hot row (a 1 in its vocabulary slot, 0 everywhere else). Multiplying by the embedding matrix E just picks out one row of E. In practice it's a table lookup, but writing it as a multiplication shows it's the same math as every other step.
Attention on its own ignores word order: it would treat "cat sat the" exactly like "the cat sat". So we add a position vector P to each token. The original paper uses sine waves of different speeds, so every position gets a unique pattern:
Before each sub-step, every token's vector is rescaled to mean 0 and variance 1. This keeps numbers from blowing up or shrinking as they pass through dozens of layers. (γ and β are learned scale and shift values. Here γ = 1 and β = 0.)
Three learned weight matrices turn each token into a query (what it's looking for), a key (what it offers to be matched on), and a value (what it passes along if it's chosen). Columns 0–1 belong to head 1 and columns 2–3 to head 2.
K and V are computed the same way, using their own weights WK and WV.
Now we follow head 1 (columns 0–1 of Q, K, V). Multiplying Q by KT takes the dot product of every query with every key. Entry (i, j) answers: "how relevant is token j to token i?"
Why divide by √dk? A dot product adds up dk terms, so its typical size grows like √dk. With dk = 128, raw scores would be huge, softmax would put nearly 100% on one token, and learning would stall. Dividing brings the scores back to a normal range.
A token may not look at tokens that come after it, so we add −∞ above the diagonal. Then softmax, applied to each row separately, turns the scores into positive weights that sum to 1. Because e−∞ = 0, the masked slots get exactly zero weight.
Each token's output is a blend of the value vectors, mixed according to its attention weights. This is the step where information actually moves between tokens: "sat" now contains a bit of "cat".
Head 2 runs the same steps 5–7 using columns 2–3 and learns its own attention pattern (shown below). The two head outputs are placed side by side (concatenated), then mixed by WO.
Attention doesn't replace X. Its result is added to X. Each layer only contributes a correction, and the original signal (along with its gradient during training) passes straight through. That's what makes stacks of 100+ layers trainable.
Each token now goes through a small 2-layer neural net on its own, with no mixing between tokens. First it expands to more dimensions (real models use 4×d), then ReLU zeroes out the negatives, then it projects back down to d. This is where most of the parameters live, and where much of the model's "knowledge" is stored.
X′ has the same 3×4 shape as X. That's one complete transformer block. A real model feeds X′ into block 2, then block 3, and so on, dozens of times, each with its own weights.
Only the last token's vector ("sat") is used to predict what comes next. After a final layer norm, we take its dot product with every word's embedding (many models reuse E here). A word whose embedding points the same way as the vector gets a high score, called a logit. Softmax turns the logits into probabilities.
The real next word was "on". The loss is how surprised the model was, measured as the negative log of the probability it gave the right answer. Being confident and right costs nothing. Being confident and wrong costs a lot.
The gradient with respect to the logits has a neat form: predicted minus actual. It says "raise the score of 'on', lower everything else, in proportion to how wrong each one was." Backpropagation pushes that signal back through every matrix above using the chain rule. Then each weight takes a small step downhill:
Stack that N times. Then compute , pick a word, append it, and repeat. Training is the step above, run over trillions of tokens.