The objective: next-token cross-entropy

θ = every learned number: the embedding table E, WQ, WK, WV, WO, the MLPs, the norms, and the output head. At every position in the training text, ask the model "what comes next?" and add up −log(probability it gave the true next token). That single number is the whole training signal.

1 · Initall of θ random and small (e.g. Normal, σ ≈ 0.02). Embeddings too.
2 · Forwardrun a batch of text, get p for every position, compute 𝓛.
3 · Backwardchain rule gives ∂𝓛/∂θ for every weight, starting from p − y.
4 · Updateθ ← θ − η·∇𝓛 (AdamW in practice). Repeat about 10⁶ times.

Watch it happen: a tiny model trained live, right here

11-word vocabulary, 2-number embeddings (so we can plot them), and the same loss and update rule. To keep it fast in a browser, it predicts the next word from only the current word (no attention). The words start scattered at random. Watch them sort themselves into groups.

step 0 · loss –

After the model predicts:

Training text (repeated):