Type any word, even a made-up one

What you learned in class: word-level vocab
Modern LLMs: subword pieces (BPE)

This is a toy tokenizer: a few hundred pieces and greedy longest-match. Real ones learn about 100k pieces from data, but they behave the same way, and they never output an unknown token, because raw bytes are the last resort.

How the pieces turn into a meaning, tied back to the layers

① Tokenizer ② Embeddings ③ Early blocks ④ Context attention: last piece looks back at the earlier pieces MLP: combines them into one whole-word meaning vector at "able" ≈ NOT + [glorp-ify] + CAN-BE (adjective) earlier in the prompt: "to glorpify = to wrap something in tinfoil" attention copies that definition into the vector "can't be wrapped in tinfoil" no learning here, justsplitting with a fixed list each piece has its ownlearned row in E learned in training, frommillions of un-…-able words in-context: no weightschange, it's all attention There is no <UNK>. An unfamiliar word is just an unfamiliar combination of familiar pieces, and the same Q/K/V + MLP machinery that builds sentences out of words builds words out of pieces.