Transformer, step by step

Your progress

Keyboard shortcuts

TStart / stop the guided tour ← →Tour: back / next. Explore: step the current animation P · RPlay/pause · reset the current animation J / KNext / previous chapter (also ] [) DToggle light / dark ? · EscThis panel · close

Settings

How does a GPT-style transformer actually work?

Follow one sentence through every part of a tiny, real language model. Nothing here is faked: a 4-layer transformer with weights trained on a toy corpus runs live in your browser, and every number you see is computed by it. Edit the sentence at the top and watch the whole page recompute.

d_model = 82 heads4 layers vocab …… parameters GPT-3 has 175,000,000,000