Stages 0 to 5 of 6 · in progress

Transformers explained, one stage at a time

Two paragraphs of real text, taken all the way through to a next word prediction. Every number on this page came out of code you can download and run. Nothing is illustrative.

Sajith Amma18 August 2026Stages 0 to 5

Read this first

We are not training anything. A model trained on two paragraphs would be useless, and that is not the point. The point is that at this size every number fits on a screen, so you can watch the machinery work instead of being told about it.

The architecture here is the real one. Same operations, same shapes, same maths as the model you used this morning, at roughly a millionth of the scale. Where a number comes from a random starting value rather than from training, the page says so on the spot.

Stage 0

The text, and why this text

The paragraph was chosen, not found. Three things had to be true, and the last of them is the reason the rest of this article exists.

One word has to appear in two unrelated senses. Crane does. A machine in the first paragraph, a bird in the second. This is the entire reason transformers were built (the architecture inside every modern AI model, and the T in GPT): the same token has to be allowed to mean different things depending on what sits around it.
A pronoun has to point back at something. It points at the load. Later, attention is precisely the machinery that makes that pointing possible, and you will be able to see it happen.
Words have to repeat. Two hundred unique words in two hundred and fifty tokens teaches nothing and makes every picture on this page look like static. Here,

Read the last sentence of the text again. You knew which was which without being told. You did that instantly and without effort. The next six stages are a machine doing the same thing the hard way.

Stage 1

Text becomes tokens

A model never sees text. It sees integers. So the first decision is what counts as one unit. That unit is called a token: here, one token is one word or one punctuation mark. Everything after this point is arithmetic.

Every token in the text. The small number under each token is its id, which the next stage explains. The two crane tokens are marked. Note that they are identical: same word, same number, no way yet to tell the machine from the bird.

We split on words and keep punctuation separate. That is the wrong choice for a real model and the right one for reading, because every token here is something you recognise. Production models use subword tokenisers, which break unbelievable into pieces so a fixed vocabulary can spell words it has never seen. That is why token counts and word counts never match on your bill.

Tokeniser	Vocabulary size, how many distinct tokens it knows	Used by
This page	43	two paragraphs, word level
GPT-2 BPE	50,257	GPT-2
cl100k_base	100,277	GPT-3.5 and GPT-4
o200k_base	about 200,000	GPT-4o and later

Two of the names need unpacking. BPE is byte pair encoding, the algorithm that decides the pieces; cl100k_base and o200k_base are just the names of two such piece lists. And vocabulary size is not the context window, which is how much text the model can read at once, and not the training set size either. All three get called "tokens" and they differ by orders of magnitude. This table is only the vocabulary.

Stage 2

Tokens become numbers, and the numbers mean nothing

The vocabulary is every distinct token, sorted. A token's id is just its position in that list. This is worth dwelling on because it is where most intuitions go wrong.

The whole vocabulary. Width shows how often each token appears. The id is alphabetical position and carries no meaning whatsoever.

id 7 is not larger than id 3 in any sense that matters. Two ids being close together tells you the words are alphabetically adjacent and nothing else. bird and beam sit next to each other purely because of how they are spelled. Every scrap of meaning arrives in the next stage, and none of it is here.

Stage 3

Numbers become vectors

Each id is used to look up a row in a matrix. That row is the token's vector. The operation is a table lookup, not a multiplication.

Two words there, defined before they are used anywhere. A matrix is nothing more than a grid of numbers, rows and columns. A vector is a row of numbers read together as one position: two numbers place a point on a school graph, three place it in space, and sixteen is the same idea carried on to sixteen axes. So "the token's vector" means the token's point in a sixteen dimensional space.

the lookup

$$x_i \;=\; E\bigl[\,\mathrm{id}_i\,\bigr]$$

Every letter, before it is used anywhere. E is the Embedding matrix, the grid of numbers pictured just below, shape : one row per vocabulary entry, each row numbers long. i here counts tokens through the text, so id_i is the i-th token's vocabulary id from stage 2, and x_i is the row that comes back, the token's vector. The whole equation reads: to get token number i's vector, open row id_i of E.

Those numbers are the model's entire idea of what a word is.

The embedding matrix, actual values. Red is negative, blue positive, intensity is magnitude. Hover a row to read it. These are the real numbers the code produced. Random, but the same random numbers every run, so what you see is exactly what the code makes.

d is the model dimension. We use so a row fits on a line. GPT-2 small uses 768. Frontier models use tens of thousands. Nothing about the operation changes with size, only the width of the row.

Every vocabulary word, projected from dimensions down to 2. A screen has two dimensions, so the sixteen have to be squashed onto the two that keep as much of the spread as possible. The standard tool for that is called principal component analysis, PCA, done here in a few lines with no library, so nothing clever hides in it. These two axes keep of the spread, which is why the picture looks flat.

This cloud has no structure, and that is the honest starting point. We did not train, so E is random. In a trained model, words used in similar ways drift together and this picture develops shape. The distance between this cloud and that one is exactly what training buys, and being able to see the difference is worth more than being shown a tidy diagram of the destination.

Both crane tokens still have the same vector. Position 1 and position , identical row, identical numbers. Nothing so far can separate the machine from the bird. Stage 4 fixes part of it and stage 5 fixes the rest.

Stage 4

Position, and why sines

Everything is in place, and nothing works yet. This stage is the one where you find out what the last stage could not say, and then have to invent the fix. So it is written the way it was found, wrong answers first.

Where we have got to

A school graph has two dimensions, x and y, so a point takes two numbers. Add depth and it is three, x, y, z, so a point takes three. Nothing changes if you keep going. At sixteen dimensions a point takes sixteen numbers and there is nothing else to it. You cannot picture it, and you do not need to.

So every word in the vocabulary now has one reserved place in a sixteen dimensional space. Not a meaningless id any more. A direction, and a real distance from every other word. That is a good position to be in.

And then you notice the hole

Words repeat. crane sits at position and again at position , and both times it is the same row of the matrix. The same sixteen numbers. A machine in the first paragraph and a bird in the second, and so far they are the same thing.

There is a worse version of the same hole, and it is the one that settles the argument. Attention, the stage after this one, is permutation invariant. Shuffle its inputs and it computes exactly the same answer. It has no idea what order anything arrived in.

Two different sentences. One input. Same five rows of the embedding matrix, in a different arrangement, and nothing downstream can tell the two apart. Order will not arrive by itself. It has to be put into the vectors here.

So the plan writes itself

Every word already carries sixteen numbers. So give every position sixteen numbers too, and add the two together. Word, plus where the word is.

$$x'_{\mathrm{pos}} \;=\; \underbrace{x_{\mathrm{pos}}}_{\text{the word}} \;+\; \underbrace{PE[\mathrm{pos}]}_{\text{the place}}$$

The shape of the answer is easy. The question underneath it is not, and it is the one nobody warns you about: what should those sixteen numbers be? The obvious answers all fail, and it is worth watching them fail, because each failure hands you a rule.

Attempt 1 Just count. 0, 1, 2, 3, up to 1000.

The first thing anyone tries, and it is not stupid. Every position gets a different number, it never runs out, and it is obvious how far apart two positions are.

Then look at word 500. Its own sixteen numbers are around 1, some a bit more, some a bit less. Now add 500 to that. The word contributes 1, the position contributes 500. You have built a machine that knows precisely where every word sits and can no longer tell you what any of them are.

Dead. The position drowns the word.

Rule 1. The numbers must stay small, whatever the position.

Attempt 2 Squeeze every text into 0 to 1.

Fix rule 1 by scaling. First word 0, last word 1, everything spread between. Now nothing is ever bigger than 1. Sounds solved.

	a 10 word sentence	a 1,000 word document
word 5 gets	0.5	0.005
word 500 gets	does not exist	0.5
gap to the next word	0.1	0.001

Two things break at once. 0.5 means "word 5" in one text and "word 500" in the other, so the same value says two unrelated things. And the step from one word to the next is a hundred times bigger in the short text, so "the very next word" looks different in every document and can never be learned.

Dead. It depends on the length of the text.

Rule 2. A position must mean the same thing everywhere, and it must not run out. Train on 512 words, then somebody pastes 3,000. Word 2,000 still needs a value.

Attempt 3 Give every position its own random values.

Small, unique, and no scaling problem. But now think about what the model actually needs to know.

In the crane lifted the beam, what matters is that lifted comes right after crane. Not that crane happens to be word 1. Paste that same sentence into the middle of a report and crane is word 500, lifted is word 501. The relationship has not changed at all. Only the numbers did.

With random values, position 500 and 501 have nothing to do with each other. So the model would have to learn "1 and 2 are neighbours" separately from "500 and 501 are neighbours", and again for every other pair. It never finishes. There are millions.

Dead. Nearness carries no information.

Rule 3. "One word later" must be the same thing everywhere, so it can be learned once and used for ever.

Three attempts, three rules, and a fourth that is obvious

Rule 4 barely needs a failure to prove it: no two positions may read the same, or the whole exercise is pointless. Now the four rules together kill almost everything.

Idea	1. stays small	2. never runs out	3. distance is fixed	4. all different
Count 0, 1, 2, 3	no	yes	yes	yes
Squeeze into 0 to 1	yes	no	no	yes
Random per position	yes	no	no	yes
One circle	yes	yes	yes	no
Many circles, different rates	yes	yes	yes	yes

Read the rules again and see how little freedom is left. Stays small for ever means the values have to come back on themselves rather than grow. Something that comes back on itself and never runs out is a circle. And going round a circle by a fixed amount is the same move wherever you started, which is rule 3, free.

One circle fails rule 4, because it comes back to where it began. So: more than one circle, each turning at a different rate. That is the entire idea, and everything below is what it looks like in numbers.

A circle, in numbers

A pair $\bigl(\sin\theta, \cos\theta\bigr)$ is a point on a circle at angle $\theta$, and here $\theta$ grows steadily with the position. Each pair is the tip of a clock hand, and the two numbers are just how far up and how far across that hand is pointing. Turn the hand and watch the two numbers change.

drag

One pair of numbers is one point on this circle. Across is the cosine, up is the sine, and neither can ever leave the range −1 to +1 because the point cannot leave the circle. The ticks are where the fastest pair, pair 0, lands at each of the first seven positions. Look at 0 and 6, marked in orange. After six positions the hand is nearly back where it started, which is the problem the next section deals with.

Why eight circles, when a token has sixteen numbers?

Because one circle costs two numbers, not one. A point needs both coordinates to be located: how far across, and how far up. So each circle takes two of the sixteen slots, and sixteen divided by two is eight. Sixteen circles would need thirty two numbers.

Position , the word , split the way the model reads it. Sixteen numbers, taken two at a time, giving circles. The columns are not shuffled or chosen: column 0 and 1 are one circle, 2 and 3 are the next, all the way across. Circle 0 here is what everything below calls pair 0. Same thing, two words for it.

One number on its own would be ambiguous. The sine alone cannot say where on the circle you are, because a hand pointing up and to the right and a hand pointing up and to the left have the same height. Position and position in this very text are a real case: their sines are , near enough identical, and only the cosines, , tell them apart. Both numbers, or no position.

One more thing worth being exact about, because it is easy to assume otherwise. This row is added to the word's vector, not multiplied by it. The word's sixteen numbers and the position's sixteen numbers are worked out with no knowledge of each other, then summed slot by slot.

Each clock has its own turning rate, written $\omega_i$. That is the only thing $\omega$ means here: how far that hand swings round when you move one position forward, measured in radians. A radian is a unit of angle, nothing more: a full circle is $2\pi \approx 6.28$ radians, so one radian is about 57°, a bit less than a sixth of a turn. Pair 0 has $\omega_0 = 1$, so it turns one radian per token and comes right back round every 6.3 tokens. Pair 7 turns 0.0003 radians per token, which is almost imperceptible.

So the fast hands tell you exactly where you are locally, and the slow ones tell you roughly how far into the text you are. Read all together and only one position fits, the same way an hour hand and a minute hand pin down one time.

The actual turning rates in this model. Not illustrative numbers: these come out of the code, for d = . The first pair completes a full turn every six positions. The last takes twenty thousand, so across our sixty four tokens it barely moves at all, which is exactly what makes it useful for saying "near the beginning" rather than "exactly here".

The first two words of the text, all the way through

Enough description. Here is the actual arithmetic for the first two words, every number taken from the file. The first word is the, at position 0. Its token id is , so its vector is row of the embedding matrix, written Embedding[], or E[] for short from here on. The position row is PositionalEncoding[0], or PE[0].

Two words, three lines each, one operation. Add the two rows above the line, column by column, and you get the row below. Nothing else happens at this stage.

Look at PE[0] first. It reads 0, 1, 0, 1, 0, 1 all the way across. Every circle is at angle zero because nothing has moved yet, and $\sin 0 = 0$, $\cos 0 = 1$. So the very first word barely changes: the even columns are untouched and every odd column gains exactly 1.

Now PE[1], one step later. It is no longer uniform, and reading it two at a time says exactly why.

circle	columns	turned by position 1	(sin, cos)

One step forward and circle 0 has swung a whole radian. Circle 7 has barely left the starting line and still reads the same as position 0. That spread is the point. The fast circles say where you are locally, the slow ones say roughly how far into the text you are.

And this is what finally separates the two cranes. Both are token id , so both start from the identical row E[]. One then has PE[] added to it and the other has PE[]. Same word half, different place half, so the sums come out different. Before this stage the machine held two identical vectors. After it, two different points.

Every one of the tokens gets the same treatment. Look up your row by word. Look up your row by position. Add.

If it is a circle, does it not come back to where it started?

It does, and one pair on its own would be useless because of it. Pair 0 comes back round every 6.3 positions, so on that pair alone position 0 and position 6 are almost the same two numbers. Position 13 too, and 19, and so on for ever.

Which is why there is never just one. All pairs are read together, and they are turning at rates that are nothing like each other. When the fast hand has come back round, the slower ones are nowhere near where they started. Move the position and watch which hands move.

Every hand at once, drawn from the actual matrix. The faint hand is where that pair was one position earlier, so the gap between the two is what that pair contributes to telling those two positions apart. Pair 0 swings a sixth of a turn every step. Pair 7 does not visibly move at all. Now drag back to position 0. Pair 0 has gone a full lap and come back to within of where it started, while every other hand has moved somewhere genuinely new. The row underneath is that position's numbers, the first two, in plum, being pair 0.

The combination is the fingerprint. Pair 0 repeats every positions, pair 1 every , pair 2 every , and by pair 7 you would have to read positions to get back to the start. Each hand on its own is ambiguous. All of them together are not.

And now the notation, which says the same thing

for pair number i, at position pos

$$\begin{aligned} PE[\mathrm{pos},\,2i] \;&= \sin\!\bigl(\mathrm{pos}\cdot\omega_i\bigr) \\[2pt] PE[\mathrm{pos},\,2i+1] \;&= \cos\!\bigl(\mathrm{pos}\cdot\omega_i\bigr) \\[6pt] \omega_i \;&= \frac{1}{10000^{\,2i/d}} \end{aligned}$$

Four letters appear, and each one has already been met above. Here they are again in one place, nothing left to guess.

$d$	how many numbers one token has. for this model, 768 for GPT-2 small. Fixed for the whole model, and it is the only thing the formula needs to know about the model.
$i$	which circle. 0 to , since numbers taken two at a time give circles. $2i$ and $2i+1$ turn a circle number into its two column numbers: circle 0 owns columns 0 and 1, circle 3 owns columns 6 and 7.
$\mathrm{pos}$	which word. 0 to 63 here, counting along the text. The only letter that changes as you read the sentence.
$\omega_i$	circle i's turning rate, radians per position, straight from the table above. Note that $\mathrm{pos}$ is not in its formula: a circle's rate never changes, only its angle does, and the angle is $\mathrm{pos}\cdot\omega_i$. The formula is a compact way of saying "circle 0 turns one radian per position, and each circle after it turns $\sqrt{10}\approx 3.16$ times more slowly". The 10000 has no deep meaning: it only sets the spread between the fastest circle and the slowest, and with $d=16$ it works out to $10000^{2/16}=\sqrt{10}$ per step.

So the two equations read: at word number pos, circle number i has turned to angle pos times $\omega_i$; write its sine in the circle's even column and its cosine in its odd column. Each row is the pairs laid end to end, giving numbers, and that row is added to the word's vector rather than stuck on the end of it:

$$x'_{\mathrm{pos}} \;=\; x_{\mathrm{pos}} \;+\; PE[\mathrm{pos}]$$

$x_{\mathrm{pos}}$ is the word's vector from stage 3. $PE[\mathrm{pos}]$ is this stage's row for that position. $x'_{\mathrm{pos}}$, with the dash, is the sum, the vector every later stage actually uses. It is the third line of the worked example above, exactly. Nothing here is learned, and every value stays between $-1$ and $+1$ however long the text gets.

Back to rule 3, and why sines rather than any other bounded thing

This is the part usually answered with a hand wave about waves, and the real answer is linear algebra. Take the dimensions in pairs. Pair i of PE[pos] is

$$\bigl(\;\sin(\omega_i \cdot \mathrm{pos}),\;\; \cos(\omega_i \cdot \mathrm{pos})\;\bigr)$$

which is a point on a circle at angle ω_i·pos. Here k is just "how many positions forward", so moving forward by k adds ω_i·k to the angle. And rotating by a fixed angle is one fixed operation, the same recipe wherever the hand happens to point (the technical term is a linear map), so

$$PE[\mathrm{pos}+k] \;=\; R_k \, PE[\mathrm{pos}]$$

where R_k is one matrix meaning "advance every circle by k steps": it turns each of the eight pairs by its own fixed angle ω_i·k and never mixes one pair with another (the technical name for that shape is block diagonal). The thing to stare at: R_k is built from k alone, pos is not in it. That is the whole reason. "Three tokens back" is one fixed linear transformation, identical everywhere in the sequence, so a layer can learn it once as a matrix instead of learning it separately for every position it might occur at.

checked, not asserted

check_rotation() in stage 4 builds R_k from k alone, applies it to PE[pos], and compares against PE[pos+k]. Across 40 positions and 19 offsets the largest disagreement is , which is as close to zero as computer arithmetic ever gets.

It failed the first time it ran, with an error of 1.63. The pair is stored as (sin, cos) and the rotation had been written for (cos, sin), so the signs were crossed. Worth mentioning because the sentence above would have read exactly as convincing while being false.

The positional encoding, actual values. One row per position, columns are dimensions. Red is negative, blue positive. The pattern is not decoration and it is worth reading, so it is unpacked below.

Reading that picture: why is the right side all blue?

Two things stand out. Red appears only on the left, and the further right you look the more the columns settle into fixed stripes that barely change down the page. Both come straight from those turning rates.

Columns alternate: even columns are $\sin$, odd columns are $\cos$, one pair per clock. The angle a clock has turned through by position 63 is $63\,\omega_i$, and that is the whole explanation.

Pair	Turned through, by position 63	sin column	cos column	What you see
0, far left	63.0 rad, about 10 full turns	−1 to +1	−1 to +1	red and blue flickering every few rows
2	6.3 rad, exactly 1 turn	−1 to +1	−1 to +1	one slow sweep through red and back
4	0.63 rad, a tenth of a turn	0 to +0.59	+0.81 to +1.00	pale blue, and deep blue. no red possible
7, far right	0.02 rad, a three hundredth of a turn	0 to +0.02	+1.000 to +1.000	almost white, and solid blue. flat all the way down

Red requires the angle to pass $\pi$, half way round. Sine and cosine only go negative once the hand crosses to the left or bottom half of the circle, and that takes $\pi \approx 3.14$ radians. The slow clocks never get near it inside sixty four positions, so their sine stays just above zero and their cosine stays pinned at one. That is the pale, solid, unchanging pair of stripes on the right. The fast clocks on the left go round ten times, so they sweep the full range repeatedly and produce the flicker.

Which means most of the right hand side is doing nothing here, and that is honest rather than a flaw. The final cosine column varies by 0.000 across all sixty four positions. It is constant. The one before it varies by 0.002. Those dimensions are not earning their place in a sixty four token text.

They exist for length. At four thousand tokens the slow clocks are the only ones that can still tell the beginning from the end, because by then the fast ones have gone round hundreds of times and are useless for anything but local detail. You are looking at an encoding built for a much longer text than ours, which is exactly what it is.

The two cranes come apart

The same 64 token vectors, before and after position is added. On the left, every repeated word sits exactly on top of itself, so 64 tokens show fewer than 64 points. On the right they separate. The two crane tokens are marked in both.

Distance between the two crane vectors: before, after. They are no longer the same vector, which is the whole job of this stage.

But nothing here knows one is a machine and one is a bird. All position did was make them different. It has no idea in what way they ought to differ, because it never looked at the surrounding words. That is stage 5, and it is the one everything else has been building toward.

How would anyone ever think of this?

Worth asking, because the honest answer is more useful than the flattering one. They did not invent it. They recognised it.

You already know this trick. You have known it since you learned to count in binary.

Bit 0 flips every step. Bit 1 every 2 steps. Bit 2 every 4. Bit 3 every 8. Read down any column and it is a hand going round at its own rate. Several channels, each cycling at its own rate, and the combination is unique. That is the positional encoding, exactly. Binary steps by a factor of 2 from one column to the next. The transformer steps by $\sqrt{10}\approx$ .

An odometer is the same thing. So is a clock: the seconds hand is bit 0, the hour hand is bit 3, and nobody has ever been confused about what time it is.

So why not use binary directly? Because bits jump. Going from 3 to 4 flips three bits at once, so 3 and 4 look wildly far apart while 4 and 5 look adjacent. The next stage does linear algebra on these numbers and needs near positions to be near. Sine and cosine are the smooth version of a bit. That is the whole leap, and it is one sentence long.

what the authors actually reported

This is the reassuring part. The paper that introduced all of this, Attention is All You Need from 2017, says they also tried the boring option, letting the model learn its position vectors like any other weights, and that the two "produced nearly identical results". They kept the sinusoids on a stated hypothesis: that it might extrapolate to texts longer than the ones trained on.

So it was not a flash of genius that obviously beat everything. It was a considered guess that tied on the benchmark and was kept for a reason they were careful to label as a guess. Years of textbooks have since made it look inevitable. It was not.

If you want to think like that

Three things, none of them about being clever.

Write what the answer must satisfy, before looking for one. The creative work in this whole stage was rule 3, and rule 3 is a sentence, not a formula. Once the four rules were written honestly, everything else died on its own and one idea was left. Most people skip straight to "how do I do this", which has only guesses for answers. "What must be true of anything that does this" has answers you can check.
Ask what else in the world already has this shape. Many unique labels, bounded, where nearness means something. Clocks solved it. Odometers, binary, musical octaves, vernier calipers. Every one of those is layered periodicity at different rates. You cannot recognise a shape you have never been shown, which is the real reason a wider reading list makes people inventive.
Build the dull baseline and measure it. They did, and it tied. If your clever idea only draws with the obvious one, that is worth knowing early and cheaply.

The transferable rule, in one line. When you need many unique labels that stay bounded, and where nearness has to carry meaning, stack periodic things at rates spaced by a constant factor. That is now yours. Use it somewhere else and you will have done the same kind of thinking, which was never the mysterious part.

Stage 5

Attention, and how a word finally reads the room

This is the one the whole architecture is named after, and it is the stage that makes the machine from the bird. Same shape as before: the hole first, then the obvious fixes failing, then what is left.

Where we have got to

Every token now carries numbers that say which word it is, plus numbers that say where it sits, added together. The two crane tokens are finally different, which felt like progress.

And then you notice the hole

They are different only because they sit at different positions. Swap the two paragraphs around and the two cranes would swap vectors. Neither one contains anything about the words around it.

That is the hole. A word's vector still does not know its neighbours. The first crane sits beside lifted and beam, the second beside is, also, bird, and so far nothing anywhere in the machine has looked at that.

Your text contains a second, sharper case, and it is the one stage 0 promised to deliver. The word it also appears twice, and points somewhere different each time.

Same word, same id, two different meanings. A human resolves both without effort. Position cannot do this: it only knows one is the 16th token and the other the 33rd. Something has to actually look at the surrounding words.

So the plan writes itself, again

Each word should pull in some of the words around it. Its new vector becomes a blend: mostly itself, plus a helping of whatever else in the sentence matters to it. The question, exactly as in stage 4, is the one underneath: how much of each other word?

Attempt 1 Average in the whole paragraph, equally.

Simple, and it does put context into every vector. But every word receives the same average, so every word gets the same context and none of them is distinguished from any other. Both cranes get identical helpings, so they are exactly as alike as before.

There is a second failure on top. the appears times in these two paragraphs, so a flat average is mostly a blend of the, the and the.

Dead. One average for everybody distinguishes nobody.

Rule 1. Each word needs its own mixture, not a shared one.

Attempt 2 Average in the nearby words, weighted by distance.

Closer words count more, far ones count less. Now every word gets a different mixture, so rule 1 is satisfied, and it genuinely helps: the words next to crane really are the ones that say which crane it is.

Then try it on the two it tokens. The first needs load, five tokens back. The second needs crane, six tokens back. A rule based on distance alone would hand both of them set, then and stands, whichever words happen to be adjacent, and those are useless. And in the site was quiet after that, the near words carry nothing at all.

Dead. Distance is not relevance.

Rule 2. How much one word matters to another has to depend on what the words are, not on how far apart they landed.

What is left: let every word choose for itself

Rules 1 and 2 together leave exactly one shape of answer. For every pair of words, work out a number saying how much the second matters to the first, and use those numbers as the mixing amounts. Every word ends up with its own recipe, and the recipe is computed from the words themselves.

That number is called a score, and it is where all the machinery comes from. Every piece that follows is the answer to one question about how to get it.

Question 1: how do you score two vectors against each other?

The standard way to ask "how much do these two vectors agree" is the dot product. It is the simplest operation there is: multiply the two vectors slot by slot, then add up the results. One number out.

A dot product, done in full. Two real rows from the file, multiplied slot by slot, then summed. Big and positive when the two vectors point the same way, negative when they point opposite ways, near zero when they are unrelated.

Question 2: so why not just score $x_i \cdot x_j$ and be done?

Because it fails in two ways, and each failure is why one of the famous letters exists.

Failure A Relevance is not mutual.

it is hunting for a noun to attach itself to. load is not equally hunting for a pronoun. But the dot product is symmetric: $x_i \cdot x_j$ and $x_j \cdot x_i$ are the same number, always. So this score physically cannot say "load matters a lot to it, but not the other way round".

Failure B Every word would love itself most.

$x_i \cdot x_i$ is the vector's own length squared, which is large and positive by definition. So every word's best match is guaranteed to be itself, and attention becomes a very expensive way of changing nothing.

Both are fixed by the same move. Before scoring, send each vector through two different transformations.

the two projections

$$\begin{aligned} q_i \;&=\; x'_i \, W_Q &&\text{what token } i \text{ is looking for} \\[3pt] k_j \;&=\; x'_j \, W_K &&\text{what token } j \text{ advertises} \end{aligned}$$

Every letter. x' is the vector from stage 4, word plus position. W_Q and W_K are two matrices of learned numbers, both , random here because we do not train. q is short for query, k for key. d_k is how long a query or key is, in this model.

Read them as a library. The query is the question a word is asking. The key is the label on a word's spine. You score by holding the question up against every spine.

And now watch both failures disappear. The score is $q_i \cdot k_j$, which written out in terms of the original vectors is

$$q_i \cdot k_j \;=\; x'_i \, \underbrace{W_Q W_K^{\top}}_{\text{one matrix, call it } M} \, x'^{\top}_j$$

Because M sits in the middle and is not symmetric, the score from i to j is free to differ from the score from j to i. That kills failure A. And a word's score with itself is now $x'_i M x'^{\top}_i$, which has no reason to be large. That kills failure B. Two matrices instead of one, and both problems go at once.

Question 3: and why a third letter, V?

Because what you match on is not what you want to take away. You find a book by its spine, then you read the pages, and the spine is not the pages. So a third projection, holding what a word actually contributes once it has been selected.

$$v_j \;=\; x'_j \, W_V$$

Key is the label on the box. Value is what is inside it. Keeping them separate lets a word be easy to find for one reason and useful for another.

Question 4: scores are any size. How do they become a mixture?

A score can be 40, or −12, or 0.3. You cannot mix with those. A mixture needs amounts that are never negative and that add up to 1, or the result is not a blend of the inputs and can grow without limit.

The tool for that is softmax. Three steps, and it is worth seeing them separately because the name hides how simple it is.

Softmax, one step at a time, on real scores from the file. Raise e to the power of each score, which makes everything positive and pushes the big ones much further ahead. Add those up. Divide each by the total, so they sum to 1. The name is just "a soft version of picking the maximum": the biggest score gets most of the weight, but not all of it.

Question 5: why divide by $\sqrt{d_k}$ first?

This is the step that looks like an arbitrary detail and is not. A dot product of two vectors numbers long is a sum of products, so it grows as the vectors get longer, roughly like $\sqrt{d_k}$. Feed big numbers into softmax and the exponential runs away with them: one weight goes to nearly 1 and the rest to nearly 0. Attention stops being a blend and becomes a single pick.

Measured on the last token of the text, which is allowed to see all :

	largest single weight	tokens getting more than 1%
without dividing
divided by $\sqrt{d_k}$ =

Same weights, same text, one division. Attention spread over tokens instead of .

Question 6: why is half the picture always blank?

One more rule, and it comes from the job rather than the maths. This model exists to predict the next word. If token 15 were allowed to look at token 16 while learning to guess token 16, it would simply read the answer.

So a token may look at itself and everything before it, and nothing after. The scores for the future are set to minus infinity before the softmax, which makes their weights exactly zero. It is called the causal mask, and it is the one visible pattern on this page that is not noise, because it does not depend on a single learned number.

All six answers, in one line

scaled dot product attention

$$\mathrm{Attention}(Q,K,V) \;=\; \mathrm{softmax}\!\left(\frac{Q K^{\top}}{\sqrt{d_k}} + \mathrm{mask}\right) V$$

Read right to left against the questions above. $QK^{\top}$ is every query scored against every key, question 1 and 2. $\div\sqrt{d_k}$ keeps softmax from saturating, question 5. $+\,\mathrm{mask}$ blanks the future, question 6. $\mathrm{softmax}$ turns scores into a mixture, question 4. And multiplying by $V$ is the mixing itself, question 3: each token's output is the weighted sum of everybody's values.

Capital $Q$, $K$, $V$ are just all the little $q$, $k$, $v$ vectors stacked into matrices, one row per token, so the whole sentence is done in one multiplication instead of a loop.

One token, all the way through

Token , the word it. Every number below is out of the file. It may look at tokens and no more.

One token, one row at a time. Its raw score against each token it can see, then those divided by , and then softmax turning those into weights. Read the last row as percentages and it says what fraction of this token's new vector came from each of the others. Only the eight strongest are shown; every weight is in the picture below.

Every token, and what it looks at

Click any word. The sentence below it lights up to show where that word's attention went, and the future is greyed out because the mask forbids it.

pick a word

Deeper background means more attention. Struck through means blocked by the mask. On the random head the pattern means nothing, because the numbers were never trained, and it is shown anyway so you can see what untrained really looks like. The hand set head is explained below and is not trained either. It was constructed.

The whole thing at once: who attends to whom. One row per token asking, one column per token being looked at. The triangle is real. Everything above the diagonal is exactly zero because of the mask, and no amount of training would change it. The texture inside the triangle is noise, since these weights were never trained.

Be clear about what is and is not meaningful here. The mechanism is real, and every number came out of the code. But $W_Q$, $W_K$ and $W_V$ are random, exactly like the embedding matrix back in stage 3, so the pattern of who attends to whom means nothing. Training is what turns this noise into heads that do nameable jobs.

Two things survive being untrained, and both are worth having. The mask is real structure. And the mechanism genuinely does what it was built for, which is the next block.

The two cranes, at last

Here is the measurement the whole page has been walking toward. How far apart are the two crane vectors, at the end of each stage?

Distance between the two crane vectors. Zero at stage 3, because they were the identical row. Position pulls them apart. Attention pulls them further, and for a different reason: their neighbours differ, so they pull in different mixtures. This number moves even with random weights, and it is the real result of the stage.

The hand set head, and what training is actually for

Everything above is honest and slightly unsatisfying, because it never points at anything. So here is one head where the numbers were chosen rather than trained, to show what a trained head would look like. It is labelled hand set everywhere it appears, and it is not pretending to be a result.

The construction is the smallest one that works. Take $d_k = 1$, so each token's query and key are single numbers:

$$\begin{aligned} q_i \;&=\; x'_i \cdot E[\texttt{it}] &&\text{how much does this token look like \textit{it}} \\[3pt] k_j \;&=\; x'_j \cdot E[\texttt{load}] &&\text{how much does this token look like \textit{load}} \end{aligned}$$

The score $q_i \, k_j$ is then large exactly when a token resembling it meets one resembling load. Switch the head above to hand set and click it at position . It lands on load with weight .

and it gets the other one wrong, on purpose

Click the second it, at position . It should point at crane. It points at load as well, with weight , because this head only knows one trick and applies it everywhere.

Left in deliberately. Hand setting a head to do one job is easy. Finding numbers that get every case right, across billions of words, with no human choosing what the head is for, is the entire difficulty of training, and it is why nobody sets these by hand.

And in a real model there are many of these

One last honest gap. Everything above is one head. A real layer runs several side by side, each with its own $W_Q$, $W_K$, $W_V$, and each free to specialise: one tracking pronouns, one tracking the previous word, one tracking quotation marks.

The arithmetic barely changes. With $h$ heads, each gets $d_k = d/h$ so the total work stays the same, they all run at once, and their outputs are laid end to end back into numbers. GPT-2 small runs 12 heads of 64 per layer, and 12 layers of those.

The crane is finally a machine in one paragraph and a bird in the other, in the only sense that matters to a neural network: the two vectors are now different, and different because of the words around them.

What is still missing is any way to turn a vector back into a word. That is stage 6, and it is the shortest one: multiply by a matrix, softmax again, and read off the prediction.

Run it yourself

The code, numbered by stage

Every figure above came from these files. Nothing on this page is drawn by hand.

Needs Python 3 and numpy. Run them in order. 99_export.py writes the data this page reads, so if you change the text at stage 0, every picture here changes with it. A zip of the whole folder will be at the end of the article once the last stage lands.

To be continued

Five stages are done. The text became tokens, the tokens became numbers, the numbers became vectors, the vectors learned where they sat, and then they learned to read the words around them. One stage remains.

Stage 6, the next word. The vectors that came out of attention still are not words. Turning one back into a word takes a matrix multiplication to score every entry in the vocabulary, a softmax to make those scores a probability for each one, and then a choice about how to pick. Always the top word, and the model repeats itself for ever. Sample too freely and it wanders. That dial is called temperature, and it will have a knob you can turn.

Written in stages on purpose. Each one lands, gets read, gets argued with, and only then does the next one start. That is also why the wrong answers are still in the text rather than tidied out of it.

\(d\)	how many numbers one token has. for this model, 768 for GPT-2 small. Fixed for the whole model, and it is the only thing the formula needs to know about the model.
\(i\)	which circle. 0 to , since numbers taken two at a time give circles. \(2i\) and \(2i+1\) turn a circle number into its two column numbers: circle 0 owns columns 0 and 1, circle 3 owns columns 6 and 7.
\(\mathrm{pos}\)	which word. 0 to 63 here, counting along the text. The only letter that changes as you read the sentence.
\(\omega_i\)	circle i's turning rate, radians per position, straight from the table above. Note that \(\mathrm{pos}\) is not in its formula: a circle's rate never changes, only its angle does, and the angle is \(\mathrm{pos}\cdot\omega_i\). The formula is a compact way of saying "circle 0 turns one radian per position, and each circle after it turns \(\sqrt{10}\approx 3.16\) times more slowly". The 10000 has no deep meaning: it only sets the spread between the fastest circle and the slowest, and with \(d=16\) it works out to \(10000^{2/16}=\sqrt{10}\) per step.

Transformers explained, one stage at a time

The text, and why this text

Text becomes tokens

Tokens become numbers, and the numbers mean nothing

Numbers become vectors

Position, and why sines

Where we have got to

And then you notice the hole

So the plan writes itself

Three attempts, three rules, and a fourth that is obvious

A circle, in numbers

Why eight circles, when a token has sixteen numbers?

The first two words of the text, all the way through

If it is a circle, does it not come back to where it started?

And now the notation, which says the same thing

Back to rule 3, and why sines rather than any other bounded thing

Reading that picture: why is the right side all blue?

The two cranes come apart

How would anyone ever think of this?

If you want to think like that

Attention, and how a word finally reads the room

Where we have got to

And then you notice the hole

So the plan writes itself, again

What is left: let every word choose for itself

Question 1: how do you score two vectors against each other?

Question 2: so why not just score \(x_i \cdot x_j\) and be done?

Question 3: and why a third letter, V?

Question 4: scores are any size. How do they become a mixture?

Question 5: why divide by \(\sqrt{d_k}\) first?

Question 6: why is half the picture always blank?

All six answers, in one line

One token, all the way through

Every token, and what it looks at

The two cranes, at last

The hand set head, and what training is actually for

And in a real model there are many of these

The code, numbered by stage