Attend
The attention mechanism, from scratch.
A patient, step-by-step walk through the one idea behind every modern language model — from a single dot product all the way to the full Transformer block, with live math on every page.
Part I · Foundations
§1
§2
A vector, a dot product
Similarity is just an inner product. We'll build the intuition by hand on two tiny vectors.
From similarity to attention
Turn scores into weights with softmax. Feel how temperature bends the distribution.
Part II · Scaled dot-product attention
§3
§4
Why we scale by √dk
Watch variance explode as dk grows, and softmax collapse to one-hot. Then fix it.
Splitting Q, K, V
A worked example on six tokens. Every number computed live; hover to trace the flow.
Part III · Multi-head and beyond
§5
§6
§7
Per-head attention
Eight heads, each with its own view of the sentence. Compare patterns side-by-side.
Positional encodings
Sinusoids of many wavelengths. Drag through dimensions to see the pattern emerge.
The full Transformer block
Attention + residual + layer-norm + FFN. Click each wire to see what flows through.