Attention & Transformers: The Idea Behind Every Modern AI Model

The last lesson left off with attention already invented — in 2014, inside an RNN, to fix one specific bottleneck. This lesson covers what happened three years later: attention, kept; the RNN carrying information step-by-step, removed entirely. What's left running on attention alone is why every serious model released since — ChatGPT, Claude, Gemini, all of them — is built the way it is, first described in a 2017 paper memorably titled "Attention Is All You Need." That idea is attention, and once you see what it's actually doing, a lot of what these models can do stops feeling mysterious.

The problem it solves

Consider the sentence: "The trophy didn't fit in the suitcase because it was too big." What does "it" refer to — the trophy, or the suitcase? You resolved that instantly, using context from elsewhere in the sentence. Now change one word: "...because it was too small" — and the answer flips, even though "it" is grammatically identical in both cases.

For a model to handle language well, it needs to do the same thing at every single word: look around at the rest of the sentence — sometimes the rest of a whole document — and figure out which other words actually matter for understanding this one. That's the problem attention was built to solve.

What attention actually does

For every word, attention computes a relevance score against every other word, then uses those scores to blend information from across the sentence into that word's representation. Words that matter a lot get weighted heavily; words that don't barely contribute at all. Try it on the ambiguous sentence below.

Do this for every word, and the model ends up with a representation of the sentence where each word "knows" which other words are relevant to it — without anyone having hand-coded a single grammar rule. The pattern was learned entirely from being shown enormous amounts of text and adjusting until its predictions got better.

The math, precisely

Everything above is the honest intuition, but it's worth seeing the actual mechanism — it's five lines of linear algebra, not a black box. Every word's embedding gets turned into three vectors via three learned weight matrices:

• A query vector Q— what this word is "looking for."
• A key vector K— what this word "offers" when compared against.
• A value vector V— the actual content this word contributes once it's attended to.

The full operation, from the original paper, is one formula:

Read left to right, that's four steps:

1. Q Kᵀ — every query dotted against every key, giving a raw similarity score between every pair of words. A large dot product means "these two point in a similar direction" — the model's learned notion of relevance.
2. / √d_k — divide by the square root of the key dimension. Without this, scores grow large as vectors get longer, which pushes softmax into a region where it outputs almost all 0s and one 1 — the gradient flattens out and the model stops learning from those weights. This one division keeps training stable.
3. softmax(...) — turns each word's row of raw scores into a proper probability distribution: all positive, summing to exactly 1. These are the attention weights you saw highlighted in the demo above.
4. ... V — use those weights to take a weighted average of every word's value vector. That weighted average is the new, context-aware representation of the word.

Why this replaced older approaches

Before attention, the standard approach processed text one word at a time, in order, carrying forward a summary of everything seen so far. Two problems followed directly from that design: information from early in a long passage tended to fade by the time the model reached the end, and — because each step depended on the previous one — the whole thing had to run sequentially, word by word, which made it slow to train on the huge datasets modern models need.

Attention looks at the whole sequence at once. Every word can connect directly to every other word, regardless of distance, and those connections can all be computed simultaneously rather than one after another. That combination — better handling of long-range context, and a computation that parallelizes well — is a large part of why this became the default architecture almost everywhere.

The transformer, briefly

A transformer is the full architecture built around this idea: layers of attention stacked on top of each other, interleaved with simpler processing steps, repeated dozens of times. Each layer refines the representation a little further — early layers tend to pick up on local patterns like grammar, later layers on more abstract relationships like meaning and intent. Stack enough of these layers, train on enough text, and you get a system that can hold a coherent conversation, write working code, and explain its own reasoning — all built from the same repeated operation: for every word, decide what else matters, and by how much.