Attention Is All You Need, in plain English

In June 2017, eight researchers working at Google published a paper called "Attention Is All You Need," which introduced a new design for AI language models called the Transformer. Almost every AI tool you use today, from ChatGPT to coding assistants to translation apps, is built on that design.

The paper itself is about translating English into German and French. Its lasting impact came from showing a way to train language models much faster, which made today's very large models possible.

Paper: Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, "Attention Is All You Need," NeurIPS 2017 (arXiv:1706.03762).

The one-line claim

The authors propose "a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely" (abstract). In plainer terms, they threw out the older machinery that read text one word at a time and replaced it with a method that looks at the whole sentence at once.

The problem it solved

Before 2017, the best translation systems used recurrent neural networks. These read a sentence word by word, carrying a running summary forward. That made them slow to train, because each step had to wait for the one before it. It also made it hard for them to connect words that sit far apart, such as a pronoun and the noun it refers to.

Researchers had already added a technique called attention to these systems. Attention lets the model look back at every input word and decide which ones matter most for the word it is producing. The Transformer's step was to make attention the whole engine, with no word-by-word reading at all.

How the Transformer works

A few ideas carry most of the design:

• Self-attention: Every word in a sentence scores its relevance to every other word, and those scores decide how much each word influences the others. This happens for all words at the same time.
• Multi-head attention: The model runs eight attention calculations in parallel, each free to pick up a different kind of relationship, such as grammar in one and meaning in another.
• Positional encoding: Because the model sees all words at once, it needs another way to know word order. The authors add a pattern of sine and cosine waves to each word's representation to mark its position.
• Stacked layers: The encoder, which reads the input, and the decoder, which writes the output, each have six identical layers of attention followed by a small feed-forward network.

The biggest practical gain was speed. Since the model processes all words together, the work spreads across many graphics processors at once instead of waiting in a queue.

The results

The paper reports these scores on standard 2014 translation tests, measured in BLEU, a common score for translation quality where higher is better (arXiv:1706.03762):

The English-to-German score beat the best earlier results, including combinations of several models, by more than 2 points. The big model trained in 3.5 days on eight NVIDIA P100 graphics cards, a fraction of the computing cost of earlier systems. A smaller base version trained in 12 hours and still beat every published model on English to German.

One detail for careful readers: the abstract gives 41.8 for English to French, while the results section of the same paper reports 41.0. The current arXiv version contains both figures.

The authors also tested the Transformer on a different task, analysing sentence grammar, and found it did well with little tuning.

What came after

The paper's own ending hints at what followed. The authors planned to apply attention to "input and output modalities other than text," such as images, audio and video. Within a few years, Transformer-based models included Google's BERT for search, OpenAI's GPT series, and image and video generators. Today's chat assistants and coding tools are much larger descendants of the decoder half of this design.

Who should care

If you are building or buying AI tools in the Caribbean, this paper explains why they behave the way they do. The cost of running a Transformer rises with the length of the text it handles, which is why AI tools limit how much text you can paste in and why longer documents cost more to process.

It also explains where local advantage lies. Everyone has access to the same basic design. What sets a Caribbean product apart is its data, such as local language, prices and regulations, and how well it fits the way people here actually work.

Limits

• The paper is about translation. Chat, coding and image generation came later, through much more data and computing power.
• Attention compares every word with every other word, so the cost grows with the square of the text length. Later work such as FlashAttention attacks that cost.
• The paper does not cover later techniques such as instruction tuning, retrieval or AI agents.

What to read next

For how later models scaled this design cheaply, see our digest on mixture of experts. For how chat models are taught to follow instructions, read our preference alignment digest. The original paper is short, and its introduction and background sections are readable without a maths background.