Attention Is All You Need¶

Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin

Published: June 2017 (NeurIPS 2017)

Paper Link: https://arxiv.org/abs/1706.03762

Why This Paper Matters¶

This paper introduced the Transformer architecture, which became the foundation for virtually every major AI breakthrough since 2017. Before Transformers, language models relied on recurrent neural networks (RNNs) and LSTMs, which processed text sequentially and struggled with long-range dependencies. The Transformer changed everything by replacing recurrence with self-attention, enabling parallel processing and better understanding of context.

The Core Innovation: Self-Attention¶

The Problem with Previous Approaches¶

RNNs/LSTMs had to process words one at a time (sequential)
This made training slow and limited their ability to connect distant words
Example: In "The cat, which was sitting on the mat, meowed," connecting "cat" to "meowed" was challenging

The Transformer Solution¶

The Transformer processes all words simultaneously using self-attention mechanisms that compute relationships between every word and every other word in parallel.

Key Components Explained¶

1. Self-Attention Mechanism¶

What it does: Allows each word to "look at" all other words and determine which ones are most relevant.

How it works: - Each word gets three vectors: Query (Q), Key (K), and Value (V) - Think of it like a search engine: - Query: What am I looking for? - Key: What do I contain? - Value: What information do I provide? - The attention score = how well each word's Key matches your Query - Output = weighted sum of Values based on attention scores

Simple Example:

Sentence: "The cat sat on the mat"
When processing "sat":
- High attention to "cat" (who is sitting?)
- High attention to "mat" (where are they sitting?)
- Low attention to "the" (less important)

2. Multi-Head Attention¶

Instead of one attention mechanism, use multiple "heads" in parallel: - Each head can learn different types of relationships - Head 1 might focus on grammatical relationships - Head 2 might focus on semantic meaning - Head 3 might focus on positional relationships - Outputs are concatenated and combined

Why multiple heads? Different relationships matter in different contexts.

3. Positional Encoding¶

The Problem: Self-attention has no inherent sense of word order. "Dog bites man" vs "Man bites dog" would look identical without position info.

The Solution: Add positional encodings to word embeddings using sine/cosine functions:

PE(pos, 2i) = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

This gives the model information about where each word appears in the sequence.

4. Feed-Forward Networks¶

After attention, each position passes through a simple neural network:

FFN(x) = max(0, xW₁ + b₁)W₂ + b₂

Applied identically to each position independently.

5. Layer Normalization & Residual Connections¶

Residual connections: Add input to output (x + Attention(x))
Layer normalization: Stabilizes training
Helps gradient flow and enables training deep networks

Architecture Overview¶

Input Sequence
    ↓
Word Embeddings + Positional Encoding
    ↓
┌─────────────────────────────────────┐
│  ENCODER (6 layers)                 │
│  Each layer:                        │
│  - Multi-Head Self-Attention        │
│  - Add & Norm                       │
│  - Feed-Forward Network             │
│  - Add & Norm                       │
└─────────────────────────────────────┘
    ↓
Encoded Representation
    ↓
┌─────────────────────────────────────┐
│  DECODER (6 layers)                 │
│  Each layer:                        │
│  - Masked Multi-Head Self-Attention │
│  - Add & Norm                       │
│  - Multi-Head Cross-Attention       │
│    (attends to encoder output)      │
│  - Add & Norm                       │
│  - Feed-Forward Network             │
│  - Add & Norm                       │
└─────────────────────────────────────┘
    ↓
Output Probabilities

Key Results¶

Machine Translation Performance: - English-to-German: 28.4 BLEU (new state-of-the-art) - English-to-French: 41.8 BLEU (new state-of-the-art)

Training Efficiency: - Trained in 3.5 days on 8 GPUs - Previous best models took weeks

Cost: $12/hour on cloud compute (affordable at scale)

Why This Was Revolutionary¶

1. Parallelization¶

RNNs: Process words sequentially (slow)
Transformers: Process all words simultaneously (fast)

2. Better Long-Range Dependencies¶

Direct connections between any two words
No information loss over long distances

3. Scalability¶

Architecture scales beautifully with more data and compute
Led to GPT, BERT, and all modern LLMs

4. Transferability¶

Originally designed for translation
Now used for: text generation, image recognition, protein folding, music generation, and more

Real-World Impact¶

Direct Descendants:¶

BERT (Google, 2018): Encoder-only for understanding
GPT (OpenAI, 2018-present): Decoder-only for generation
T5 (Google, 2019): Unified text-to-text framework
Vision Transformer (2020): Applied to images
AlphaFold (DeepMind, 2021): Protein structure prediction

Applications Today:¶

ChatGPT, Claude, Gemini (conversational AI)
Google Translate (translation)
GitHub Copilot (code generation)
DALL-E, Midjourney (image generation using adapted Transformers)
Drug discovery, scientific research

Key Takeaways for Practitioners¶

Self-attention is the killer feature: Learn different types of relationships in data
Positional encoding matters: Order information must be explicitly added
Multi-head attention adds expressiveness: Different heads learn different patterns
Residual connections enable depth: Can stack many layers effectively
Parallelization = speed: Major advantage over sequential models

Limitations & Future Directions¶

Limitations:¶

Quadratic complexity: Attention is O(n²) in sequence length
Memory intensive: Storing attention weights for long sequences
Not naturally suited for very long sequences (>1000 tokens was challenging)

Solutions Developed Later:¶

Sparse attention (Longformer, BigBird)
Linear attention approximations (Performers, Linear Transformers)
Efficient Transformers (FlashAttention)
Alternative architectures (Mamba, RWKV)

Citation¶

@article{vaswani2017attention,
  title={Attention is all you need},
  author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, {\L}ukasz and Polosukhin, Illia},
  journal={Advances in neural information processing systems},
  volume={30},
  year={2017}
}