Paper Comparisons and Analysis¶

Detailed side-by-side comparisons of related papers to understand trade-offs and evolution.

Table of Contents¶

1. Architecture Comparisons
2. Training Approaches
3. Alignment Methods
4. Reasoning Methods
5. Image Generation
6. Efficiency Techniques
7. Inference and Serving
8. Retrieval and Knowledge
9. Agents and Tool Use
10. Evaluation
11. Beyond Language
12. Evolution Over Time
13. Technique Combinations
14. When to Use Which Paper's Techniques

Architecture Comparisons¶

Transformer Variants: Encoder vs Decoder vs Encoder-Decoder¶

When to use: - BERT: Classification, entity recognition, semantic search - GPT: Text generation, chatbots, creative writing - T5: Translation, summarization, structured generation

Seq2Seq Progression: RNN to Attention to Transformer¶

The encoder-decoder idea predates the Transformer by several years. Understanding the lineage clarifies why each component was invented.

The key insight chain: 1. Seq2Seq: "Compress input to a vector, then decode" - works but bottleneck hurts long sentences 2. Bahdanau attention: "Let the decoder look back at every encoder state, weighted by relevance" - solved the bottleneck 3. Transformer: "What if attention is the whole model?" - removed RNNs entirely, enabling parallelism

Vision: CNN vs Transformer¶

Key Insight: ViT needs more data but scales better. For production with limited data, CNNs still competitive. For multimodal or large-scale, ViT wins.

Training Approaches¶

Pre-training Paradigms¶

GPT Scaling Progression¶

The GPT-2 moment: OpenAI staged the release because they feared misuse - the first time a language model was considered too capable to release freely. GPT-2 was the proof that scale alone enables emergent zero-shot behavior.

Scaling: GPT-3 vs LLaMA¶

What Changed: Scaling laws (paper #12) showed GPT-3 was undertrained. LLaMA applied this learning to train smaller models longer, proving 13B with proper training matches 175B.

Practical Impact: - GPT-3 era: "Bigger is better" - LLaMA era: "Better trained is better" - Democratized LLMs for researchers and startups

Compute Efficiency: Original vs Chinchilla-Optimal¶

Models following Chinchilla: - ✅ LLaMA - 1.4T tokens for 65B params - ✅ Chinchilla - 1.4T tokens for 70B params - ❌ GPT-3 - 300B tokens for 175B params (undertrained) - ❌ PaLM - 780B tokens for 540B params (undertrained)

Alignment Methods¶

RLHF vs Constitutional AI vs DPO vs KTO vs GRPO vs RLVR¶

PPO's role: PPO (Schulman et al. 2017) is the RL backbone of classic RLHF. It clips policy updates to prevent catastrophic reward hacking - the "proximal" constraint keeps the fine-tuned model close to the base model. DPO and GRPO emerged partly to sidestep PPO's complexity and hyperparameter sensitivity.

Stage-by-Stage (RLHF vs Constitutional AI):

Hybrid Approach (Best Practice): - Use RLHF for helpfulness (harder to specify as principles) - Use Constitutional AI for harmlessness (easier to write rules) - Combine both signals in final model

Real-World: - ChatGPT: Primarily RLHF - Claude: Constitutional AI + some RLHF - Trend: Moving toward more Constitutional AI for scalability

Reasoning Methods¶

Prompt-time vs Train-time Reasoning¶

The field moved from asking a model to reason to training it to. These are the stops along the way.

How to choose

• You have an API and a problem today → Chain-of-Thought, then Self-Consistency if accuracy matters more than cost. This is the entire budget-conscious answer.
• Answers are checkable and you can train → RLVR with GRPO. This is what produced DeepSeek-R1.
• Answers are checkable but you cannot afford RL → STaR: sample, filter by correctness, fine-tune, repeat.
• The failure mode is a wrong intermediate step → Process Reward Models. Outcome supervision rewards lucky guesses; process supervision does not.
• The problem needs exploration, not a single chain → Tree of Thoughts, or rStar-Math for the MCTS version.

The cost trap: Self-Consistency at N=40 costs 40x a single answer. A reasoning model trained with RLVR gets similar or better accuracy in one (longer) generation. Above a certain query volume, training is cheaper than sampling.

Outcome vs Process Supervision¶

Image Generation¶

VAE vs GAN vs Diffusion vs Latent Diffusion¶

The generative model lineage: 1. VAE (2013): First principled probabilistic generative model - structured latent space but blurry outputs due to pixel-wise reconstruction loss 2. GANs (2014): Sharp images via adversarial training, but mode collapse and instability 3. DDPM (2020): Beat both on quality and diversity, sacrificed speed 4. Stable Diffusion (2022): Runs diffusion in VAE's latent space - borrows VAE's compression to make diffusion tractable

VAE's lasting contribution: Even though VAEs lost the image quality race to GANs and diffusion, the VAE encoder/decoder is the latent space backbone of Stable Diffusion.

Use Cases: - VAE: Structured generation, interpolation, anomaly detection, latent representations - GANs: Real-time applications, style transfer (when speed matters) - DDPM: Research, highest quality needs - Stable Diffusion: Production text-to-image, balance of quality and speed

Image Generation: Computational Cost Comparison¶

Speedup techniques: - DDIM sampling: 50× faster than DDPM - Latent space: 8-10× compression - Distillation: Train student to match in fewer steps

Diffusion Samplers: DDPM vs DDIM vs Flow Matching¶

The generator quality is set by training; the speed is set by how you sample from it.

Practical read: if you use diffusion, you are almost certainly using a DDIM-style sampler on a latent model. DDPM is the theory; DDIM is what made it usable; flow matching is what current frontier image models train with.

Conditioning and Control¶

Four different problems that get confused with one another.

These compose. A typical production stack runs a latent model with CFG, a subject LoRA, and a ControlNet for pose - all at once. They are answering different questions.

Image Tokenizers: VAE vs VQ-VAE vs VQ-GAN¶

The through-line: compress first, model second. Both branches - continuous latents for diffusion, discrete tokens for autoregressive models - come from the same insight, and both trace back to the VAE.

Efficiency Techniques¶

Mixture of Experts: Dense vs Sparse Routing¶

The MoE trade-off: Sparse MoE gives near-dense-model quality at a fraction of the FLOPs per token, but at the cost of much larger total parameter counts, complex routing, and communication overhead across devices.

Top-1 vs top-2 routing: - Switch (top-1): Simpler, lower compute, but each token sees only one expert - higher variance - Mixtral (top-2): Each token mixes two experts - more stable, better quality, slightly more compute

Fine-Tuning Methods¶

LoRA in Detail (7B model example):

Full fine-tuning: 7B trainable params, ~28GB memory
LoRA (r=8):       ~4M trainable params, ~8GB memory
Reduction:        1,750× fewer params, 3.5× less memory

When to use LoRA: - ✅ Limited GPU memory - ✅ Need to fine-tune multiple times - ✅ Want to deploy multiple adaptations - ❌ Unlimited resources and want absolute best - ❌ Catastrophic domain shift (full fine-tuning better)

Knowledge Enhancement: RAG vs Fine-tuning vs Prompting¶

Decision Matrix:

Combination (Best):

Base Model
    ↓
Fine-tune (domain language/style)
    ↓
RAG (dynamic facts)
    ↓
Prompting (task-specific instructions)

Inference and Serving¶

Where the Money Goes¶

Four techniques, four different bottlenecks. They stack.

Order to reach for them: if you serve someone else's model, start with vLLM (PagedAttention) and quantisation - no retraining, largest wins. Speculative decoding next if latency is the complaint. GQA and MoE are decisions made before training, not fixes afterwards.

Attention Variants¶

Positional Encoding¶

Retrieval and Knowledge¶

Sparse vs Dense vs Late-Interaction vs Graph¶

The practical answer is hybrid. Dense retrieval alone fails on product codes, error strings and proper nouns; BM25 alone fails on paraphrase. Most production systems run both and fuse the rankings. Reach for GraphRAG only when the questions are genuinely global - "what themes run through these 10,000 documents?" - because the indexing pass costs real money.

Grounding Strategies¶

The rule that saves the most money: RAG for facts, fine-tuning for behaviour. Fine-tuning to inject knowledge is the single most common expensive mistake - it is unreliable, and the facts go stale the moment the model is trained.

Agents and Tool Use¶

Toolformer is the odd one out: it trains the model to call APIs rather than prompting it to. MCP is orthogonal to all of them - it standardises how the tools are exposed, not how the agent thinks.

Evaluation¶

Read Emergent Abilities alongside these. Its pairing with the "Mirage" rebuttal is the clearest lesson in the collection that a metric choice - exact-match versus partial credit - can manufacture a discontinuity that isn't there.

Beyond Language¶

Protein Structure¶

Self-Play and World Models¶

Evolution Over Time¶

Word Embeddings: Static vs Contextual¶

The key shift: Word2Vec proved that dense vector representations capture semantic relationships (king - man + woman = queen). BERT proved those representations should be contextual - the same word needs a different embedding depending on its sentence. This shift from static to contextual embeddings is the foundation of modern NLP.

Language Model Performance (on Common Benchmarks)¶

Trends: - 2018-2020: Scale up parameters - 2020-2022: Alignment via RLHF - 2022-2023: Compute-optimal training - 2023+: Multimodal, reasoning, efficiency

Parameter Efficiency Over Time¶

What this means: - 2020: "Need 175B params for GPT-3 performance" - 2023: "Need only 13B params with better training" - 13× parameter reduction in 3 years through better training

Image Generation Quality Over Time¶

FID Score: Lower is better (measures distribution similarity)

Transformers Beyond Language¶

The Transformer architecture generalized far beyond NLP. These applications use the same attention mechanism but on fundamentally different sequence types.

AlphaFold 2's significance in this context: It demonstrated that the core insight of attention - letting every element attend to every other element - applies to non-linguistic structure prediction. Amino acid residues attending to each other to infer 3D spatial relationships is conceptually identical to tokens attending to each other to infer semantic relationships. AlphaFold 2 effectively closed the protein structure prediction problem (GDT > 90 on CASP14), a benchmark that had resisted 50 years of computational biology.

Technique Combinations¶

What Works Well Together¶

Production Stack Comparison¶

Scenario: Enterprise Chatbot

Cost Comparison (1M tokens): - GPT-4 API: $60 (generation) - Claude API: $24 (generation) - Self-hosted LLaMA-13B: ~$2 (compute only) - BERT: <$1 (compute only)

Research Impact Comparison¶

Influence by Descendants¶

Citation counts go stale the month you write them down, so this ranks by something more durable: how much of the current stack descends from the paper.

Sleeper hits - unglamorous papers doing enormous work in production: RoPE, GQA, FlashAttention and PagedAttention. None changed what models can do; all four changed what they cost.

When to Use Which Paper's Techniques¶

Quick Decision Tree¶

Need to generate text? - Long-form, creative → GPT-3 / LLaMA - Factual, grounded → RAG - With reasoning → Chain-of-Thought - Specific style → Fine-tuning + LoRA - Translation or structured output → T5 (encoder-decoder)

Need to understand text? - Classification → BERT - Semantic search → BERT / CLIP (for images) - Q&A → BERT + RAG - Simple/fast embeddings (no GPU) → Word2Vec / GloVe

Need to generate images? - Artistic, text-to-image → Stable Diffusion, or SD3 / Flux (flow matching) - Faster sampling from a trained model → DDIM - Prompt adherence → Classifier-Free Guidance (the CFG slider) - Control the composition → ControlNet - A specific subject → DreamBooth, or a LoRA - Video → Sora / DiT - Fast, real-time single pass → GANs - Structured latent space / interpolation → VAE

Need better reasoning? - Cheapest improvement → Chain-of-Thought - More accuracy, more budget → Self-Consistency - Needs exploration and backtracking → Tree of Thoughts - You can verify answers and train → RLVR + GRPO - Bad intermediate steps → Process Reward Models

Need to align a model? - General helpfulness → RLHF / PPO (InstructGPT) - Safety focus → Constitutional AI, Llama Guard for moderation - Efficient preference learning → DPO - Only thumbs-up/down data → KTO - Maths / verifiable reasoning → GRPO + RLVR

Need to adapt a model? - Full resources → Fine-tuning - Limited resources → LoRA, or QLoRA on one GPU - New facts, not new behaviour → RAG (do not fine-tune for this) - No training at all → Prompting

Need it cheaper or faster in production? - Throughput → vLLM / PagedAttention - Latency → Speculative Decoding - Won't fit on the GPU → GPTQ / AWQ 4-bit - KV cache too large → GQA - Capacity without inference cost → Mixture of Experts

Need to retrieve over your own data? - General semantic search → Dense Retrieval, hybrid with BM25 - Precision-critical → ColBERT late interaction - Corpus-level "what are the themes" questions → GraphRAG

Building an agent? - Core loop → ReAct - Learn from failures → Reflexion - Long-lived, accumulating skills → Voyager - Wiring tools to models → MCP - Measuring it → SWE-bench

Planning a project? - Estimate resources → Scaling Laws - Choose model size and token budget → Chinchilla - Decide train vs. think-longer → Test-Time Compute

Applying this outside NLP? - Protein structure → AlphaFold 2, ESMFold if speed matters - Molecular complexes and drug binding → AlphaFold 3 - Images → ViT, MAE for self-supervised pretraining - Code → Codex, and SWE-bench to evaluate - Games and control → AlphaZero, DreamerV3 - Interactive worlds → Genie

Key Insights from Comparisons¶

1. Scaling isn't everything - LLaMA proved training matters more than size
2. Hybrid is best - Combine RLHF + Constitutional AI, RAG + fine-tuning
3. Efficiency advances - LoRA makes fine-tuning accessible, Stable Diffusion makes diffusion practical, MoE makes scale affordable
4. Open vs closed - Open models (LLaMA) spawned more innovation than closed (GPT-3)
5. Architecture consolidation - Transformers won for text, vision, protein structure, and more
6. Alignment evolution - From RLHF/PPO to DPO to GRPO (each iteration simpler or more targeted)
7. Knowledge grounding - RAG reduces hallucination better than any architecture change
8. Embeddings matured - Static (Word2Vec) to contextual (ELMo) to Transformer-based (BERT) over ~5 years
9. Seq2Seq lineage - Every encoder-decoder model (T5, Stable Diffusion decoder, etc.) inherits from Sutskever 2014 via Bahdanau attention
10. Compute moved from training to inference - Test-Time Compute and o1 showed a small model that thinks longer can beat a much larger one that answers immediately
11. Alignment kept shedding machinery - PPO needed a reward model and a critic; DPO dropped the reward model, GRPO dropped the critic, KTO dropped paired data, RLVR dropped learned rewards entirely
12. Sparsity beat density - the dense scaling line topped out at PaLM's 540B; every frontier model since activates a fraction of its parameters
13. Compress first, then model - the VAE insight recurs everywhere: latent diffusion, VQ-VAE tokens, VQ-GAN, and video as spacetime patches
14. The cheap wins are in serving - RoPE, GQA, FlashAttention and PagedAttention changed no capability and changed every budget
15. Evaluation is the weak link - static benchmarks leak, LLM judges are biased, and metric choice alone can invent an "emergent" jump

Last updated: 2026-08-20 · Covers all 107 papers in the collection.