Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG)¶

Authors: Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. (Facebook AI Research, UCL, NYU) Published: May 2020 (NeurIPS 2020) Paper: arxiv.org/abs/2005.11401

Why This Matters¶

RAG solved a fundamental problem with language models: they can't access updated information or cite sources. By combining neural retrieval with generation, RAG:

Grounds responses in facts: Retrieves relevant documents before generating
Updates knowledge without retraining: Just update the document database
Enables citations: Can point to source documents
Reduces hallucinations: Generates from retrieved evidence

Real-world impact: - Powers ChatGPT plugins, Bing Chat, Perplexity AI - Enables enterprise AI with private knowledge bases - Standard pattern for production LLM applications - Foundation for question answering, summarization, fact-checking

RAG made LLMs practical for knowledge-intensive real-world applications.

The Problem¶

Limitations of pure language models (2020):

Static knowledge
Knowledge frozen at training time
Can't access new information
Requires retraining to update
Hallucination
Generate plausible but false information
No grounding in verifiable sources
Confident-sounding mistakes
No attribution
Can't cite sources
Difficult to verify claims
Low trust in critical applications
Limited memory
All knowledge must fit in parameters
Struggles with rare facts
Poor at long-tail knowledge

The question: Can we give language models access to external knowledge dynamically?

Core Innovation¶

Retrieval + Generation = RAG¶

The key insight: Treat generation as a two-stage process:

Retrieve: Find relevant documents from a large corpus
Generate: Condition language model on retrieved documents

Architecture:

Input Question
     ↓
Retrieve (top-k documents from knowledge base)
     ↓
Concatenate: [question + retrieved docs]
     ↓
Generate (using seq2seq model)
     ↓
Output Answer

Two RAG Variants¶

RAG-Sequence: - Retrieve once for entire sequence - Use same documents for all tokens - Simpler, faster

RAG-Token: - Can retrieve different docs for each token - Marginalize over retrieved docs per token - More flexible, slower

Architecture Details¶

Components¶

1. Retriever: Dense Passage Retrieval (DPR) - Bi-encoder architecture - Query encoder: BERT_Q - Document encoder: BERT_D

Retrieval process:

# Encode query
q = BERT_Q(question)

# Encode all documents (pre-computed)
d_i = BERT_D(doc_i) for all docs

# Find top-k by dot product similarity
scores = q · d_i
top_k = argsort(scores)[-k:]

2. Generator: BART - Pre-trained seq2seq model (BART-large) - 400M parameters - Fine-tuned on task with retrieved context

3. Knowledge Source - Wikipedia (21M passages of 100 words) - Pre-indexed with document embeddings - Can swap for domain-specific corpus

Training Approach¶

End-to-end training: - Jointly train retriever and generator - Retriever learns what to retrieve for generation - Generator learns to use retrieved content

Loss function:

L = -log P(y | x, z)

Where:
  x = input question
  y = target answer
  z = retrieved documents

Key trick: Marginalize over top-k retrieved documents

P(y | x) = Σ_z P(z | x) P(y | x, z)

P(z | x) = softmax(query · doc)
P(y | x, z) = BART(y | x, z)

Gradient flow: - Generator loss backprops through retriever - Retriever learns to fetch documents that help generation - Uses differentiable top-k approximation

How It Works (Step-by-Step)¶

Example: Question Answering¶

Input: "When was the first iPhone released?"

Step 1: Encode query

q_embed = BERT_Q("When was the first iPhone released?")

Step 2: Retrieve top-k passages

# Pre-computed document embeddings
doc_scores = q_embed · all_doc_embeddings

# Get top-5
top_docs = [
  "The iPhone was announced by Steve Jobs on January 9, 2007...",
  "Apple Inc. released the first generation iPhone on June 29, 2007...",
  "The original iPhone was marketed as combining three products...",
  ...
]

Step 3: Generate answer

# RAG-Sequence: Use all docs together
input = "Question: When was the first iPhone released?\n\n" +
        "Context: " + join(top_docs) + "\n\n" +
        "Answer:"

output = BART.generate(input)
# "June 29, 2007"

Step 4: (Optional) Return sources

# Can cite which documents were used
sources = [doc_urls[i] for i in top_k_indices]

Experimental Results¶

Tasks Tested¶

Open-domain Question Answering: - Natural Questions - TriviaQA - WebQuestions - CuratedTREC

Performance (Exact Match accuracy):

Model	Natural Questions	TriviaQA	WebQuestions
BART (no retrieval)	24.0%	35.0%	28.0%
DPR + BART (pipeline)	41.5%	56.8%	42.4%
RAG-Token	44.5%	56.1%	45.2%
RAG-Sequence	44.1%	56.8%	45.5%

Key finding: RAG beats pipeline approach (retrieve then generate separately).

Fact Verification (FEVER)¶

Model	Accuracy
BERT baseline	75.6%
RAG-Sequence	78.1%

Jeopardy Question Generation¶

Human evaluation: - RAG generates more factual questions - Better at incorporating specific knowledge - Fewer hallucinations

Key Advantages¶

1. Dynamic Knowledge¶

Update knowledge base without retraining
Access to latest information
Add domain-specific documents easily

Example use case: - Corporate knowledge base with internal docs - Medical database with latest research - Legal system with current statutes

2. Reduced Hallucination¶

Grounded in retrieved evidence
Can't generate facts not in corpus
Explicit provenance

Comparison: - Pure LM: "The Eiffel Tower is 450 meters tall" (hallucinated) - RAG: "According to [doc], the Eiffel Tower is 330 meters tall"

3. Interpretability¶

Can inspect retrieved documents
Understand why model gave answer
Debug by examining retrieval quality

4. Efficiency¶

Don't need to store all knowledge in parameters
Smaller models can access vast knowledge
Modular: Improve retrieval or generation independently

5. Verifiability¶

Cite sources for fact-checking
Transparent reasoning process
Build trust in high-stakes domains

Limitations¶

1. Retrieval Quality Bottleneck¶

If retrieval fails, generation fails
Out-of-domain queries struggle
Need high-quality, comprehensive corpus

Example failure: - Question about very recent event - Not in knowledge base → can't answer

2. Latency¶

Two-stage process is slower
Retrieval adds overhead
Not suitable for real-time applications without optimization

3. Context Length Limits¶

Can only include top-k docs (e.g., 5)
May miss relevant information in doc #6
Generator has limited context window

4. Training Complexity¶

End-to-end training is challenging
Requires large corpus and compute
Balancing retrieval and generation objectives

5. Contradictory Sources¶

Retrieved docs may conflict
Model must resolve contradictions
No built-in source reliability weighting

Practical Applications¶

Production RAG Systems¶

1. Customer Support Chatbots

# Retrieve from company knowledge base
docs = retrieve(user_question, kb="company_docs")
answer = generate(user_question, context=docs)

2. Research Assistants - Perplexity AI: Search + RAG for citations - Bing Chat: Web search + GPT-4 - Elicit: Academic paper search + summarization

3. Enterprise AI - Internal document search and QA - Compliance checking against regulations - Technical support with manuals

4. Educational Tools - Textbook-grounded tutoring - Homework help with citations - Exam preparation with verified facts

5. Content Creation - Blog writing with source material - Report generation from data - News summarization with links

Implementation Guide¶

Basic RAG Pipeline¶

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration

# Load RAG model
tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
retriever = RagRetriever.from_pretrained(
    "facebook/rag-token-nq",
    index_name="exact",
    use_dummy_dataset=True  # Use real index for production
)
model = RagTokenForGeneration.from_pretrained(
    "facebook/rag-token-nq",
    retriever=retriever
)

# Question answering
question = "When was the first iPhone released?"
inputs = tokenizer(question, return_tensors="pt")

# Generate (retrieval happens inside)
generated = model.generate(**inputs)
answer = tokenizer.decode(generated[0], skip_special_tokens=True)

print(answer)  # "June 29, 2007"

# Access retrieved documents
retrieved_docs = retriever(inputs["input_ids"], n_docs=5)
for doc in retrieved_docs:
    print(doc["title"], doc["text"][:100])

Custom RAG with LangChain¶

from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

# Build knowledge base
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(documents, embeddings)

# Create RAG chain
qa_chain = RetrievalQA.from_chain_type(
    llm=OpenAI(),
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    return_source_documents=True
)

# Query
result = qa_chain({"query": "When was the first iPhone released?"})
print(result["result"])
print(result["source_documents"])

Key Hyperparameters¶

Retrieval: - k: Number of documents to retrieve (typically 5-10) - similarity_metric: dot product, cosine, L2 - index_type: exact, HNSW, IVF (for large scale)

Generation: - max_length: Maximum generation length - num_beams: Beam search width - temperature: Sampling randomness

Training: - batch_size: 32-128 (depends on GPU memory) - learning_rate: 1e-5 for fine-tuning - n_docs: Number of docs to marginalize over

Evolution and Improvements¶

Subsequent Work¶

1. Fusion-in-Decoder (FiD) (2020) - Process each retrieved doc independently - Fuse in decoder layer - Better parallelization

2. RETRO (2021, DeepMind) - Retrieval-enhanced Transformer - Chunk-based retrieval during training - Scales to 7B parameters

3. Atlas (2022) - Few-shot RAG - Joint pre-training of retriever and LM - State-of-the-art on many tasks

4. Self-RAG (2023) - Model learns when to retrieve - Retrieval as needed, not always - Self-reflection on retrieved content

5. RALM (2023) - Reinforcement learning for retrieval - Optimizes for end task performance - Better retrieval strategies

Modern RAG (2024-2025)¶

Current Best Practices¶

1. Hybrid Search - Combine dense (embedding) and sparse (BM25) retrieval - Better coverage of different query types

2. Reranking - Two-stage retrieval: fast retrieval → slow reranking - Use cross-encoder for reranking top-k

3. Chunking Strategies - Semantic chunking vs fixed-size - Overlapping chunks for context - Metadata filtering

4. Query Rewriting - Expand or clarify user query before retrieval - Hypothetical document embeddings (HyDE)

5. Multi-hop Reasoning - Iterative retrieval for complex questions - Chain retrieved docs for reasoning

RAG vs Fine-tuning vs Prompt Engineering¶

Approach	Best For	Pros	Cons
RAG	Dynamic knowledge, citations	Updatable, grounded	Latency, retrieval dependency
Fine-tuning	Task specialization, style	Fast inference, internalized knowledge	Static, expensive to update
Prompting	Quick prototypes, few-shot	No training, flexible	Token limits, no new knowledge

Combined approach: - Fine-tune base model for domain/task - Add RAG for dynamic facts - Use prompting for instructions

Key Takeaways¶

RAG combines retrieval and generation for knowledge-grounded text
End-to-end training teaches retriever what's useful for generation
Reduces hallucinations by grounding in retrieved evidence
Dynamic knowledge without retraining the model
Production standard for enterprise AI applications
Modular design allows independent improvement of components