RAG Architecture and Vector Databases¶

📖 NVIDIA NeMo Retriever - NVIDIA's RAG pipeline components 📖 NVIDIA RAG Tutorial - Building RAG with NVIDIA tools

Retrieval-Augmented Generation (RAG) Overview¶

What is RAG?¶

RAG combines information retrieval with text generation to produce grounded, factual responses. Instead of relying solely on the knowledge encoded in model parameters during training, RAG retrieves relevant information from external sources and includes it in the generation context.

Why RAG?¶

• Reduces hallucination - grounds responses in retrieved evidence
• Dynamic knowledge - external data can be updated without retraining
• Source attribution - can cite specific documents for transparency
• Domain adaptation - adds domain-specific knowledge without fine-tuning
• Cost-effective - cheaper than retraining for knowledge updates
• Privacy - keeps sensitive data in a controlled retrieval system

RAG vs Fine-Tuning¶

RAG Pipeline Components¶

1. Document Ingestion¶

Document Types: - PDF, Word, HTML, Markdown, plain text - Structured data (CSV, JSON, databases) - Semi-structured (emails, wikis, Confluence) - Code repositories

Processing Steps: 1. Parse documents into text (PDF extraction, HTML stripping) 2. Clean and normalize text (remove headers/footers, fix encoding) 3. Extract metadata (title, date, author, source URL) 4. Handle tables, images, and non-text content

2. Chunking Strategies¶

📖 LangChain Text Splitters - Chunking implementations

Fixed-Size Chunking: - Split by character or token count (e.g., 512 tokens) - Add overlap between chunks (10-20% of chunk size) - Simple and predictable - May split mid-sentence or mid-concept

Recursive Chunking: - Try splitting by multiple separators in order of preference - Typical hierarchy: paragraphs -> sentences -> characters - Respects natural text boundaries better than fixed-size - Most commonly used in practice

Semantic Chunking: - Split based on topic or meaning changes - Uses embedding similarity between segments - Higher quality but more computationally expensive - Good for documents with mixed topics

Document-Aware Chunking: - Respects document structure (headers, sections, lists) - Keeps related content together - Best for structured documents (technical docs, legal docs) - May produce variable-size chunks

Chunk Size Trade-offs: - Small chunks (128-256 tokens): More precise retrieval, less context per chunk - Medium chunks (256-512 tokens): Good balance for most use cases - Large chunks (512-1024 tokens): More context, but may include irrelevant info - Overlap: Prevents information loss at chunk boundaries

3. Embedding Models¶

📖 NVIDIA Embedding Models - NVIDIA embedding model catalog

How Embeddings Work: - Convert text into dense vector representations (arrays of floating-point numbers) - Semantically similar texts have similar vectors - Typically 384 to 4096 dimensions - Generated by encoder models trained for semantic similarity

Popular Embedding Models: - NVIDIA NV-Embed - NVIDIA's embedding models, available via NIM - E5-large-v2 - Strong open-source embedding model - BGE (BAAI General Embeddings) - High-quality multilingual embeddings - GTE (General Text Embeddings) - Alibaba's embedding models

Embedding Considerations: - Match embedding model to your domain and language - Same model must be used for indexing and querying - Dimensionality affects storage requirements and search speed - Normalize embeddings for cosine similarity search

4. Vector Database and Indexing¶

Similarity Metrics: - Cosine similarity - measures angle between vectors (most common) - Dot product - measures magnitude-weighted similarity - Euclidean distance (L2) - measures geometric distance - Inner product - similar to dot product, used in some implementations

Index Types:

HNSW (Hierarchical Navigable Small World): - Graph-based index with multiple layers - Excellent query latency (sub-millisecond for millions of vectors) - Higher memory usage (stores graph structure) - Good recall (typically >95%) - Best for: real-time search, sub-100ms latency requirements

IVF (Inverted File Index): - Partitions vectors into clusters (Voronoi cells) - Searches only relevant clusters (nprobe parameter) - Good balance of speed and accuracy - Lower memory overhead than HNSW - Best for: large-scale search with memory constraints

5. Vector Databases¶

FAISS (Facebook AI Similarity Search): - Open-source library (not a full database) - GPU-accelerated similarity search - Multiple index types (Flat, IVF, HNSW, PQ) - No built-in persistence, filtering, or scaling - Best for: research, batch processing, GPU-heavy workloads

📖 FAISS Documentation - FAISS library guide

Milvus: - Purpose-built open-source vector database - Horizontal scaling with distributed architecture - Attribute filtering combined with vector search - Built-in persistence and replication - Multiple index types supported - Best for: production deployments, large-scale applications

📖 Milvus Documentation - Milvus documentation

Other Options: - Pinecone - fully managed, serverless vector database - Weaviate - open-source with hybrid search (vector + keyword) - Chroma - lightweight, developer-friendly, good for prototyping - Qdrant - high-performance with advanced filtering - pgvector - PostgreSQL extension for vector search

6. Retrieval Strategies¶

Dense Retrieval: - Uses embedding similarity for finding relevant documents - Good for semantic matching (meaning-based) - May miss exact keyword matches

Sparse Retrieval (BM25): - Traditional keyword-based matching - Good for exact term matches - Misses semantic similarity

Hybrid Search: - Combines dense and sparse retrieval - Uses reciprocal rank fusion (RRF) or weighted combination - Better coverage than either method alone - Recommended for production systems

Re-ranking: - Cross-encoder model scores query-document pairs - Applied to top-k results from initial retrieval - Significantly improves precision - More computationally expensive (runs on each pair) - Typical pipeline: retrieve top-50 with bi-encoder, re-rank to top-5 with cross-encoder

7. Generation with Context¶

Context Integration: - Retrieved chunks are inserted into the prompt as context - System prompt instructs the model to answer based on provided context - Include source attribution instructions

Context Window Management: - Total tokens = system prompt + retrieved context + user query + generated response - Must fit within model's context window - Prioritize most relevant chunks when context is limited - Consider summarizing long retrieved passages

Advanced RAG Techniques¶

Query Transformation¶

• Query expansion - rephrase or expand the user query
• HyDE (Hypothetical Document Embeddings) - generate a hypothetical answer, use it for retrieval
• Multi-query - generate multiple query variations, combine results
• Step-back prompting - generate a broader query for complex questions

Advanced Retrieval¶

• Parent-child chunking - retrieve small chunks, return parent (larger) chunks for context
• Sentence window retrieval - retrieve at sentence level, expand to surrounding sentences
• Knowledge graph integration - combine vector search with graph traversal
• Multi-hop retrieval - iterative retrieval for complex multi-step questions

Evaluation¶

Retrieval Metrics: - Recall@k - fraction of relevant documents in top-k results - Precision@k - fraction of top-k results that are relevant - MRR (Mean Reciprocal Rank) - average of 1/rank of first relevant result - NDCG - normalized discounted cumulative gain

Generation Metrics: - Faithfulness - is the answer supported by retrieved context? - Answer relevancy - does the answer address the question? - Context relevancy - are retrieved documents relevant to the question?

Key Concepts for the Exam¶

RAG Pipeline Design¶

• Choose chunking strategy based on document type
• Select embedding model appropriate for domain
• Use HNSW for low-latency, IVF for memory-constrained scenarios
• Implement hybrid search for best retrieval quality
• Add re-ranking for precision-critical applications

Common Exam Questions¶

• When to use RAG vs fine-tuning? (RAG for dynamic knowledge, fine-tuning for behavior)
• What causes poor RAG quality? (bad chunking, wrong embedding model, no re-ranking)
• FAISS vs Milvus? (FAISS is a library, Milvus is a full database with scaling)
• What is HNSW? (graph-based ANN index with fast query, high memory)
• Hybrid search advantage? (captures both semantic and keyword matches)
• How to reduce hallucination with RAG? (better retrieval, instruct to cite sources, add guardrails)