Skip to content

Evaluation and Benchmarking - NCP-GENL

Overview

Evaluation and benchmarking are critical for assessing model quality, comparing fine-tuning approaches, and ensuring production readiness. This document covers standard benchmarks, evaluation methodologies, performance profiling, and quality assurance practices for LLM deployments.

📖 NVIDIA AI Model Evaluation - Model evaluation in NeMo 📖 NVIDIA Model Analyzer - Performance analysis tool

Key Topics

1. Language Model Benchmarks

General Knowledge and Reasoning - MMLU (Massive Multitask Language Understanding): 57 subjects, multiple choice - HellaSwag: Commonsense reasoning completion tasks - ARC (AI2 Reasoning Challenge): Science question answering - Winogrande: Commonsense reasoning with pronoun resolution - TruthfulQA: Evaluates tendency to generate truthful answers

Code Generation - HumanEval: 164 Python programming problems, Pass@k metric - MBPP: 974 Python programming problems - CodeContests: Competitive programming problems - DS-1000: Data science coding tasks

Math and Reasoning - GSM8K: Grade school math word problems - MATH: Competition-level math problems - BBH (Big Bench Hard): Challenging reasoning tasks

Conversation and Instruction Following - MT-Bench: Multi-turn conversation evaluation - AlpacaEval: Instruction following quality - Chatbot Arena: Human preference-based ranking (ELO scores)

2. Evaluation Metrics

Language Modeling Metrics - Perplexity: Exponential of cross-entropy loss. Lower is better. - Cross-entropy loss: Standard training loss metric - Perplexity measures language modeling quality, not task quality - Useful for comparing model quality during pre-training

Generation Quality Metrics - BLEU: N-gram overlap with reference text - ROUGE: Recall-oriented text overlap metrics - BERTScore: Semantic similarity using contextual embeddings - Pass@k (Code): Probability of correct solution in k attempts

Classification Metrics - Accuracy: Fraction of correct predictions - F1 Score: Harmonic mean of precision and recall - AUC-ROC: Area under receiver operating characteristic curve

Human Evaluation - Likert scale ratings for quality, helpfulness, safety - Pairwise preference comparisons (A vs B) - Inter-annotator agreement metrics - Gold standard for final quality assessment

3. LLM-as-Judge Evaluation

Approach - Use a strong LLM (e.g., GPT-4) to evaluate outputs of another model - Score on multiple dimensions: helpfulness, accuracy, safety - More scalable than human evaluation - Correlates well with human judgments for many tasks

Best Practices - Use detailed evaluation rubrics in the judge prompt - Include reference answers when available - Average scores across multiple evaluations - Validate judge accuracy against human annotations - Be aware of position bias (order of presented options)

Limitations - Self-preference bias (models may prefer their own outputs) - Limited for specialized domain evaluation - Cannot fully replace human evaluation for safety - Judge quality depends on judge model capability

4. RAG-Specific Evaluation

Retrieval Quality - Precision@k: Relevant documents in top-k / k - Recall@k: Relevant documents found / total relevant - MRR: Mean Reciprocal Rank of first relevant result - NDCG: Considers both relevance and ranking position - Hit Rate: Queries with at least one relevant result

Generation Quality for RAG - Faithfulness: Answer consistent with retrieved context - Answer Relevancy: Answer addresses the user question - Context Relevancy: Retrieved context is relevant to query - Groundedness: Claims in answer are supported by context

RAGAS Framework - Automated RAG evaluation combining multiple metrics - Measures faithfulness, answer relevancy, context precision/recall - Works without ground-truth answers for some metrics - Uses LLM-as-judge for quality assessment

5. Performance Benchmarking

Key Performance Metrics - Time-to-First-Token (TTFT): Latency from request to first generated token - Inter-Token Latency (ITL): Time between consecutive generated tokens - Tokens per Second (TPS): Generation throughput - Requests per Second (RPS): Serving throughput - GPU Utilization: Percentage of GPU compute used - GPU Memory Usage: VRAM consumption during inference

Benchmarking Methodology - Use representative workloads (realistic input/output lengths) - Test at various concurrency levels (1, 10, 50, 100+ users) - Measure both throughput and latency distributions (p50, p95, p99) - Compare against baseline configuration - Run multiple iterations for statistical significance

NVIDIA Benchmarking Tools - TensorRT-LLM benchmarking scripts - NVIDIA Model Analyzer for Triton deployments - GPU profiling with NVIDIA Nsight Systems - DCGM for GPU-level metrics collection

📖 TensorRT-LLM Benchmarking - Performance benchmarking guide

6. Model Comparison Framework

Controlled Comparison - Same evaluation data and prompts across models - Same hardware and inference configuration - Same generation parameters (temperature, top-p, max tokens) - Report confidence intervals and statistical significance

Multi-Dimensional Analysis - Quality (benchmark scores, human evaluation) - Latency (TTFT, ITL at target concurrency) - Throughput (TPS, RPS at target quality) - Cost (GPU hours, cost per 1M tokens) - Memory (VRAM requirements, scaling characteristics)

Decision Framework - Define requirements: quality threshold, latency SLA, budget - Rank candidates on each dimension - Identify Pareto-optimal configurations - Validate winner on production-representative workloads

7. Safety and Alignment Evaluation

Safety Benchmarks - Toxicity detection accuracy - Bias evaluation across demographic groups - Jailbreak resistance testing - PII leakage assessment

Alignment Metrics - Instruction following accuracy - Refusal appropriateness (not over-refusing, not under-refusing) - Helpfulness rating - Honesty (admitting uncertainty, avoiding fabrication)

Red-Teaming - Adversarial prompt testing - Multi-turn attack scenarios - Automated red-teaming with adversarial models - Document and mitigate discovered vulnerabilities

📖 NeMo Guardrails Evaluation - Testing guardrails effectiveness

8. Continuous Evaluation in Production

Online Evaluation - Sample and evaluate production requests periodically - Track quality metrics over time - Detect model quality degradation - A/B test new model versions

Monitoring Dashboards - Real-time latency and throughput metrics - Quality score trends - Error rates and failure analysis - User feedback and satisfaction tracking

Feedback Loops - Collect user feedback (thumbs up/down, corrections) - Use feedback for ongoing model improvement - Track feedback trends for quality signals - Integrate feedback into fine-tuning data

Alerting - Set thresholds for latency SLA violations - Alert on quality metric drops - Monitor GPU health and utilization anomalies - Track cost-per-query trends

Exam Focus Areas

Critical Concepts

  • Know standard benchmarks and what they measure
  • Understand performance metrics (TTFT, ITL, TPS)
  • Know RAG evaluation metrics (faithfulness, relevance)
  • Understand LLM-as-judge methodology and limitations
  • Be familiar with NVIDIA performance benchmarking tools

Common Exam Questions

  • "Which metric measures code generation quality?" - Pass@k
  • "What does TTFT measure?" - Time from request to first generated token
  • "How to evaluate RAG faithfulness?" - Check if answer matches retrieved context
  • "Best approach for scalable LLM evaluation?" - LLM-as-judge with detailed rubrics
  • "Which tool profiles Triton deployment performance?" - Model Analyzer