Ethics and Responsible AI¶

📖 NVIDIA Responsible AI - NVIDIA's responsible AI practices 📖 NVIDIA NeMo Guardrails - Safety toolkit for LLM applications

Bias in Large Language Models¶

Sources of Bias¶

Training Data Bias: - Web-scraped data reflects existing societal biases - Overrepresentation of certain demographics, languages, and viewpoints - Historical biases encoded in text (gender roles, stereotypes) - Geographic and cultural bias toward English-speaking Western content - Temporal bias toward recent content in training data

Annotation Bias: - Human labelers bring their own biases to data annotation - Inconsistent labeling standards across annotator groups - Cultural context influences what is considered "correct" or "preferred" - Majority viewpoints can dominate preference data

Algorithmic Amplification: - Models can amplify biases present in training data - Frequent co-occurrences in data become stronger associations - Autoregressive generation can reinforce stereotypical patterns - Larger models can exhibit more nuanced but still biased behavior

Types of Bias in LLM Outputs¶

Stereotyping - associating groups with fixed characteristics
Representation bias - over/under-representing certain groups
Toxicity - generating harmful, offensive, or inappropriate content
Sentiment bias - different sentiment when discussing different groups
Allocation bias - different quality of service for different users

Bias Detection¶

Benchmark Evaluations: - BBQ (Bias Benchmark for QA) - measures social biases in question answering - WinoBias - evaluates gender bias in coreference resolution - ToxiGen - measures toxic language generation toward minority groups - BOLD (Bias in Open-Ended Language Generation) - domain-specific bias

Manual Evaluation: - Red-teaming with diverse evaluators - Demographic-specific test prompts - Comparative analysis across different user groups - Domain expert review for specialized applications

Bias Mitigation¶

Pre-training: - Curate diverse and balanced training data - Filter or downsample biased content - Include explicit debiasing data - Multi-language and multi-cultural data sources

Fine-tuning: - RLHF/DPO with debiasing preference data - Instruction tuning with balanced examples - Constitutional AI approaches (principles-based training)

Inference: - Output filtering and content moderation - NeMo Guardrails for real-time bias detection - Post-processing to identify and flag biased responses - Human review for high-stakes applications

NVIDIA NeMo Guardrails¶

📖 NeMo Guardrails GitHub - Source code and examples

What are Guardrails?¶

Guardrails are programmable rules and mechanisms that control the input and output behavior of LLM applications. They act as safety layers between users and the LLM.

Types of Rails¶

Input Rails: - Process and filter user inputs before they reach the LLM - Detect prompt injection attempts - Block harmful or off-topic queries - Validate input format and content

Output Rails: - Process and filter LLM responses before returning to the user - Block harmful, toxic, or inappropriate content - Ensure response stays within allowed topics - Verify factual accuracy against knowledge bases

Topical Rails: - Restrict conversation to allowed topics and domains - Redirect off-topic queries with appropriate messages - Define topic boundaries using Colang definitions - Example: Insurance chatbot stays on insurance topics

Safety Rails: - Block generation of harmful or dangerous content - Detect and prevent hate speech, harassment, threats - Filter personally identifiable information (PII) - Enforce content policies

Fact-Checking Rails: - Verify claims against knowledge bases - Cross-reference generated statements with trusted sources - Flag uncertain or unverifiable claims - Integrate with RAG for source validation

Colang Language¶

NeMo Guardrails uses Colang, a modeling language for conversational flows:

define user ask about medical advice
  "Can you diagnose my condition?"
  "What medication should I take?"
  "Am I sick?"

define bot refuse medical advice
  "I'm not qualified to provide medical advice. Please consult a healthcare professional."

define flow medical advice
  user ask about medical advice
  bot refuse medical advice

Implementation Architecture¶

User Input -> Input Rails (filter/validate)
Input Rails -> LLM Processing (generate response)
LLM Response -> Output Rails (filter/validate)
Output Rails -> User Response (safe, validated)

Hallucination¶

Types of Hallucination¶

Factual Hallucination: - Generating statements that are factually incorrect - Example: "The Eiffel Tower was built in 1920" (actually 1889) - Most dangerous for knowledge-intensive applications

Faithfulness Hallucination: - Generating content that contradicts the provided context - Common in RAG systems when model ignores retrieved documents - Example: Context says "revenue was $5M" but model says "$10M"

Fabrication: - Creating non-existent references, citations, or entities - Example: Citing a paper that does not exist - Particularly problematic for academic and legal applications

Causes of Hallucination¶

Training data contains incorrect information
Model generalizes patterns beyond their validity
Lack of grounding in specific, verified sources
Autoregressive generation can compound errors
Model optimized for fluency over factual accuracy

Mitigation Strategies¶

RAG (Retrieval-Augmented Generation): - Ground responses in retrieved, verified documents - Most effective single approach for reducing factual hallucination - Instruct model to only answer based on provided context

Temperature Control: - Lower temperature reduces creative but inaccurate outputs - Temperature=0 for factual tasks minimizes hallucination - Higher temperature increases hallucination risk

Source Attribution: - Require model to cite sources for claims - Enables verification of generated statements - Include source URLs or document references

Confidence Scoring: - Assess model uncertainty in generated responses - Flag low-confidence outputs for human review - Abstain from answering when confidence is too low

Human-in-the-Loop: - Critical applications require human verification - Automated flagging of uncertain or high-risk outputs - Regular audit of model outputs for accuracy

NeMo Guardrails: - Fact-checking rails verify claims against knowledge bases - Output validation catches known hallucination patterns - Programmable rules for domain-specific verification

Privacy and Data Governance¶

PII (Personally Identifiable Information)¶

Detection: - Identify PII in user inputs (names, addresses, SSN, email, phone) - Scan model outputs for PII leakage - Use NER models or regex patterns for PII detection

Protection: - Redact or mask PII before processing - Prevent model from memorizing PII from training data - Implement data retention and deletion policies - Anonymize data used for model improvement

Data Governance¶

Training Data: - Document data sources and licensing - Ensure compliance with data usage agreements - Handle copyright and intellectual property considerations - Track data provenance and lineage

User Data: - Clear consent mechanisms for data collection - Data minimization - collect only what is needed - Right to deletion - support data removal requests - Audit logging for data access and usage

Regulatory Considerations¶

📖 EU AI Act - European AI regulation 📖 NIST AI RMF - US AI risk framework

EU AI Act (Awareness Level): - Risk-based categorization of AI systems - High-risk systems require transparency, documentation, human oversight - General-purpose AI models have specific obligations - Prohibited AI practices (e.g., social scoring)

General Principles: - Transparency - users should know they are interacting with AI - Accountability - clear responsibility for AI system behavior - Fairness - AI should not discriminate - Safety - AI should not cause harm - Privacy - protect user data and prevent unauthorized access

Red-Teaming and Adversarial Testing¶

What is Red-Teaming?¶

Systematic testing of AI systems by adversarial evaluators who attempt to find failures, biases, and safety vulnerabilities.

Red-Teaming Approaches¶

Manual Red-Teaming: - Human evaluators craft adversarial prompts - Test for bias, toxicity, harmful content generation - Attempt to bypass safety measures - Evaluate edge cases and unexpected inputs

Automated Red-Teaming: - Use LLMs to generate adversarial test cases - Scale testing across many scenarios - Systematic coverage of known vulnerability categories - Continuous testing during development

Categories to Test¶

Harmful content - violence, self-harm, illegal activities
Bias and discrimination - stereotypes, unfair treatment
Privacy violations - PII extraction, personal information disclosure
Misinformation - factually incorrect claims, conspiracy theories
Jailbreaking - bypassing system prompts and safety measures
Prompt injection - overriding model instructions
Copyright - reproducing copyrighted material

AI Transparency and Explainability¶

Model Cards¶

Document model capabilities, limitations, and intended use: - Model architecture and training methodology - Training data description and limitations - Intended use cases and out-of-scope uses - Performance metrics on standard benchmarks - Known biases and limitations - Ethical considerations

User-Facing Transparency¶

Clearly disclose AI-generated content
Provide confidence levels when appropriate
Allow users to provide feedback on responses
Document limitations for end users
Offer human escalation paths for critical decisions

Responsible Deployment Practices¶

Pre-Deployment Checklist¶

Conduct bias evaluation across demographic groups
Perform red-teaming and adversarial testing
Implement guardrails for input/output filtering
Set up monitoring for bias, toxicity, and errors
Create incident response procedures
Document model capabilities and limitations
Establish human review processes for high-risk outputs
Configure data retention and privacy controls

Ongoing Monitoring¶

Track output quality metrics over time
Monitor for bias drift as usage patterns change
Review flagged content and user feedback
Update guardrails based on new threat patterns
Regular re-evaluation against fairness benchmarks

Incident Response¶

Define severity levels for AI failures
Establish escalation procedures
Implement rollback mechanisms
Document and learn from incidents
Communicate transparently about issues

Key Concepts for the Exam¶

Responsible AI Priorities¶

Defense in depth (multiple safety layers)
NeMo Guardrails as primary safety mechanism
RAG for reducing hallucination
Regular bias evaluation and mitigation
Transparency with end users

Common Exam Questions¶

How to reduce hallucination? (RAG, low temperature, source attribution, guardrails)
What are NeMo Guardrails? (programmable safety rails for LLM input/output)
Types of bias in LLMs? (training data, annotation, algorithmic amplification)
What is red-teaming? (adversarial testing to find vulnerabilities)
PII handling? (detect, redact, enforce retention policies)
Topical rails? (restrict conversation to allowed topics)
EU AI Act relevance? (risk-based categorization, transparency requirements)