Skip to content

AI security

Securing systems that have a language model somewhere in the request path.

Classic application security still applies: you still need authentication, least privilege, input validation, and logging. What changes is that one component in your architecture now takes untrusted natural language as input, produces text that other components trust, and can be persuaded to do things you did not intend. That single property breaks assumptions that most security controls were built on.

This directory covers the attack surface, the defenses that actually work, and how to test them.

Why LLM systems need their own security treatment

flowchart TB
  subgraph classic[Classic app]
    U1[User input] --> V1[Validate / parse]
    V1 --> C1[Code with fixed control flow]
    C1 --> A1[Actions]
  end

  subgraph llm[LLM app]
    U2[User input] --> P[Prompt]
    RET[Retrieved docs] --> P
    TOOLOUT[Tool output] --> P
    P --> M[Model]
    M --> A2[Actions via tools]
    A2 --> TOOLOUT
  end

  classic ~~~ llm

In a classic app, control flow is code you wrote. In an LLM app, control flow is decided at inference time by a model reading a prompt, and that prompt is assembled from sources an attacker may control: the user's message, a retrieved document, the output of a previous tool call, a web page the agent fetched. There is no reliable separator between "instructions" and "data" inside that prompt.

Everything in this directory follows from that one fact.

Pages

Page What it covers
OWASP Top 10 for LLM Applications The reference risk taxonomy, entry by entry, with cloud-specific mitigations
Prompt injection defense Direct and indirect injection, why filtering fails, what actually reduces risk
Agent and tool security Excessive agency, tool scoping, sandboxing, human-in-the-loop design
Model supply chain security Model provenance, weight integrity, poisoned datasets, dependency risk
LLM red teaming How to test an LLM system adversarially, and what to measure

Governance and compliance

Security controls are one half. The other half is being able to demonstrate governance to a regulator, an auditor, or an enterprise customer.

Where this connects

Learn the concepts first - Prompt injection explained - the plain-English version - AI threat modeling - how to reason about a system you are building - Guardrails and safety - the control layer - Tool use and function calling - the mechanism agents use to act - Agentic loops - why autonomy multiplies blast radius

Build - Set up an eval harness - regression testing, including safety regressions - Build a RAG pipeline - the retrieval layer that indirect injection targets - Implement zero trust - the identity model agents should sit inside

Certify

No certification covers this material end to end yet. The closest coverage:

Reference

📖 OWASP GenAI Security Project - the working group behind the LLM Top 10 and related guidance 📖 NIST AI Risk Management Framework - AI RMF 1.0 and the Generative AI Profile 📖 MITRE ATLAS - adversarial threat landscape for AI systems, structured like ATT&CK 📖 Anthropic: Claude's constitution and safeguards - published safety research 📖 Google Secure AI Framework (SAIF) - Google's conceptual framework for securing AI systems 📖 Microsoft Responsible AI Standard - Microsoft's governance model and tooling