AI security¶

Securing systems with a language model in the request path. Classic appsec still applies; what is new is a component that reads untrusted natural language, decides what to do at inference time, and can be talked into doing something else.

flowchart LR
  subgraph sources[Context sources - mixed trust]
    U[User message]
    D[Retrieved docs]
    W[Fetched pages]
    T[Tool output]
  end
  sources --> M[Model]
  M --> TB{Tool boundary<br/>authz on user identity}
  TB -- denied --> AUD[Audit + alert]
  TB -- allowed --> ACT[Scoped action]
  M --> OUT[Output] --> SAN[Treat as untrusted input]
  SAN --> DOWN[Renderer, DB, shell, another agent]

The two boxes that carry the weight: the context window, where untrusted text mixes with your instructions, and the tool boundary, the last non-model control that can refuse.

Learn¶

• Prompt injection explained - why it exists, why filters do not fix it
• AI threat modeling - three questions that surface most of the risk
• Guardrails and safety - the control layer around the model
• Tool use and function calling - the mechanism agents act through
• Agentic loops - why autonomy multiplies blast radius
• Evals for LLMs - measuring safety regressions, not just accuracy

Reference¶

• AI security hub - the full engineering-depth set
• OWASP Top 10 for LLM Applications - the risk taxonomy, entry by entry
• Prompt injection defense - direct and indirect injection, what actually works
• Agent and tool security - least privilege for agents, sandboxing, audit
• Model supply chain security - provenance, weight integrity, AI-BOM
• LLM red teaming - adversarial testing method and metrics

Govern¶

• EU AI Act - risk tiers, provider vs deployer, timelines
• NIST AI RMF - Govern, Map, Measure, Manage
• ISO/IEC 42001 - the certifiable AI management system
• GDPR - applies independently to any personal data in the pipeline
• SOC 2 - the service controls an AI feature inherits

Build¶

• Set up an eval harness - including injection and safety regression tests
• Build a RAG pipeline - the retrieval layer indirect injection targets
• Build a Claude agent with MCP - tool boundaries in practice
• Implement zero trust - the identity model agents belong inside

Certify¶

No certification covers this end to end yet. Closest coverage, in rough order of relevance:

AI-specific - AWS GenAI Developer Professional (AIP-C01) - dedicated AI safety, security, and governance domain - AWS AI Practitioner (AIF-C01) - responsible AI at foundational depth - Azure AI Engineer (AI-102) - Content Safety and responsible AI tooling - NVIDIA GenAI and LLMs Associate - ethics and responsible AI domain - Oracle OCI Generative AI Professional - securing GenAI workloads on OCI

Cloud security foundations the AI controls sit on - Microsoft Cybersecurity Architect (SC-100) - designing the security strategy AI systems live inside - Microsoft Information Security Administrator (SC-401) - Purview DSPM for AI, data protection for Copilot - Cloud Security Alliance CCSK - cloud governance - ISC2 CCSP - cloud security architecture at depth - AWS Security Specialty (SCS-C02) - IAM and detection controls agents inherit

Offensive method - OSCP (PEN-200) - the testing mindset red teaming borrows

Roadmap¶

Security path: Security Engineer roadmap. AI path: AI/ML Engineer roadmap. This topic sits at the intersection, and no single roadmap owns it yet.