OWASP GenAI Security Training Course

Day 1: Foundations and Core Threats

Module 1: Introduction to the OWASP GenAI Security Project (1 hour)

Learning Objectives:

• Understand the transition from OWASP Top 10 to GenAI-specific security issues
• Explore the ecosystem and resources of the OWASP GenAI Security Project
• Distinguish between traditional application security and AI security

Topics Covered:

• The mission and scope of the OWASP GenAI Security Project
• An introduction to the Threat Defense COMPASS framework
• Understanding the AI security landscape and regulatory obligations
• AI attack surfaces compared to traditional web application vulnerabilities

Practical Exercise: Configuring the OWASP Threat Defense COMPASS tool and conducting an initial threat assessment

Module 2: OWASP Top 10 for LLMs - Part 1 (2.5 hours)

Learning Objectives:

• Gain mastery over the first five critical LLM vulnerabilities
• Comprehend attack vectors and exploitation methods
• Apply effective mitigation strategies

Topics Covered:

LLM01: Prompt Injection

• Techniques for direct and indirect prompt injection
• Hidden instruction attacks and cross-prompt contamination
• Practical examples, such as jailbreaking chatbots and evading safety measures
• Defensive strategies, including input sanitization, prompt filtering, and differential privacy

LLM02: Sensitive Information Disclosure

• Extraction of training data and leakage of system prompts
• Analyzing model behavior to identify sensitive information exposure
• Privacy implications and regulatory compliance considerations
• Mitigation techniques, such as output filtering, access controls, and data anonymization

LLM03: Supply Chain Vulnerabilities

• Security of third-party model dependencies and plugins
• Compromised training datasets and model poisoning
• Risk assessment of AI vendors
• Best practices for secure model deployment and verification

Practical Exercise: A hands-on lab focusing on prompt injection attacks against vulnerable LLM applications and the implementation of defensive measures

Module 3: OWASP Top 10 for LLMs - Part 2 (2 hours)

Topics Covered:

LLM04: Data and Model Poisoning

• Techniques for manipulating training data
• Modifying model behavior through poisoned inputs
• Backdoor attacks and verifying data integrity
• Prevention methods, including data validation pipelines and provenance tracking

LLM05: Improper Output Handling

• Insecure processing of content generated by LLMs
• Code injection via AI-generated outputs
• Cross-site scripting through AI responses
• Frameworks for output validation and sanitization

Practical Exercise: Simulating data poisoning attacks and establishing robust output validation mechanisms

Module 4: Advanced LLM Threats (1.5 hours)

Topics Covered:

LLM06: Excessive Agency

• Risks associated with autonomous decision-making and boundary violations
• Managing agent authority and permissions
• Unintended system interactions and privilege escalation
• Implementing guardrails and human oversight controls

LLM07: System Prompt Leakage

• Vulnerabilities related to the exposure of system instructions
• Disclosure of credentials and logic through prompts
• Methods for extracting system prompts
• Securing system instructions and external configurations

Practical Exercise: Designing secure agent architectures with appropriate access controls and monitoring

Day 2: Advanced Threats and Implementation

Module 5: Emerging AI Threats (2 hours)

Learning Objectives:

• Understand cutting-edge AI security threats
• Implement advanced detection and prevention techniques
• Design resilient AI systems to withstand sophisticated attacks

Topics Covered:

LLM08: Vector and Embedding Weaknesses

• Vulnerabilities in RAG systems and vector database security
• Embedding poisoning and attacks on similarity manipulation
• Adversarial examples in semantic search
• Securing vector stores and implementing anomaly detection

LLM09: Misinformation and Model Reliability

• Detecting and mitigating hallucinations
• Bias amplification and fairness considerations
• Mechanisms for fact-checking and source verification
• Integrating content validation and human oversight

LLM10: Unbounded Consumption

• Resource exhaustion and denial-of-service attacks
• Strategies for rate limiting and resource management
• Cost optimization and budget controls
• Performance monitoring and alerting systems

Practical Exercise: Constructing a secure RAG pipeline with vector database protection and hallucination detection

Module 6: Agentic AI Security (2 hours)

Learning Objectives:

• Understand the unique security challenges of autonomous AI agents
• Apply the OWASP Agentic AI taxonomy to real-world systems
• Implement security controls for multi-agent environments

Topics Covered:

• Introduction to Agentic AI and autonomous systems
• The OWASP Agentic AI Threat Taxonomy, covering Agent Design, Memory, Planning, Tool Use, and Deployment
• Security and coordination risks in multi-agent systems
• Attacks involving tool misuse, memory poisoning, and goal hijacking
• Securing agent communication and decision-making processes

Practical Exercise: Threat modeling exercise using the OWASP Agentic AI taxonomy on a multi-agent customer service system

Module 7: Implementing OWASP Threat Defense COMPASS (2 hours)

Learning Objectives:

• Master the practical application of Threat Defense COMPASS
• Integrate AI threat assessment into organizational security programs
• Develop comprehensive AI risk management strategies

Topics Covered:

• In-depth exploration of the Threat Defense COMPASS methodology
• Integration of the OODA Loop: Observe, Orient, Decide, Act
• Mapping threats to MITRE ATT&CK and ATLAS frameworks
• Creating AI Threat Resilience Strategy Dashboards
• Integration with existing security tools and processes

Practical Exercise: Conducting a complete threat assessment using COMPASS for a Microsoft Copilot deployment scenario

Module 8: Practical Implementation and Best Practices (2.5 hours)

Learning Objectives:

• Design secure AI architectures from the ground up
• Implement monitoring and incident response for AI systems
• Create governance frameworks for AI security

Topics Covered:

Secure AI Development Lifecycle:

• Security-by-design principles for AI applications
• Code review practices for LLM integrations
• Testing methodologies and vulnerability scanning
• Deployment security and production hardening

Monitoring and Detection:

• Logging and monitoring requirements specific to AI
• Anomaly detection for AI systems
• Incident response procedures for AI security events
• Forensics and investigation techniques

Governance and Compliance:

• AI risk management frameworks and policies
• Regulatory compliance considerations, such as GDPR and the AI Act
• Third-party risk assessment for AI vendors
• Security awareness training for AI development teams

Practical Exercise: Designing a complete security architecture for an enterprise AI chatbot, including monitoring, governance, and incident response procedures

Module 9: Tools and Technologies (1 hour)

Learning Objectives:

• Evaluate and implement AI security tools
• Understand the current landscape of AI security solutions
• Build practical detection and prevention capabilities

Topics Covered:

• The AI security tool ecosystem and vendor landscape
• Open-source security tools, including Garak, PyRIT, and Giskard
• Commercial solutions for AI security and monitoring
• Integration patterns and deployment strategies
• Criteria for tool selection and evaluation frameworks

Practical Exercise: Hands-on demonstration of AI security testing tools and implementation planning

Module 10: Future Trends and Wrap-up (1 hour)

Learning Objectives:

• Understand emerging threats and future security challenges
• Develop strategies for continuous learning and improvement
• Create action plans for organizational AI security programs

Topics Covered:

• Emerging threats, such as deepfakes, advanced prompt injection, and model inversion
• Future developments and roadmap for the OWASP GenAI project
• Building AI security communities and facilitating knowledge sharing
• Continuous improvement and threat intelligence integration

Action Planning Exercise: Developing a 90-day action plan to implement OWASP GenAI security practices within participants' organizations