This comprehensive case study details Microsoft's extensive experience with red teaming generative AI systems in production, providing valuable insights into the practical challenges and methodologies for ensuring AI system safety and security at scale. The Microsoft AI Red Team (AIRT), established in 2018, has evolved significantly in response to two major trends: the increasing sophistication of AI systems and Microsoft's growing investment in AI products. Initially focused on traditional security vulnerabilities and ML model evasion, the team's scope expanded to address new capabilities, data modalities, and the integration of AI systems with external tools. A key contribution is their development of a systematic threat model ontology comprising five main components: * System: The end-to-end model or application being tested * Actor: The person being emulated (adversarial or benign) * TTPs (Tactics, Techniques, and Procedures): Attack methodologies mapped to MITRE frameworks * Weakness: System vulnerabilities enabling attacks * Impact: Downstream consequences (security or safety-related) The case study presents several important operational insights about running AI red teaming at scale: Automation and Tooling: The team developed PyRIT, an open-source Python framework that enables automated testing while preserving human judgment. This was crucial for achieving coverage across their large portfolio of AI products. PyRIT includes components for prompt datasets, encodings, automated attack strategies, and multimodal output scoring. Types of Vulnerabilities: The study identifies both traditional security risks (like SSRF, data exfiltration, credential leaking) and AI-specific vulnerabilities (like prompt injections and jailbreaks). Importantly, they found that simpler attack techniques often work better than sophisticated approaches - "real attackers don't compute gradients, they prompt engineer." Safety vs Security: The team's operations increasingly focused on Responsible AI (RAI) impacts alongside security vulnerabilities. RAI harms proved particularly challenging to measure due to their subjective nature and the probabilistic behavior of AI models. This led to the development of specialized evaluation approaches combining manual and automated methods. Human Expertise: Despite the emphasis on automation, the case study strongly argues for preserving the human element in red teaming. This is especially crucial for: * Subject matter expertise in specialized domains * Cultural competence for multilingual models * Emotional intelligence in assessing user impacts * Defining novel categories of harm Production Considerations: The study emphasizes several key aspects for running AI systems in production: * The importance of system-level testing rather than just model-level evaluation * The need to consider both benign and adversarial failure modes * The challenge of scaling testing across many products * The role of economic incentives in security (making attacks more costly than their potential benefits) Looking forward, the case study identifies several open questions for the field: * How to probe for emerging capabilities like persuasion and deception * Methods for translating red teaming practices across linguistic and cultural contexts * Ways to standardize and communicate findings across organizations The lessons learned emphasize that securing AI systems is an ongoing process rather than a one-time achievement. The team advocates for: * Break-fix cycles with multiple rounds of testing and mitigation * Defense-in-depth approaches combining technical and operational measures * Recognition that perfect safety is impossible but systems can be made increasingly robust This case study is particularly valuable because it provides concrete insights from extensive real-world experience rather than theoretical discussion. The systematic approach to threat modeling and the balance between automation and human expertise offers important guidance for organizations deploying AI systems in production. The emphasis on both security and safety impacts, along with the recognition of both traditional and AI-specific vulnerabilities, provides a comprehensive framework for thinking about AI system risks.