Lessons Learned from Red Teaming 100 Generative Al Applications

No ratings

Presented at Bluehat / BlueHat India 2024 by

This talk covers the big lessons learned by the Microsoft Al Red Team in identifying safety and security vulnerabilities in flagship Al systems like Bing Copilot, Security Copilot, M365 Copilot, and models such as GPT-4, DALLE, and the Phi series: 1. Prompt Injection gets all the attention, but traditional security failures are still top billing cexample case study: credentials in Copilot source code, code execution via jailbreak in Code Interpreter) 2. AS models get better, risk evolves (case study: GPT-40 which supported audio, video modalities had to be assessed for its ability to have romantic relationship with user) 3. LLM Guided Red Teaming can help us cover more of the risk landscape but is still finicky. Here we walk through an example of how our OSS automation tool PyRIT helped with saving close to 160 hours of manual probing, but how the scorer we used in evaluating frequently broke when we did RAI red teaming. 4. No free lunch in making Al systems safe: Tradeoffs that we have observed (example: ina facial recognition model, the more attempts were made to suppress the model from observing the face, the more the model focused on clothing. In another example, we found that smaller models are more immune to jailoreaks compared to larger counterparts since they 5. The difficulty in making Al systems safe: simple attacks have large impact (we show how a simple jailbreak could lead to dropping tables in production database that had Copilot turned on) and the inability to distinguish inadvertent failures and intentional failures