The Silver Key: What 275k Prompt Injections Taught Us About Hacking AI

No ratings

Presented at BSidesKC 2026 by

Bryan Smith

Prompt injection has quickly become one of the most discussed security issues surrounding large language models. But understanding how these attacks work in practice requires observing how people actually try to manipulate AI systems. To explore this, we created [Legend of Selara](https://selara.ai), a retro-styled prompt injection Capture-the-Flag challenge inspired by classic pixel adventure games. Participants interacted with Selara, a guarded AI oracle, attempting to extract a hidden secret known as the Silver Key. Over the course of the month-long challenge, hundreds of players submitted nearly 300,000 prompts, each attempting to bypass the model’s instructions. This talk shares what those attempts looked like. We’ll explore the strategies participants tried, the defenses that failed, and the approaches that ultimately succeeded. Using emerging community research such as the [Arcanum Prompt Injection Taxonomy](https://github.com/arcanumsec/arcanum-prompt-injection-taxonomy), thousands of prompts were categorized into intents, techniques, and evasions to better understand how attackers interact with LLMs. Along the way we’ll examine how an LLM fundamentally operates, and what these attempts reveal about guardrails, system prompts, and how language models behave when placed under pressure. Attendees will walk away with practical knowledge and resources to apply these insights themselves, including ways to use the dataset for adversarial testing, evaluate their own AI systems, experiment with the Selara CTF, or apply similar techniques when exploring AI-related bug bounty opportunities.