Bolt's product security team secures applications for over 200 million customers and 4.5 million partners across 600+ cities in 50 countries. Despite this scale, our request to gain access to Mythos-class models was left on read. So we decided to build our own tooling around models we already had to figure out whether we should feel scared or empowered due to the rise of AI-powered vulnerability finders. This talk is a story of what happens when you duct-tape "good enough" models into a security reviewer for a codebase serving more than 5 billion requests per day. We will share our experience of building layers of deterministic scaffolding and multi-agent cross-checking to keep the TP to FP ratio under control. **Attendees will gain insights into:** * **Architecture on a budget:** Why "deterministic pipeline first, LLM last" beats throwing a whole codebase at a model and praying. * **Multi-agent QA:** How worker agents plus a validator agent cut false positives, assign vibes-based severity ratings and where that pattern still falls apart. * **Wins and misses:** Concrete findings our tool caught that humans missed - and why vice versa might not matter. * **Prompting for security:** Why narrow, focused scopes produce accurate findings and generic "find the vulns" prompts produce finding-shaped garbage.