The Hard Part Isn't Building the Agent: On Measuring Agent Effectiveness to Improve It

No ratings

Presented at [un]prompted 2026 by

Joshua Saxe

As AI coding tools drive the cost of building security agents toward zero, the hard problem becomes knowing whether they'll actually work in the wild against real attacks and vulnerabilities we haven't seen before. This talk shares a practical journey from naive precision/recall metrics on old data toward multi-dimensional evaluation that captures reasoning quality, evidence gathering, and tool-calling logic --and shows how proper measurement unlocks automated agent improvement using genetic algorithms and AI coding tools. Live demo included.