Most of what the industry is providing in "black box" application security testing today is invalid. This talk will attempt to demonstrate ways we can be more consistant, more thorough, and more honest about the results from "black box" application security testing. At this talk we will provide insights we've learned from performing application testing, writing application testing tools, and the OSSTMM (3.0) methodology for for application testing. This will be the first public demonstration of the Cruiser web application testing tool.