Towards new standards for real-time evaluation of anti-virus products
No ratings
Presented at Virus Bulletin 2007 by
"There is a desperate need for new test standards for today's anti-virus products. The dominant testing paradigm, scanning directories of files, is focused on old and known threats, and reveals little about product efficacy in the wild. Indeed, recent data from Symantec showed that 70% of computers with anti-virus get infected, in spite of having protection from products that pass tests with flying colors. The testing paradigm must change to include new and unknown threats, to provide meaningful measures of efficacy. Since most defences against new threats are proactive or behaviour-based, testing them adequately requires executing malicious code. This raises many issues including containment, repeatability, selection of test samples, how to define successful detection and removal, technical complexity, time and not least a lack of consensus in the industry on how such tests should be conducted. With the aim of building consensus, this paper proposes a set of standards for the real-time evaluation of anti-virus products. It discusses in detail the issues surrounding testing and presents a set of objective criteria for efficacy measurement. The critical innovations are to test against new and unknown threats from a variety of sources (live websites, independent malware feeds etc.), and to measure consistency in performance over time, rather than performance in any one test. The paper will include implementation details for a flexible automated testing framework along with some initial results. "