3A Unique Analogs for Validating Security and Performance Claims (A Case Study)
No ratings
Presented at COSAC 2015 by
This case study is focused on a test and evaluation model designed and used to evaluate an internally-developed security tool, which was developed to detect remote access Trojan activity in streaming traffic. The paper includes a general description of the tool under test, the claims of the tool, the test design and the results of the testing. To authorize the tool for use internally, the tool had to detect the remote access Trojan successfully and had to have a false positive rate of less than .001%. Additionally, there was a requirement to validate which operating systems were usable from both an attacker and victim standpoint. These two requirements forced our joint operations analysis and cyber security teams to look at a number of statistical sampling methods to find a sound, proven methodology, which could withstand peer review. The team evaluated methods ranging from those used to detect manufacturing defects in cloth to air and water quality measurement methods before selecting assembly line testing of manufactured parts as an appropriate methodology. The paper includes a description of the selection process and the reasoning behind the method selected. Finally, the paper includes a detailed description of the test environment, the conduct of the test and the results measured. The paper contains a description of future applicability and areas for further investigation. The uniqueness and value of this case study lies in three primary areas. First, many organizations struggle to quantify what the ramifications of a given security solution will be in their environment. Vendor claims typically claim scalability, low false positive rates, and the like without solid, independent testing to validate their claims. In many cases this is due to the inability of the vendor or assessor to identify and apply appropriate statistical analysis measures to the process. This case study reveals not just what we selected, but how we selected it. Second, many times security organizations are blamed for overall network performance problems due to the inability of the vendor and the organization to quantitatively demonstrate the true performance overhead of the security controls being implemented. In this paper we discuss how we tackled that problem and lay out the process we went through. Finally, almost every security organization is asked something along the lines of “how much is a pound of security worth?” In this paper we present a model for addressing that question and briefly discuss how to use security metrics.