Streaming Analysis: An Alternate Analysis Paradigm,

No ratings

Presented at Flocon 2014 by

Since the development of the SiLK tools early in this century NetFlow data has been a mainstay of network security analysis in DoD DHS and some commercial settings. As network traffic volumes and rates have increased complete capture and retention at the packet level has become impossible and even NetFlow is straining the ability of archiving and analysis facilities. One of the problems is that the vast majority of the traffic captures is of little or no interest from a forensic or analytic standpoint. Streaming analysis offers an alternative to capture and archive approaches. Streaming analytics perform analysis on the fly preferably close to the origin of the data where volumes may be more tractable. The analytics can be broadly based or narrowly targeted. They lend themselves to distributed computations taking advantage of inexpensive multi-core processors and cheap memory. Streaming computations must operate in a single pass over the data and can retain only limited state. This requires rethinking of some computational processes in order to avoid requirements for multiple passes and excessive state retention. In this talk we will present the results of prototype implementations of two streaming analytics initially implemented in an IBM InfoSphere Streams processing environment and currently being reimplemented in RedJack's Fathom streaming environment. One produces a statistical characterization of a packet stream producing partial statistics (counts sums sums of squares) for some 260000 variables (ports protocols subnets etc.) at one minute intervals. This can be used to create models to identify outbreaks of abnormal activity that could be used to generate alerts or control selective capture. The other is an implementation of the TRW scan detection algorithm extended to create and maintain service specific oracles that allow the detection of both horizontal vertical and mixed TCP scans ICMP scans and non protocol specific host scans including those based on UDP. This could be used to block or blacklist scanners at the border of the monitored network and partial results can be combined to detect distributed or very slow scans.