Analyzing Large Flow Data Sets Using Modern Open-Source Data Search and Visualizaton Tools,
No ratings
Presented at Flocon 2014 by
Although the command-line and API-driven tools and libraries found in CERT’s NetSA Security Suite are extremely robust and powerful in analyzing large flow data sets, its extensive set of commands and options can be daunting, especially for a new user. Experienced analysts often combine a series of proven SiLK commands to form shell scripts and use external tools such as GnuPlot to visualize the data. Although this process provides abstraction and facilitates reuse, it can be very tedious and time-consuming, leaving an analyst with less time to dissect network traffic. Three modern open-source tools—Log stash, Elastcsearch, and Kibana—can be combined to form a web-based data visualization and analysis platform that is powerful and easy to use. Other data sources, such as web server and firewall access logs, can also be combined with flow-based records to provide both real-time and historical correlation capabilities. After an overview of the platform architecture, several real-world use cases will be demonstrated using the Lawrence Berkeley National Laboratory (LBNL) reference SiLK dataset. Comparisons will also be made to similar proprietary solutions, such as Splunk.