Mothra: A Large-Scale Data Processing Platform for Network Security Analysis

No ratings

Presented at Flocon 2017 by

Anthony Cebzanov

Netflow was designed to retain the key attributes of network conversations between TCP/IP endpoints on large networks without having to collect, store, and analyze all of the network's packet-level data. Over time, however, demand has increased for a platform that can support analytical workflows that make use of attributes beyond the transport layer. With the advent of template-based flow formats such as IPFIX, flow collectors are capable of collecting and exporting some of these attributes, but retaining finer-grained details of network conversations in a more flexible format has made efficient storage and analysis of this data at scale challenging. The Mothra network analysis platform, built on the Apache Spark cluster computing framework, enables scalable analytical workflows that extend beyond the limitations of conventional flow records. In this presentation, I will describe the Mothra architecture and demonstrate some of its capabilities, with a focus on how the platform can provide for increased analytical fidelity, simplified sharing of analysis techniques and results, and reduced training time for new analysts.