EXPERIENCES IN THREAT DATA PROCESSING AND ANALYSIS USING OPEN SOURCE SOFTWARE

No ratings

Presented at First 2017 by

Morton Swimmer

In this presentation, we show how we, the FTR Team in Trend Micro, Inc., processes our data effectively. The most important requirement was that we needed a single platform that was good enough for most datasets, under the added constraint of having to support diverse use cases, from day-to-day actor attribution to one-off extended researches. While there exists proprietary platforms for threat data analysis, we chose to use a stack based on Elasticsearch, itself based on the open source Lucene engine, and this has proven very effective. Early on, we used traditional databases, but found that they are fairly rigid in structure and require refactoring if new, unanticipated queries pop up or the data structure of the feeds drifts over time. They also don't always scale well without expensive hardware. We also experimented with various NoSQL databases, which are very promising but often lacked the upper layers of the stack that we'll get back to later in the presentation. Graph databases are very tempting as they are often an excellent fit for our data, and essentially provide total indexing, but they don't scale out as advertised and have hefty preprocessing requirements. Some data, for instance, time-series data, does not fit well to the graph model. The most important principle that guided us was the principle of 'no surprises' and 'good usability' in data labeling, i.e. the field names need to have some consistency and the values need to be always consistent in representation when there exists multiple ways of expressing a value, such as is the case with IP addresses. This guarantees that the experience a researcher gained while working on a given index remains useful when working with other datasets. We define a pipeline that comprises of data acquisition using appropriate scripts and preprocessing in Streamsets, which allows us to define the data mutations needed to homogenize the data and track data drift. The data terminates with Elasticsearch at which point a postprocessing step enriches the data, by, for example, adding geo information to IP addresses. The beauty of using Elasticsearch as a platform is the community that has sprung up around it and has already provided multiple user interfaces, from Elasticsearch's own Kibana, through Jupyter Notebooks and native scripting. As the use of Elasticsearch grows, we are able to include other Elasticsearch clusters without our organization in our search giving our researchers more reach from the same user interface. It is important to realise that this is not a solution that will ever be ideal, but it represents a way of handling most datasets we throw at it adequately well. We have been able to use if for a number of papers released in 2016. We've found this setup to be at the very least, a good start to an analysis, and at best fully adequate to all the researcher's needs.