Transmogrifying Other People's Marketing into Threat Hunting Treasures Using Machine Learning Magic
No ratings
Presented at BlueHatIL 2019 by
As the practice of threat intelligence has matured, machine readable feeds from home-grown OSINT systems or third-party vendors are common. These usually contain IOCs or IOAs with little contextual information. On the other hand, we have rich data describing adversary operations and TTPs contained in blogs and whitepapers that remain unstructured, and require human processing. Here we are bridging that gap by using Natural Language Processing for extracting STIX-like entities from unstructured text.This presentation will cover the use of "Custom Entity Extraction" techniques from Natural Language Processing to extract this information from text. Named Entity Extraction/Recognition is a sub-task of information extraction that aims to classify phrases into pre-defined categories (e.g. Threat Actor, Malware Family, attack technique etc.). This is usually a preprocessing step for other more complex tasks like identifying aliases, relationship extraction between actors and TTPs etc.This talk will describe our solution for building an entity extraction system from unstructured text specific to the security domain using open source tools; architecture, sources of ground truth, algorithms that worked (and those that didn't) and what problems you can solve using this pipeline.