Sceadan – Systematic Classification Engine for Advanced Data Analysis

No ratings

Presented at OSDFCon 2014 by

Nicole Beebe

Simson Garfinkel Data type classification involves the determination of “type” (e.g. image, HTML text, video, etc.) without reference to metadata (e.g. file names, extensions, or network headers) or explicit reference to file signatures (e.g. “magic numbers”). Use cases involve classification of deleted data fragments, triage and disk profiling, search hit relevancy ranking, intrusion detection, and data loss prevention. Naïve statistical classification via machine learning has been studied by academics for over a decade, however, the research has netted varied results and failed to net any widely available open source tools. Researchers at UTSA and NPS have significantly advanced the state of the art in naïve statistical data type classification and have instantiated their findings in an open-source tool called SCEADAN. We used skillfully crafted training sets to perform statistical, n-gram based analysis to predict data type across 48 file/data/encoding types. Prediction accuracies vary across data types, but SCEADAN averages 75-90% accuracy and automatically produces confusion matrices. SCEADAN comes with a built-in model but researchers can easily train their own.