Cruising on a Security Data Lake: Solving Big Data Challenges in SECOPS

No ratings

Presented at TechnoSecurity&Digital 2020 by

Charles Herring

Researchers at WitFoo in conjunction with The University of Chicago and representatives from Law Enforcement, US Military and Fortune 500 organizations conducted more than 2000 controlled experiments on production networks from 2016 through 2018 to establish a Big Data pipeline for use in CyberSecurity Operations that allows for the application of investigative workflows and indicators of compromise in near realtime as well as providing for retrospective analysis of the complete data stack when new insights and indicators are made available. The first section of the session will evaluate the strengths and limitations of Big Data technologies including Elasticsearch, Splunk, Hadoop, Kafka, MySQL NDB, Cassandra, NoSQL vs RDBM as well as pipeline philosophies including streaming and batch processing. The second section will outline the specific approaches that are used in the discovered pipeline. Detailed demo and code will be provided to illustrate adaptive and retrospective parsing, event generation and data evolution. The third section will provide a demonstration of the pipeline in use to detect emerging threats and to retrospectively find threats missed historically. Upon completion of the session, attendees will understand the philosophies, components and steps in creating an effective big data pipeline that addresses the challenges in Cyber Security Operations.