Lessons Learned in Growing a Big Data Capability for Network Defense

No ratings

Presented at flocon 2018 by

Steve Wagner

The advent of big data and data science presents tremendous opportunities for cyber operations and network security. These opportunities are in great demand within the United States Department of Defense (DoD) with its high operational tempo, sensitive global mission set, and massive inventory of networked systems. For the past three years, the Department of Defense Information Systems Agency (DISA), United States Army, and US Cyber Command have rapidly pioneered development of a state-of-the-art, full stack Big Data Platform to meet this demand. This capability is currently supporting network defense and cyber operations around the globe via multiple Petabyte-scale instances running on hundreds of scalable cloud-based compute nodes. Most importantly, it exists as a government-owned platform architecture built on open-source technology with the express goal of integrating contributions from defense agencies, national labs, industry partners, and academia. This vision is the result of leading-edge DoD thinkers and decision makers who recognized the promise of big data and acted boldly. In this talk, we will discuss the evolution of the Big Data Platform, examples of how it is being used today, and key lessons learned in its development.