Rapid Mission Assurance Assessment via Sociotechnical Modeling and Simulation Michael Jay ...

375 Pages·2015·8.55 MB·English

Rapid Mission Assurance Assessment via
Sociotechnical Modeling and Simulation
Michael Jay Lanham
CMU-ISR-15-104
May 2015

School of Computer Science
Institute of Software Research
Carnegie Mellon University
Pittsburgh, PA

Thesis Committee
Kathleen M. Carley (Chair)
Virgil D. Gligor
Jürgen Pfeffer
Robert Elder (George Mason University)
John Graham (United States Military Academy)
Submitted in partial fulfillment of the requirements
for the Degree of Doctor of Philosophy
Copyright © Michael J. Lanham

This work was supported in part by the Office of Naval Research MURI N000140811186, a
joint grant from the National Security Agency and Army Research Office (ARO) under
grants W911NF1310154, the Center for Computational Analysis of Social and
Organizational Systems (CASOS), and the United States Military Academy. The views and
conclusions contained in this document are those of the author and should not be interpreted
as representing the official policies, either expressed or implied, of the Office of Naval
Research, the National Security Agency, the Army Research Office, the US Army, or the U.S.
Government.
Keywords: Mission Assurance, Resilience, Assessment, Cyberspace Operations, Cyber,
Organization Resilience, Rapid Modeling, Agent Based Model, ABM, Simulation,
Modeling and Simulation
Dedicated to my wife, my children, and my country

Acknowledgements
It has been my great fortune to work with and for an outstanding dissertation
committee in Kathleen M. Carley, Virgil D. Gligor, Jürgen Pfeffer, Robert Elder (Lieutenant
General, US Air Force, Retired) and John Graham (Colonel, US Army). I am particularly
indebted to Kathleen for her willingness to take on yet another Army officer, constrained to a
very challenging Army time line. I have insufficient words or eloquence to thank everyone
adequately, as I know my wife, my children, my family and my friends have missed me these
last years.
Throughout this latest journey and on the paths to it, so many people have helped me
in more ways than I can enumerate. I have been incredibly fortunate to have so many good
people touch my life! Nevertheless, I offer my thanks to the supporting staff, administrators,
and developers in the Center for Computational Analysis of Social and Organizational
Systems (CASOS). I also offer special thanks to Geoffrey P. Morgan for all the many hours
of brain storming, selfless assistance, rapid prototyping, tool development, conducting
analysis, and being a friend! Many thanks also to Kenneth ‘Kenny’ Joseph for the hours he
spent in developing, configuring, and debugging Construct, and helping me do the same. To
Peter Landwehr, Wei Wei, and my other fellow doctoral seeking peers in CASOS, I offer my
thanks!
The support I have received from the leadership and faculty at the United States
Military Academy (USMA). Department of Electrical Engineering and Computer Science
(D/EECS ) has also been superb. The energy and effort they have contributed to me finishing
this trek continues to humble me and generate my gratitude. From covering teaching and
grading duties, simple words of encouragement, and forcing me to simply close my door to
get work done, my friends and colleagues have been extremely helpful. Thank you also to Dr.
Chris Okasaki, for his particular gift to me in being a volunteer editor of this written work.
All errors in this text are despite his efforts at helping me be a clearer writer.

Abstract
How do organizations rapidly assess command-level effects of cyber attacks? Leaders
need a way of assuring themselves that their organization, people, and information
technology can continue their missions in a contested cyber environment. To do this, leaders
should: 1) require assessments be more than analogical, anecdotal or simplistic snapshots in
time; 2) demand the ability to rapidly model their organizations; 3) identify their
organization’s structural vulnerabilities; and 4) have the ability to forecast mission assurance
scenarios. Using text mining to build agent based dynamic network models of information
processing organizations, I examine impacts of contested cyber environments on three
common focus areas of information assurance—confidentiality, integrity, and availability. I
find that assessing impacts of cyber attacks is a nuanced affair dependent on the nature of the
attack, the nature of the organization and its missions, and the nature of the measurements.
For well-manned information processing organizations, many attacks are in the nuisance
range and that only multipronged or severe attacks cause meaningful failure. I also find that
such organizations can design for resiliency and provide guidelines in how to do so.

Table of Contents
Acknowledgements ............................................................................................................. v
Abstract ............................................................................................................................. vii

Introduction ......................................................................................................................... 1

Thesis Statement ............................................................................................................. 1

Scope ............................................................................................................................... 3

Definitions....................................................................................................................... 4

Why is this important? .................................................................................................... 8

What will this dissertation do? ...................................................................................... 10

Literature Review.............................................................................................................. 11

Introduction ................................................................................................................... 11

Related Areas of Research ............................................................................................ 16

Related literature corpus and assessment ...................................................................... 32

Conclusions ................................................................................................................... 67

Data and Models ............................................................................................................... 70

Introduction ................................................................................................................... 70

Organization—a working definition ............................................................................. 71

Organizations’ Self-Documentation ............................................................................. 72

The Data to Model Process—an Overview .................................................................. 73

Two D2M generated DoD organizational models ........................................................ 77

Changes to empirical models in support of Agent Based Modeling .......................... 115

Conclusions ................................................................................................................. 116

Network Analytics and Resilience .................................................................................. 118

Resilience in what context? ........................................................................................ 119

Static resilience indicators .......................................................................................... 121

Metanetwork resilience indicators .............................................................................. 122

New and adjusted metanetwork resilience indicators ................................................. 123

Resilience indicators for strategic and operational models......................................... 132

Entropic and Targeted Attacks .................................................................................... 150

Conclusions ................................................................................................................. 166

Agent Based Models and Modeling ................................................................................ 169

Agent based models and sociotechnical systems ........................................................ 170

Agent based models and cyber security research ....................................................... 171

Overview of Construct ................................................................................................ 172

Augmentation of D2M generated models ................................................................... 175

Experimental Design Setup......................................................................................... 191

Conclusion .................................................................................................................. 197

Simulations ..................................................................................................................... 198

Changes to Construct .................................................................................................. 198

Analysis....................................................................................................................... 203

Omitting New Resilience Metric ................................................................................ 203

Mitigations .................................................................................................................. 212

Heuristics ........................................................................................................................ 217

Limitations and Future Work .......................................................................................... 223

Text-mining as basis of organizational modeling ....................................................... 223

i
Limitations of doctrine and written products as the basis of models .......................... 223
Cyber effects vs. methods-of-attack ........................................................................... 225

Modeling and Simulations .......................................................................................... 225

Contributions................................................................................................................... 231

Insights and Surprises ................................................................................................. 233

References ....................................................................................................................... 237

Appendix 1 Definitions .................................................................................................... 1-1

Central Definitions ....................................................................................................... 1-1

Alphabetical Definitions .............................................................................................. 1-7

Appendix 2 Literature Review Bibliometrics .................................................................. 2-1

Introduction .................................................................................................................. 2-1

Process ......................................................................................................................... 2-1

Searches and Search Results ........................................................................................ 2-1

Pre-processing Collected Data ..................................................................................... 2-3

Importing Collected Data into ORA™ ........................................................................ 2-3

Import Node Sets and Networks .................................................................................. 2-4

Cleaned Article node set ............................................................................................ 2-10

Cleaned Author node set ............................................................................................ 2-10

Cleaned Concept node set .......................................................................................... 2-10

Manipulate Networks ................................................................................................. 2-10

Appendix 3 Data to Model Implementation Details ........................................................ 3-1

Introduction .................................................................................................................. 3-1

Retrieving input corpus ................................................................................................ 3-1

Corpus augmentation ................................................................................................... 3-5

Retrieving input corpus ................................................................................................ 3-6

Pre-processing DoD corpus ......................................................................................... 3-6

Metanetwork encoding heuristics for thesaurus refinement ........................................ 3-8

Using frequency as culling decision input variable ................................................... 3-11

Appendix 4 Virtual Experiment How-To Guide ............................................................. 4-1

SVN or other Shared-work Repository ........................................................................ 4-1

Create a Directory Structure ........................................................................................ 4-1

Input Files .................................................................................................................... 4-3

Updating Executables sent to the Condor cluster ........................................................ 4-7

Perl on submitting machine ......................................................................................... 4-8

Directories for Condor Virtual Experiment ................................................................. 4-9

Submitting to Condor for the Virtual Experiment ....................................................... 4-9

Post-Processing Outputs of Condor from a Virtual Experiment ............................... 4-10

Using R for Graph Generation ................................................................................... 4-14

Appendix 5 Construct Input Files .................................................................................. 5-15

Parameters files .......................................................................................................... 5-15

Experimental Configuration File (Box-Behnken implementation) .............................. 16

Appendix 6 Construct input deck for operational and strategic simulation ..................... 6-1

Appendix 7 Additional Model Analysis, Tables, and Figures ......................................... 7-1

Descriptive Statistics of Operational Model, Attacks, and Mitigations ....................... 7-1

Appendix 8 Lists of Tables, Figures, Equations and Acronyms...................................... 8-1

List of Tables ............................................................................................................... 8-1

ii