Preface xxiii
Introduction xxv
CHAPTER 1 An EROM Primer for Organizations Concerned with Technical Research, Integration, and Operations (TRIO Enterprises) 1
1.1 EROM Scope and Objectives for TRIO Enterprises 1
1.1.1 What Is EROM? 1
1.1.2 Why Is EROM Important to TRIO Enterprises? 2
1.1.3 What Kinds of Risk and Opportunity Are Considered within EROM for TRIO Enterprises? 3
1.1.4 How Does EROM for Nonprofit and Government TRIO Enterprises Differ from EROM for Typical Commercial Enterprises? 4
1.1.5 To What Extent Does EROM Work within the Existing Management Structure of a TRIO Enterprise? 5
1.1.6 How Does EROM Facilitate Negotiations between a TRIO Enterprise and the Entities That Provide Funding and Governance? 6
1.1.7 Can Various Management Units within the Organization Separately Apply EROM as Though Each Were an Enterprise? 7
1.1.8 In What Areas Does EROM Facilitate Strategic Planning, Implementation, and Evaluation of Performance for TRIO Enterprises? 8
1.2 EROM Definitions and Technical Attributes for TRIO Enterprises 9
1.2.1 What Is Meant by Risk and Opportunity within the Context of EROM? 9
1.2.2 How Do We Differentiate between Risks and Opportunities during Strategic Planning versus during Plan Implementation and Performance Evaluation? 11
1.2.3 How Does EROM Help Achieve an Optimal Balance between Risk and Opportunity? 11
1.2.4 What Is Meant by the Terms Risk Scenario, Opportunity Scenario, Cumulative Risk, and Cumulative Opportunity? 13
1.2.5 How Does EROM Incorporate Risk-Informed Decision Making and Continuous Risk Management within the rganization as a Whole and within Different Management Units? 14
1.2.6 Is the Analysis in EROM Principally Qualitative or Quantitative? 16
1.2.7 Can EROM Account for Unknown and Underappreciated (UU) Risks? 17
Notes 18
References 19
CHAPTER 2 Coordination of EROM with Organizational Management Activities 21
2.1 The Executive, Programmatic, and Institutional/Technical Management Functions and Their Interfaces 21
2.2 EROM-Relevant Management Activities 23
2.2.1 Activities within Each Management Level 23
2.2.2 Roles and Responsibilities within and between Each Management Level 26
2.3 Coordination of EROM with Management Activities 31
2.3.1 Organizational Planning and Plan Implementation 31
2.3.2 Evaluation of Organizational Performance and Replanning 31
2.3.3 Alignment with Management-Level Roles and Responsibilities 35
2.4 Communication across Extended Partnerships 35
2.4.1 Nature of the Strategic Objectives That Require Extended Partnerships 35
2.4.2 The Challenges of Conducting EROM across Extended Partnerships 42
2.5 Contribution of EROM to Compliance with Federal Regulations and Directives 43
2.5.1 OMB Circular A-11 and GPRAMA (Government Performance, Results, and Budgeting) 43
2.5.2 EROM and Internal Controls from the Viewpoint of Federal Regulations and Guidance 45
2.5.3 OMB Circular A-123 (Management’s Responsibility for ERM and Internal Control) and the Required Statement of Assurance 47
2.5.4 Example Risk Profile from OMB Circular A-123 49
Notes 52
References 52
CHAPTER 3 Overview of EROM Process and Analysis Approach 55
3.1 Organizational Objectives Hierarchies 55
3.1.1 Objectives Hierarchies for Each Management Unit 55
3.1.2 Objectives Hierarchy for the Enterprise as a Whole 57
3.2 Populating the Organizational Objectives Hierarchies with Risk and Opportunity Information 61
3.3 Establishing Risk Tolerances and Opportunity Appetites 63
3.3.1 Risk and Opportunity Parity Statements 63
3.3.2 Response Boundaries and Watch Boundaries 65
3.4 Identifying Risk and Opportunity Scenarios and Leading Indicators 66
3.4.1 Risk and Opportunity Taxonomies 67
3.4.2 Risk and Opportunity Scenario Statements 68
3.4.3 Risk and Opportunity Scenario Narratives 72
3.4.4 Risk and Opportunity Leading Indicators 73
3.4.5 Leading Indicators of Unknown and Underappreciated (UU) Risks 74
3.5 Specifying Leading Indicator Trigger Values and Evaluating Cumulative Risks and Opportunities 78
3.5.1 Leading Indicator Trigger Values 80
3.5.2 Cumulative Risks and Opportunities 80
3.6 Identifying and Evaluating Risk Mitigation, Opportunity Exploitation, and Internal Control Options 82
3.6.1 Deducing Risk and Opportunity Drivers 82
3.6.2 Deducing Risk and Opportunity Scenario Drivers 83
3.6.3 Evaluating Risk and Opportunity Scenario Likelihoods and Impacts 85
3.6.4 Identifying Options for Risk Response, Opportunity Action, and Internal Control 87
3.6.5 Evaluating Options for Risk Response, Opportunity Action, and Internal Control 89
3.6.6 Brief Comparison of this Approach with the COSO Internal Control Framework and the GAO Green Book 91
Notes 94
References 94
CHAPTER 4 The Development and Utilization of EROM Templates for Performance Evaluation and Strategic Planning 97
4.1 Overview 97
4.2 Demonstration Example: The NASA Next-Generation Space Telescope as of 2014 99
4.3 Example Objectives Hierarchies 101
4.3.1 Objectives Hierarchies for Different Management Levels 101
4.3.2 Integrated Objectives Hierarchies for the Enterprise as a Whole 103
4.4 Risks, Opportunities, and Leading Indicators 103
4.4.1 Known Risk and Opportunity Scenarios 105
4.4.2 Cross-Cutting Risks and Opportunities 105
4.4.3 Unknown and Underappreciated Risks 112
4.5 Example Templates for Risk and Opportunity Identification and Evaluation 113
4.5.1 Risk and Opportunity Identification Template 113
4.5.2 Leading Indicator Evaluation Template 113
4.6 Example Templates for Risk and Opportunity Roll-Up 126
4.6.1 Objectives Interface and Influence Template 126
4.6.2 Known Risk Roll-Up Template 126
4.6.3 Opportunity Roll-Up Template 144
4.6.4 Composite Indicator Identification and Evaluation Template 147
4.6.5 UU Risk Roll-Up Template 151
4.7 Example Templates for the Identification of Risk and Opportunity Drivers, Responses, and Internal Controls 159
4.7.1 Risk and Opportunity Driver Identification Template 159
4.7.2 Risk and Opportunity Scenario Likelihood and Impact Evaluation Template 161
4.7.3 Risk Mitigation, Opportunity Action, and Internal Control Identification Templates 161
4.7.4 High-Level Display Template 165
4.8 Upward Propagation of Templates for Full-Scope EROM Applications 165
4.8.1 Scope of the Problem 165
4.8.2 Propagation of Templates 173
4.8.3 Development of an Integrated EROM Database 175
4.9 Application of the Templates to Organizational Planning and the Selection from among Alternative Candidate Portfolios 175
Notes 181
References 181
CHAPTER 5 Management and Implementation of EROM at the Institutional/Technical Level (Technical Centers or Directorates) 183
5.1 EROM from a Technical Center’s Perspective 183
5.2 Extended Enterprises and the Technical Center’s Extended Organization 184
5.2.1 Overview 184
5.2.2 Relationship of Each Technical Center to the Other Entities in the Center’s Extended Organization 187
5.2.3 EROM Organizational Structure for a Technical Center’s Extended Enterprises 189
5.2.4 Challenges of Creating and Managing an Integrated Database 191
5.3 EROM-Informed Budgeting of Resources across a Technical Center’s Extended Organization 192
5.3.1 Objectives-Based Distribution of Human, Physical, and Instructional Assets 192
5.3.2 Representative Templates for Distributions of Allocated Assets 192
5.3.3 Asset Risks, Opportunities, and Risk/Opportunity Scenario Statements 198
5.3.4 Leading Indicators of a Technical Center’s Health 200
5.3.5 Correlations between Internal Leading Indicators and Gaps in the Distributions of Human, Physical, and Instructional Assets 201
5.3.6 Optimization of the Acquisition, Allocation, and Retirement of Human, Physical, and Instructional Assets 203
5.3.7 Relevance to Provider Acquisition Decisions Made by Technical Centers 206
References 206
CHAPTER 6 Special Considerations for EROM Practice and Analysis at Commercial TRIO Enterprises 207
6.1 Overview 207
6.2 Risk and Opportunity Scenarios and Leading Indicators 210
6.2.1 Risk and Opportunity Taxonomies 210
6.2.2 Risk and Opportunity Branching Events and Scenario Event Diagrams 210
6.2.3 Risk and Opportunity Templates 215
6.2.4 Risk and Opportunity Matrices 221
6.3 Controllable Drivers, Mitigations, Actions, and Internal Controls 229
CHAPTER 7 Examples of the Use of EROM Results for Informing Risk Acceptance Decisions 237
7.1 Overview 237
7.2 Example 1: DoD Ground-Based Midcourse Missile Defense in the 2002 Time Frame 238
7.2.1 Background 238
7.2.2 Top-Level Objectives, Risk Tolerances, and Risk Parity 239
7.2.3 Risks and Leading Indicators 242
7.2.4 Leading Indicator Trigger Values 244
7.2.5 Example Template Entries and Results 247
7.2.6 Implications for Risk Acceptance Decision Making 247
7.3 Example 2: NASA Commercial Crew Transportation System as of 2015 249
7.3.1 Background 249
7.3.2 Top-Level Objectives, Risk Tolerances, and Risk Parity 251
7.3.3 Remainder of Example 2 253
7.4 Implication for TRIO Enterprises and Government Authorities 254
References 254
CHAPTER 8 Independent Appraisal of EROM Processes and Results to Assure the Adequacy of Internal Controls and Inform Risk Acceptance Decisions 255
8.1 Background 255
8.1.1 OMB Motivation 255
8.1.2 Department of Energy Guidance 256
8.1.3 Institute of Internal Auditors Guidance 257
8.2 Queries for an Independent Appraisal of EROM in the Contexts of Internal Control and Risk Acceptance 258
8.2.1 Overview 258
8.2.2 Template for Evaluating EROM Process and Results 259
References 265
CHAPTER 9 Brief Overview of the Potential Integration of EROM with Other Strategic Assessment Activities 267
9.1 Technical Capability Assessment (TCA) 267
9.2 Strategic Annual Review (SAR) 270
9.3 Portfolio Performance Review (PPR) 271
References 274
CHAPTER 10 An Integrated Framework for Hierarchical Internal Controls 275
10.1 Internal Control Principles and the Integration of Internal Control, Risk Management, and Governance 275
10.2 Methodological Basis 280
10.2.1 Hierarchical Control Loops 280
10.2.2 RACI Matrices 282
10.3 Examples 285
10.3.1 Example 1: Institutional Responsibility for Risk Management and System Safety 285
10.3.2 Example 2: NASA Commercial Crew Program Risk-Based Assurance Process and Shared Assurance Model 287
10.4 Incorporation of Internal Control Principles into the Control Loop Approach 297
10.5 Summary of Observations 302
References 306
APPENDIX A Acronyms 309
APPENDIX B Definitions 311
About the Companion Website 314
About the Author 315
Index 317
