DISTRIBUTION A:
Approved for public release; distribution is unlimited.

Document created: 1 Jun 2008
Air & Space Power Journal - Summer 2008

A Practical Approach to
Effects-Based Operational Assessment

Capt Clinton R. Clark, USAF, Retired
Capt Timothy J. Cook, USAF

Effects-based operations (EBO) are "actions taken against enemy systems designed to achieve specific effects that contribute directly to desired military and political outcomes."¹ Huh? Perhaps more clearly, EBO is simply a "way of thinking" about military operations.² An effects-based approach to operations (EBAO) offers a "better way of expressing what EBO really is," and Air Force doctrine has recently adopted the term EBAO to add clarity to these concepts.³ The crux of EBAO lies in the explicit linkage of tactical actions to operational and strategic military effects. Ultimately, its goals call for the efficient and effective use of scarce resources to produce the commander's desired effects.

The joint force air component commander (JFACC) derives specified and implied tasks from the joint force commander's (JFC) guidance. Translated into the JFACC's mission, these tasks serve as the basis for determining his or her operational objectives. The JFACC utilizes the joint air and space operations center (JAOC) as the primary means of commanding and controlling the planning, execution, and assessment of operations designed to fulfill his or her objectives. Within the JAOC, the strategy division has responsibility for developing, refining, disseminating, and assessing the JFACC's air and space strategy.⁴ The operational assessment (OA) team supports the division throughout the strategy-­development process; however, it focuses primarily on "evaluating the effectiveness and efficiency" of joint air operations.⁵ In other words, the team provides joint air operational-level assessments to the JFACC. Thus, this article confines itself to OA.

Several senior Air Force leaders have shown interest in developing and refining OA methodologies and tools, believing that the service needs a sound, effects-based OA methodology to implement EBAO successfully. This article details an effects-based OA framework that emerged from a survey of existing OA techniques, an in-depth review of joint and Air Force doctrine, and consultation and collaboration with numerous strategists and war fighters.

The Evolution of
Operational Assessment

Responsible for attaining multiple operational objectives that compete for scarce air, space, and cyber resources, the JFACC makes resource-allocation decisions for each air tasking order (ATO), based on his or her assessment of the operation. Consequently, the OA team exists to help the JFACC make informed decisions. Fundamentally then, OA deals with decision making-a potentially complicated and confusing process, though one that need not rely exclusively on "gut feel."⁶ To develop and refine its OA methodologies, the JAOC can leverage a large body of decision-making techniques that have been successfully implemented across "a wide variety of situations."⁷ According to John S. Hammond, Ralph L. Keeney, and Howard Raiffa, an effective decision-making process fulfills these six criteria:

. It focuses on what is important.

. It is logical and consistent.

. It acknowledges both subjective and objective factors and blends analytical with intuitive thinking.

. It requires only as much information and analysis as is necessary to resolve a particular dilemma.

. It encourages and guides the gathering of relevant information and informed opinion.

. It is straightforward, reliable, easy to use, and flexible.⁸

All of the OA techniques in use across the ­JAOCs through mid-2006 violated two or more of these criteria. This section briefly reviews the evolution of OA and the most common practices in the field today.

Going with Your Gut

Assessing the situation is an integral component of decision making. Before a strategy division and its OA team existed, commanders relied exclusively on gut feel to guide their assessment, drawing on years of tactical experience to process all of the intelligence and mission reports and using their intuition to assess how things were going. Although producing a sound assessment depends upon such experience, the absence of an analytic approach for interpreting the data can leave room for bias and ultimately lead to bad decisions.

Adm Chester Nimitz demonstrated the shortcomings of this method when he assessed the preparatory bombardment of Iwo Jima, believing the explosive tonnage dropped by his forces "sufficient to pulverize everything on the island." The Marines, however, discovered an entirely different set of circumstances. During the bombing campaign, the Japanese actually increased the number of major defensive fortifications from 450 to over 750.⁹ By relying exclusively on his experience, Admiral Nimitz reached a conclusion exactly the opposite of reality; namely, he believed that he had rendered the island indefensible, but in reality the Japanese had substantially increased their defensive capability.

Strategy to Task

The strategy-to-task framework, a hierarchical structure, establishes a coherent chain linking tactical-level tasks all the way up to the national security strategy. Since its introduction to strategy-to-task thinking in 1989, the Air Force has widely applied it to the planning of joint air operations and is typically documenting this technique in a joint air operations plan or an air operations directive.¹⁰ In general, strategy-to-task hierarchies have focused on targets, using the following structure:

- operational objective (OO)

- tactical objective (TO)

- tactical task (TT)

- measure of performance (MOP)

Table 1 depicts a notional, admittedly incomplete, strategy-to-task hierarchy for a single OO. In general, a JFACC has multiple such objectives, each requiring a strategy-to-task hierarchy. The strategy-to-task hierarchy introduced a logical thought process into military planning and assessment activities, but it lacked a means of accurately determining the resulting effects of military operations.

The "Roll-Up" Model

A roll-up model of the strategy-to-task hierarchy served as the foundation for the first major effort to add quantitative analysis to JAOC OA, pioneered by United States Air Forces in Europe (USAFE). The logic and mathematics of this model are quite simple, the former assuming that the completion of a set of activities at one level of the hierarchy implies the completion of another at the next. For example, completing all of the TTs (destroy enemy SA-2 systems and degrade enemy SA-3 systems) implies achievement of the TO (degrade enemy SAM systems). Carrying out the TOs (degrade enemy SAM systems; degrade enemy air forces; and degrade command, control, and communication of the enemy's integrated air defense) implies meeting the OO (gain and maintain air superiority). To create a mathematical model, we assign weights to each line in the hierarchy, indicating the relative importance of each MOP, TT, and TO. Rolling up (using weighted averages) the lower-level scores, beginning with an initial value for each MOP, generates a score for each OO. We typically refer to such roll-up models as linear weighted-additive models.

Although the USAFE model made great strides within OA, it suffered from two major shortfalls. First, the logic assumes that our understanding of the enemy system matches reality. In other words, faulty intelligence combined with traditional planning approaches can lead to lower-level actions that do not roll up to complete higher-level objectives. Second, this model focuses solely on carrying out tasks in the strategy-to-task hierarchy while disregarding the key elements of the operational plan-the commander's desired effects. Not perfectly suited for assessment of EBAO, this model nevertheless provides the natural stepping-stone to methodologies that combine performance and effects in a mathematically logical, yet straightforward, approach.

Rolling Up with Effects

As EBAO spread, the joint air estimate process evolved to support its concepts. Although varying approaches exist, each JAOC has begun to transform the strategy-to-task structure into an effects-based planning and assessment tool. OA models began providing a "roll-up score" that combined both performance and effects metrics. Doing so, however, violated the major mathematical assumptions of linear weighted-additive models, often yielding meaningless results. In all cases, the OA team had to employ qualitative "override" scoring inputs. In terms of the bottom line, evaluating performance and effects metrics together broke the model, and OA teams regressed to relying on gut feel.

A Brief Discussion of Measures

Measures define the degree to which we accomplish something.¹¹ For our purposes, measures of effectiveness (MOE) define the degree to which we produce effects, and MOPs define the degree to which we have completed tasks. The use of MOEs and MOPs lets us provide unambiguous evaluations of how well we generate effects or perform tasks.¹²

The proposed assessment model takes the form of a linear weighted-additive model-sometimes called an additive utility function.¹³ Therefore, the units of measurement must be uniform: we can't add apples to oranges without first applying a mathematical transformation to equate the units. To facilitate this process, we transform apples and oranges into a normalized "value" via an individual utility function.¹⁴ That is, we transform the attributes associated with an apple or an orange into a value on the range [0, 1] based on the commander's belief system. An example appropriate to an air operations plan would equate the number of enemy fighters destroyed to the combat effectiveness of enemy ground forces. Figure 1 offers examples of individual utility functions for notional MOPs and MOEs.

As an extension of the models briefly discussed, the methodology presented in this article overcomes the mathematical limitations and enables the OA team to assess both desired operational effects and the performance of planned actions simultaneously. Its development grew out of experience in the JAOC strategy divisions along with the support of many JAOC OA personnel, EBAO experts, and Air Force senior mentors. The approach meets the six requirements for a sound decision-making tool and thus provides a clear, simple structure for conducting solid OAs.

A Practical Methodology for
Operational Assessment

Given one tenet of this article-that OA must support the effective and efficient use of air, space, and cyberspace power-OA must answer two fundamental questions: Are we doing things right? Are we doing the right things? The first question addresses the performance of planned air operations by focusing the assessment on the completion of tasks. The second considers the efficient use of scarce airpower resources by focusing on production of the JFACC's desired effects. The synergy between the answers enables the OA team to provide the commander with actionable information upon which to base decisions about the direction of the strategy. Inherent in this process is the capability to point out areas with greater operational risk-highlighting potential trade-offs for allocation decisions.

Figure 2 provides an overview of the effects-based OA process, which ties directly to the air operations plan. The "plan" should detail the JFACC's desired operational-level effects with corresponding MOEs and success indicators. In addition, it should detail the tasks the JFACC considers necessary to achieve his or her objectives as well as the corresponding MOPs for these tasks. The remainder of this article assumes the validity of the operational-planning structure of Joint Publication (JP) 5-0, Joint Operation Planning.¹⁵ To assess an air operations plan, we construct two mathematically independent models-one to evaluate fulfillment of the JFACC's desired effects and a second to evaluate the performance of the JFACC's planned tasks. "Several good reasons" exist for objectively quantifying the subjectively built plan into models.¹⁶ The primary reason: they help clarify the meaning of the effects and "[facilitate] all aspects of decisionmaking."¹⁷

Tasks, Effects, and Causal Links

We perform tasks to create effects-the necessary links to achieving objectives. While independently scoring effect and performance, OA teams must maintain task and effect linkages when making overall assessments. Separating tasks and effects may marginalize or over­emphasize one or the other and may diminish the linkage between the two, which lies at the very heart of effects-based thinking. This assessment methodology is designed to explicitly assess these linkages through the juxtaposition of effect and performance results.

In addition, when we pay attention to causal links, mathematically independent scoring models for effect and performance provide great utility since they help highlight "weight of effort" and "achievement of objective" trade-offs. This approach proves especially useful during the planning phase since it helps mitigate the dangers of assessments becoming too "fuzzy"; however, we must balance it against a desire to perform an overly quantitative assessment.

Notation

Before describing the detailed mathematics in our methodology, we would do well to introduce the notation that we will use, especially that dealing with weighting, scoring, and indexing in relation to our overview of assessment methodology (fig. 3). A w represents the relative importance weight. For example, w_(i) refers to the relative importance weighting of objective i. M, O, E, and T represent calculated scores for various plan levels: missions, objectives, effects, and tasks, respectively. Subscripts _E and _P indicate effect and performance scores, respectively. For example, O_E(i) refers to the objective-level effect score for objective i.

Effect and Performance Scoring

We assess effects and performance with two mathematically independent, linear weighted-additive models. The mathematical mechanics involve an iterative process that repeats the similar steps for each level in the model hierarchy. At the lowest levels, each effect has a number (x) of MOEs associated with it, and each task has a number (y) of MOPs associated with it. In addition, we assign each MOE and MOP a weight reflecting relative importance. For each assessment period, we observe values associated with each MOE and MOP and input them into their respective models. Figure 4 outlines the effects-scoring model; figure 5 outlines the performance-scoring model. The MOE and MOP scores, between 0 and 1, indicate the level of a particular effect or task, respectively. A score of 1 indicates complete success-at least temporarily. This holds true for all scores at each level.

Operational Assessment

The JFACC allocates resources to perform tasks, which create effects. The methodology presented gives the OA team a process to assess our performance of tasks and determine if these tasks produce the desired effects. A high score for performance suggests completion of many of the planned tasks. A high score for effect suggests achievement of many of the JFACC's desired effects. Low scores for performance and effect naturally have an opposite interpretation. Drawing inferences based on comparing the resulting scores for performance and effect represents one key to this methodology. Table 2 provides some generalized interpretations for various combinations of high and low scores for performance and effect.

We can characterize the independent scores for performance and effect in three ways: (1) similar, (2) performance > effect, and (3) effect > performance. Similar scores suggest that the operation is proceeding as expected-that our understanding of the enemy system and the causal linkages between tasks and effects appears correct. In this case, we produce effects in proportion to the level at which we carry out subordinate tasks.

Disconnects between scores for effect and performance indicate that portions of the plan may require further examination. When performance scores are higher than effect scores, the completion of tasks, to this point, has not created the desired effects. Numerous issues-including data latency, delayed effects, or a misunderstanding of the enemy system-can drive such score mismatches. For example, we may have confirmation of successful leaflet drops (performance) supporting special-operations efforts to turn the local populace against the government (effect), but due to a communications outage we cannot receive reports of civilian uprisings (MOE). In addition, we may have battle damage assessment indicating destruction of all enemy fuel storage (performance), but we won't see how it affects enemy operations (effects) for two weeks. Finally, we may have destroyed all national power production (performance) to limit enemy command and control, but because the enemy employs couriers and handheld radios as his primary means of communication, command and control remains intact (effect).

In other words, our assumptions about direct links between the achievement of objectives and their prerequisite, lower-level effects and tasks may be flawed. In fact, the OA process may prove most valuable under these conditions. In this case, OA should focus primarily on quickly identifying and recommending required changes to the plan.

Conversely, when effect scores are higher than performance scores, we have produced desired effects without the comparable completion of subordinate tasks. Numerous issues, including data latency, enemy deception, good fortune, and a misunderstanding of the enemy system could lead to these score mismatches. For example, we do not have battle damage assessment from our strikes on the enemy's strategic SAMs (performance), but he has not launched them during the last five ATOs (effect). Further, although we haven't taken any action against enemy fighters (performance), the enemy has chosen not to fly. This situation may arise simply due to the fact that the enemy has hidden these aircraft in caves; regardless, our air operations have proceeded without inhibition (effect).

In this case, our potentially mistaken assumptions about task and effect linkages may enable a reallocation of resources. Identifying these opportunities will allow the JFACC to ­execute operations more efficiently. The OA team should now focus on identifying which objectives warrant additional resources and on determining operational risk (based on remaining enemy capability) assumed by the JFACC if resources shift to other objectives. Situations of high scores for effect with low scores for performance can quickly reverse themselves, for example, if the enemy brings his aircraft out of hiding.

Where Is the Operational Art?

The process of developing an effective strategy requires "significant creativity and hard thinking" and must involve the entire strategy team, consisting of operations, intelligence, logistics, analysis, and sister-service personnel.¹⁸ Development of the plan's structure-the decomposition from missions to tasks-is an entirely qualitative process based on the experiences and judgment of strategists. Additionally, assigning weightings for relative importance and choosing success indicators, MOEs, and MOPs must be based on the knowledge and experiences of the entire strategy team.

Well-structured plans provide the basis for the use of quantitative-assessment models.¹⁹ Therefore the OA team must play a critical role in developing the air operations plan to ensure the ability to assess results accurately. But offering effective strategy recommendations requires that we view the results produced by this quantitative model in the context of the operation. At this point, the strategist's application of operational art becomes critical.

The science of this methodology generates scores, not assessments. Producing OAs requires a blend of operational art and mathematical science. The models produce scores that draw attention to areas of interest. Nevertheless, we must investigate the results for cause-effect relationships and bring into play the trained eyes of experienced strategy professionals to interpret them. The scores will highlight opportunities for recommendations to "stay the course," "change the plan," or "shift weights of effort"; ultimately, though, such decisions will emerge only after collaboration with the entire strategy team.

Where Is the Data?

Lack of data represents a real problem for all analytic OA methodologies, including this one. We find data-collection and dissemination challenges in every theater, and we must plan for them. Experience and sound judgment, already a necessary ingredient for quality assessments, increase in importance when we do not possess the required information (military intelligence, battle damage assessments, mission reports, etc.) for assessment models. The reality of limited data, however, does not relieve the OA team of its responsibilities to develop a sound assessment structure, identify intelligence and other information requirements, and conduct a sound OA.

Even in the worst cases of data deficiency, great benefits accrue to implementing an assessment methodology such as that described in this article since this "structuring . . . results in a deeper and more accurate understanding of . . . the decision context."²⁰ Further, by providing a sound analytic framework, the OA team will have a frame of reference when it discusses confidence in results. The team can couch OA results and recommendations in terms of data availability, providing the JFACC greater insight into the balance of art and science in the current assessment.

Finally, a consistent and methodical approach to OA can counter the inevitable effects of a lack of continuity in the JAOCs. Although a lack of data, combined with the constant rotation of personnel assigned to the JAOC, may seem an impenetrable barrier to sound OA, a method such as the one proposed here can reassure the JFACC that assessments and recommendations are based upon a consistent approach.

Methodology Demonstration

This section applies the OA methodology developed in the previous section to a notional example (see table 3 for a generic plan framework). Admittedly incomplete, the plan nevertheless highlights the benefits of effects-based OA. The responsibility for developing such a plan falls to the strategy division, of which the OA team is a critical component. Therefore, the team should not undertake this task alone; conversely, it must not be excluded during development of the hierarchy. Any strategy-to-task hierarchy constructed without assessment in mind from the beginning will likely contain immeasurable portions that will force assessment back into the realm of an exclusively gut feel.²¹

Now that the plan is complete, we can build the effect and performance models. Figure 6 depicts the effect-scoring model for our generic plan, including the model structure and relative importance weights for each objective, effect, and MOE. Figure 7 provides the structure of the performance-scoring model, with relative importance weights shown for each objective, effect, task, and MOP.

We can use several techniques, such as "pricing out," "swing weighting," or "lottery weights" to derive the hierarchy weights.²² A detailed discussion of these methods lies beyond the scope of this article, but it is important to note that the method chosen depends upon the personality, values, and experience of the commander-not the analyst. The method most straightforward to the commander will prove most useful in eliciting his or her true belief system.

With the structure defined and weights elicited, we can build an assessment tool. The calculations required by this methodology are rudimentary enough to be performed by hand, with a calculator, or in a simple spreadsheet model. The next section highlights the simple mathematics required to produce effect and performance scores for this notional example.

Model Calculations for Air Tasking Order "A"

This section walks the reader through the mathematical mechanics of our methodology for a sample data set. Tables 4 and 5 supply notional data for one ATO period we call "ATO A." The "Observed" column contains notional observations, and the "Value" column the resulting individual utility scores. Again, higher scores are better, with a maximum value of one.

The calculations below determine the individual effect score for the notional effect "friendly fighter operations unaffected by enemy action," using equation 3 ("individual effect scores"), the weights from figure 6, and the values from table 4. For each MOE, we multiply the assigned relative-importance weighting by its observation value from ATO A. We then sum the three MOE scores to produce the individual effect score of 0.25. As previously stated, scores are between 0 and 1; a score of 0.25 would indicate to the OA team that we have far to go to realize the desired effect.

E_E(1,1) = w_(1,1)1 MOE(1,1)1+ w _(1,1)2 MOE_(1,1)2+ w_(1,1)3 MOE_(1,1)3

E_E(1,1) = (0.6)(0.4) + (0.3)(0) + (0.1)(0.1)

E_E(1,1) = 0.24 + 0 + 0.01

E_E(1,1) = 0.25

Using inputs from figures 4 and 5, the weights from figures 6 and 7, and equations 1-7, we computed the effect and performance scores for the mission, objective, and effect levels as well as the performance scores for each task. Table 6 contains all the calculated effect and performance scores for ATO A. The next section discusses interpretation of results.

Presentation of Assessment Results

The methodology described in this article and its associated calculations are critical to producing a sound, effects-based OA. For the JFACC, however, a picture is often worth 1,000 words. Because a briefing typically presents the JFACC with the OA results, we must convey this large amount of information clearly and concisely in a short period of time, tailoring presentation techniques to the preferences of each JFACC. We offer some sample presentation options here.

For demonstration purposes, we present results for a notional subsequent ATO that we call "ATO D," which has four objectives: air superiority (AS), counterland (CL), countermaritime (CM), and information superiority (IS). In addition, we set thresholds for "stop-light charts" so that scores less than 0.3 are red, scores from 0.3 to 0.7 are yellow, and scores above 0.7 are green. We would set actual assessment thresholds through collaboration with the JFACC.

The first, and perhaps most important, assessment slide presented to the JFACC provides an overall assessment across his or her objectives. It offers a quick status of the operation; allows the JFACC to swiftly determine the progress of air, space, and cyber activities; and identifies risk areas and potential resource trade-offs between missions. Figure 8 provides a notional macro ATO D assessment across the four JFACC missions described above.

This figure clearly indicates attainment of our desired air-superiority effects early in the campaign (effect score: 0.95)-significantly more than expected, given the level of performance (performance score: 0.59) thus far. Before recommending strategy changes (such as shifting the weight of effort to other missions), the OA team should further investigate these results. Figure 9, an alternative display option, provides greater insight into air superiority. This graphic, focused on a single mission, supplies the JFACC with critical trend information, allowing quick observation of the daily progression of this objective and again reminding the JFACC that, although we can observe our desired effects, the enemy appears to retain a significant capability. Further, it seems that we have reached or are approaching a point of diminishing returns, in which continued effort applied to this objective will produce limited gains in desired effects. This presentation format additionally affords the opportunity to observe the impacts of risk-­acceptance decisions made across multiple ATOs by observing the daily interaction between effect and performance results.

To provide greater insight to the JFACC, the OA team should "peel the onion back" an additional layer. Figure 10, an in-depth look at a single air-superiority objective, permits additional insight by examining effect-performance discrepancies at the lowest levels. This "stop-light chart" highlights the cause that drives the difference in our overall effect and performance scores for air superiority. Although enemy fighters have not affected friendly fighter aircraft ("green" effect score), we have done little to degrade the adversary's fighter capability ("red" performance score).

This outcome highlights a potential disconnect in our assumed causal linkage for this task and effect, as well as an opportunity to reallocate scarce airpower resources. However, we could cite numerous explanations for this observation. The enemy may have redeployed his fighters deep within his borders-beyond our reach but available for use later (high future risk). He may have buried his aircraft in the desert, never to use them again (opportunity to reallocate resources). The enemy may be using his fighter aircraft for purposes we did not anticipate-ones that do not affect friendly fighter aircraft. However, enemy aircraft may be significantly affecting the JFC's campaign by posing a viable threat to neighboring nations (we may not understand the enemy system).

As needed, this approach allows the OA team to provide greater depth and breadth of assessment that will help the JFACC execute air operations more effectively and efficiently. Designed to support a strategist's recommendations to the JFACC, this methodology does not eliminate the need for operational art; rather, it quickly highlights areas of opportunity and risk for strategists to consider when they make recommendations to the JFACC.

Summary and
Recommendations

OA will prove useful to the JFACC if it adds to his or her understanding of the campaign's progress. By independently evaluating performance and effect, the OA methodology described in this article provides a better understanding of the relationship between the actions of friendly forces and their impact on the enemy system. Previous OA methodologies suffered from various limitations that yielded difficult-to-interpret information when events did not proceed according to plan. Comparing performance and effect enables the JFACC to determine if he or she is doing things right and doing the right things. Armed with this knowledge, the JFACC can make adjustments to the strategy as required.

Real-World Implementations

Numerous JAOCs have implemented the assessment methodology presented in this article, Seventh Air Force having done so the most completely and effectively. After the OA team demonstrated the methodology during Exercise Ulchi Focus Lens 2006, the chief of the strategy division and the JFACC issued immediate guidance to adopt it. Following the exercise, the division chief focused his strategy rewrite conference on "planning for assessment," fully implementing the methodology in support of his strategy-development process for Seventh Air Force's primary war plan. The Seventh's Reception, Staging, Onward-Movement, and Integration 2007 exercise validated the value of this methodology.

Thirteenth Air Force modified this approach to assess steady-state operations. The current pace of operations is such that the commander's decision brief (including OA reporting) occurs weekly. Due to the relatively low operations tempo, OA team members found that assessing performance on a weekly basis was a straightforward matter, but discerning changes in effects from week to week proved extremely difficult. To address this situation, they applied a similar approach to the one described in this article, separating the assessment of effects and performance. Under the new approach, the team briefs its assessment of performance to the JAOC commander each week. To accurately assess changes in effects, the OA team examines them over a longer time span (generally 60-90 days), thus providing the commander with a longer-term look at each objective while still allowing sufficient time for the changes in effect to become apparent.

Deployed analysts in Ninth Air Force implemented a similar approach in late 2005. The OA team assessing Operations Iraqi Freedom and Enduring Freedom struggled to provide the commander with an effects-based assessment of his objectives. In that case, team members decided to limit themselves to assessing performance, leaving the assessment of effects to the supported command, who briefed this to the deputy combined force air component commander along with a performance assessment conducted by the OA team.²³

First Air Force's strategy division adopted the methodology presented here in 2006, during development of the joint air operations plan for Defense Support to Civil Authorities, designed to provide guidance for joint air operations during events similar to Hurricane Katrina. Exercise Ardent Sentry stressed this plan, and the OA methodology proved successful in supporting JFACC decision making during the exercise.

Applied across multiple theaters for a wide variety of operations, this methodology has supported homeland-defense scenarios as well as the development and exercising of strategy for a major theater war; a modified version has supported steady-state operations. However, we still have room to improve this approach.

The Way Ahead and Future Research
Recommendations

The way ahead for OA calls for adopting a standard methodology across the JAOCs. Although each JFACC faces unique issues, a core set of assessment processes exists. We developed this methodology to support the core OA needs of the JFACCs while offering the flexibility needed to address their unique, area-specific issues. The first practical benefit of adopting a standardized approach would involve rapid methodological improvements arising from the inevitable collaboration across JAOCs.

The first step to establishing standard tools and training entails adopting a standard OA methodology. By developing a standard set of tools, we can reduce the workload of the OA teams' chiefs by eliminating the need to develop and maintain their own tools. Further, we could link a standard set of tools to the backbone of JAOC software-Theater Battle Management Core Systems or its successors-potentially automating much of the data-collection effort. Currently, the collection and input of relevant data make for a very labor-intensive process for OA teams, reducing the time they have for interacting with the strategy division during development and refinement of courses of action. Additionally, each team requires augmentation during contingency operations. A standard OA methodology would enable us to provide initial qualification training for OA augmentees, minimizing the "pickup game" approach to assessing operations. This training would certainly incorporate the use of a standardized tool set, enabling deployed OA team members to contribute to strategy and assessment quickly during contingency operations.

Initially, future research efforts should focus on methodology. Any assessment faces the problem of missing data-a major issue addressed by many existing statistical approaches. JAOCs can exploit these techniques to enable better assessments. Gaining insight into causal linkages, during both planning and execution, is a growth area for strategy and OA. Strategists often use the term assumed causal linkages because they develop them based on limited, often biased, understanding of the enemy system. By assessing operations according to the methodology described in this article, we could use the raw results to develop causal relationships between our performance and effect results. That is, we could correlate the completion of our tasks with the achievement of our desired effects. Further, we could employ numerous statistical techniques, such as canonical correlation, neural networks, and logistic regression, to add greater understanding to our causal linkages.

*The authors would like to express their sincere thanks to Mr. Doug Lee, Lt Col Marc Jameson, Lt Col Kirsten Messer, Lt Col John Schaefer, Maj Maurice Azar, Maj Steve Cox, Maj Stewart Greathouse, Maj Alan Kastner, Maj Joseph Morgan, Maj Patrick Ritchie, and Maj Christopher Solo. Their advice, commentary, witticisms, and criticisms were invaluable.

[Feedback? Email the Editor ]

Notes

1. Air Force Doctrine Document (AFDD) 1, Air Force Basic Doctrine, 17 November 2003, 98, https://www. doctrine .af.mil/afdcprivateweb/AFDD_Page_HTML/Doctrine_Docs/afdd1.pdf.

2. Lt Gen David A. Deptula, “Effects Based Operations,” Air and Space Power Journal 20, no. 1 (Spring 2006): 4, http://www.airpower.maxwell.af.mil/airchronicles/apj/apj06/spr06/spr06.pdf.

3. J. P. Hunerwadel, “The Effects-Based Approach to Operations: Questions and Answers,” Air and Space Power Journal 20, no. 1 (Spring 2006): 53–62; and Headquarters Air Force Doctrine Center, Doctrine Watch no. 26, “The Effects-Based Approach to Operations,” 31 January 2007, https://www.doctrine.af.mil/ Main.asp.

4. Air Force Instruction (AFI) 13-1AOC, vol. 3, Operational Procedures—Air and Space Operations Center, 1 August 2005, 14, http://www.e-publishing.af.mil/shared/media/epubs/AFI13-1AOCV3.pdf.

5. Ibid., 19.

6. Craig W. Kirkwood, Strategic Decision Making: Multiobjective Decision Analysis with Spreadsheets (Belmont, CA: Duxbury Press, 1997), 1.

7. Ibid.

8. John S. Hammond, Ralph L. Keeney, and Howard Raiffa, Smart Choices: A Practical Guide to Making Better Decisions (Boston: Harvard Business School Press, 1999), 4.

9. James Bradley with Ron Powers, Flags of Our Fathers (New York: Bantam Books, 2000), 135.

10.Glenn A. Kent, A Framework for Defense Planning (Santa Monica, CA: RAND Corporation, 1989), https://rand.org/pubs/reports/2006/R3721.pdf.

11. Ralph L. Keeney, Value-Focused Thinking—A Path to Creative Decisionmaking (Cambridge, MA: Harvard University Press, 1992), 100.

12. Kirkwood, Strategic Decision Making, 25. A thorough discussion of measurement theory lies beyond the scope of this article; however, interested readers can consult Maj Richard K. Bullock, “Theory of Effectiveness Measurement” (PhD diss., Air Force Institute of Technology, Wright-Patterson AFB, OH, 2006).

13. Robert T. Clemen and Terence Reilly, Making Hard Decisions with Decision Tools (Belmont, CA: Duxbury Press, 2001), 100.

14. Ibid., 610.

15.Joint Publication (JP)5-0, Joint Operation Planning, 26 December 2006, III-13, III-60, http://www.dtic. mil/doctrine/jel/new_pubs/jp5_0.pdf.

16. Keeney, Value-Focused Thinking, 129.

17. Ibid.

18. Ibid., 56.

19. Ibid.

20. Ibid., 69.

21. For a cogent discussion of the characteristics of well-written objectives, effects, tasks, MOEs, MOPs, and success indicators, see Maj John J. Schaefer III and Maj Alan D. Kastner, “Operational Assessment of Effects Based Strategies: Proposed Approaches to Measurement Assessment in Effects Based Operations: Tools for Effects Based Assessment” (presentation, 12th International Command and Control Research and Technology Symposium, Hyatt Regency Hotel, Newport, RI, 19–21 June 2007), http://www.dodccrp.org/events/12th_ICCRTS/CD/html/
papers/160.pdf.

22. Clemen and Reilly, Making Hard Decisions, 614–20.

23. Maj Kirsten Messer and Maj Shane Dougherty, “A New Operational Assessment Paradigm: Splitting the Stoplights,” Air and Space Power Journal 20, no.3 (Fall2006): 65–68, http://www.airpower.maxwell.af.mil/ airchronicles/apj/apj06/ fal06/Fal06.pdf.

Contributors

Capt Clinton R. Clark, USAF, Retired (BS, MEd, Lamar University; MS, Air Force Institute of Technology), is a graduate student at Rice University in Houston, Texas. He previously served as current operations analyst at Headquarters Air Combat Command A9 (Studies, Analyses, Assessments, and Lessons Learned Division), responsible for developing and implementing operational assessment techniques; continuous process improvements; and identification, validation, resolution, and dissemination of lessons learned across the combat air forces. He has supported operational assessments for First, Seventh, Eighth, Thirteenth, and Fourteenth Air Forces. Captain Clark, a distinguished graduate of the Air Force Institute of Technology, is the 2005 Air Force Company Grade Analyst of the Year.

Capt Timothy J. Cook (USAFA; MS, Air Force Institute of Technology) is an executive officer at Headquarters Air Force A9 (Studies, Analyses, Assessments, and Lessons Learned). He previously served as the lead action officer for the combat analyst program at Headquarters Air Force, responsible for developing and guiding implementation of operational assessment models, procedures, and tools. He has mentored operational planners and assessors as they developed operational assessment plans for First, Seventh, Twelfth, and Fourteenth Air Forces. Additionally, he served on operational assessment teams supporting Operation Noble Eagle and Exercises Ulchi Focus Lens; Terminal Fury; Reception, Staging, Onward-Movement, and Integration; and Ardent Sentry. Captain Cook graduated in the top third of his class at Squadron Officer School.

Disclaimer

The conclusions and opinions expressed in this document are those of the author cultivated in the freedom of expression, academic environment of Air University. They do not reflect the official position of the U.S. Government, Department of Defense, the United States Air Force or the Air University

[ Home Page | Feedback? Email the Editor]