Siambabala Bernard Manyena’s 2006 paper entitled “The Concept of Resilience Revisited” (Disasters, Vol. 30, No. 4, pp. 433-450, doi:10.1111/j.0361-3666.2006.00331.x) provided a nice summary of alternative definitions for the word “resilience” gleaned from a variety of academic publications (copied below; see original paper for citations). The number of definitions are fewer than that for the word vulnerability as talked about in my previous post.
Wildavsky (1991) Resilience is the capacity to cope with unanticipated dangers after they have become manifest, learning to bounce back.
Holling et al., (1995) It is the buffer capacity or the ability of a system to absorb perturbation, or the magnitude of disturbance that can be absorbed before a system changes its structure by changing the variables.
Horne and Orr (1998) Resilience is a fundamental quality of individuals, groups and organisations, and systems as a whole to respond productively to significant change that disrupts the expected pattern of events without engaging in an extended period of regressive behaviour.
Mallak (1998) Resilience is the ability of an individual or organisation to expeditiously design and implement positive adaptive behaviours matched to the immediate situation, while enduring minimal stress.
Miletti (1999) Local resiliency with regard to disasters means that a locale is able to withstand an extreme natural event without suffering devastating losses, damage, diminished productivity, or quality of life without a large amount of assistance from outside the community.
Comfort (1999) The capacity to adapt existing resources and skills to new systems and operating conditions.
Paton, Smith and Violanti (2000) Resilience describes an active process of self-righting, learned resourcefulness and growth—the ability to function psychologically at a level far greater than expected given the individual’s
capabilities and previous experiences.
Kendra and Wachtendorf (2003) The ability to respond to singular or unique events.
Cardona (2003) The capacity of the damaged ecosystem or community to absorb negative impacts and recover from these.
Pelling (2003) The ability of an actor to cope with or adapt to hazard stress.
Resilience Alliance (2005) Ecosystem resilience is the capacity of an ecosystem to tolerate disturbance without collapsing into a qualitatively different state that is controlled by a different set of processes. A resilient ecosystem can withstand shocks and rebuild itself when necessary. Resilience in social systems has the added capacity of humans to anticipate and plan for the future.
UNISDR (2005) The capacity of a system, community or society potentially exposed to hazards to adapt, by resisting or changing in order to reach and maintain an acceptable level of functioning and structure. This is determined by the degree to which the social system is capable of organising itself to increase this capacity for learning from past disasters for better future protection and to improve risk reduction measures.
I just got word of a report summarizing a recent workshop held in McLean, VA focused on the “simple” question “why have we not been attacked again” since 9/11? The workshop was sponsored by the Defense Threat Reduction Agency (DTRA) in cooperation with Science Applications International Corporation (SAIC). The cover page of the report is shown in the Scribd window below. Per the dissemination control markings, this product is available to the public release.
Basically, this report examines a total of 29 well-defined hypotheses that address the question at issue, namely reasons for the non-occurrence of another attack (a challenging question indeed). These hypotheses were the compiled as part of an extensive literature review centered on publications that offered reasons why he haven’t experienced a successful attack. The fact that cells were foiled, according to the authors, was taken to mean that attacks were attempted, but were unsuccessful (a summary of failed attacks is provided in Appendix C of the report). Using the information collected during this literature review, a comprehensive list of critical assumptions, supporting evidence and contradictory evidence was compiled for each hypothesis. The goal for the workshop was to identify the likely reasons for non-occurrence of an attack based on all available evidence and assumptions combined with the opinions of subject matter experts.
The 29 hypotheses were sorted into four bins as follows, two for each of capabilities and motivations (i.e., the two variables that comprise the DoD definition of threat). Note that these hypotheses are NOT mutually exclusive nor are they collectively, and as stated in the report, it is quite possible that one or more (or perhaps none) of these hypothesis contribute to the fact that the US hasn’t experienced a successful attack in the last 7 years.
Bin I: US and Allied Counterterrorism Efforts (Capabilities part of Threat)
Hypothesis A: US homeland security efforts
Hypothesis B: US and allied counterterrorism operations
Hypothesis C: The wars in Iraq and Afghanistan have drawn jihadists away
Hypothesis D: Reduced state support for terrorism
Hypothesis E: Crackdowns on private financing of terrorism since 9/11
Bin II: Terrorist Attack Capabilities (Capabilities part of Threat)
Hypothesis F: Terrorist threat has been massively exaggerated
Hypothesis G: Time is required to rebuild al-Qaeda’s capabilities
Hypothesis H: Al-Qaeda is waiting to acquire a CBRN capability
Hypothesis I: The assimilation of US muslims
Hypothesis J: A lull is occurring between Iraq and the next generation of al-Qaeda
Hypothesis K: Non-Salafist groups have lacked the capability
Bin III: Another Attack Ill-Advised (Motivations part of Threat)
Hypothesis L: Al-Qaeda’s next attack must surpass 9/11
Hypothesis M: 9/11 was a strategic miscalculation
Hypothesis N: Al-Qaeda is safeguarding its sanctuary in Pakistan
Hypothesis O: Striking the US homeland again could rally support for America
Hypothesis P: Al-Qaeda has become more sensitive to killing American civilians
Hypothesis Q: Al-Qaeda is warning the US of its intent to attack
Hypothesis R: Al-Qaeda needs success, resulting in conservative planning
Hypothesis S: Domestic extremist organizations have lacked the motivation
Hypothesis T: “Lone Wolf” terrorists have lacked the motivation
Hypothesis U: Hezbollah has been restrained by Iran and Syria
Bin IV: Other Attack Priorities (Motivations part of Threat)
Hypothesis V: Opportunities in Iraq have diverted jihadist resources
Hypothesis W: Al-Qaeda has shifted its focus to Europe
Hypothesis X: Al-Qaeda’s focus has returned to toppling Middle Eastern regimes
Hypothesis Y: Regional groups are focusing on regional targets
Hypothesis Z: Al-Qaeda’s goal is to “bleed” the United States dry economically
Hypothesis AA: 9/11 was meant to be a one-time attack
Hypothesis BB: Al-Qaeda is focused on preventing Shia ascendancy
Hypothesis CC: Non-Salafist groups have lacked the motivation to attack
Methodology
This information on each hypothesis, to include a full description, critical assumptions, supporting and contradictory evidence, was then fed to one of three working groups comprised of national security practitioners (see Appendix A for a full list). The goal of each working group was to walk through the available information and assumptions, discuss it amongst the group, and then each expert would render an independent judgment on the “likelihood that the hypothesis is valid” and the “confidence in the assessment given the quality of available evidence, knowledge, experience, …” The aggregate results for each hypothesis produced a matrix as shown below given a five-tier likelihood scale and a three-tier confidence scale (don’t be fooled by the placement of the X’s along the Y-axis - there is no meaning to the relative position of an X in the box. The editors simply forgot to center them vertically in the box.).
Several other hypotheses were developed during the meeting as follows, though none were assessed for validity (because no evidence was provided, I suppose). I am suprised, actually, that these seemingly obvious hypotheses didn’t make the original list.
Al-Qaeda is still coasting on 9/11
Terrorists are simply waiting for the right conditions to attack
The US responses to another attack is too uncertain to jeopardize current successes
For those familiar with the analytic techniques taught by Professor Frank Hughes of the National Defense Intelligence College, this exercise is essentially an implementation of Chamberlin’s Method of Multiple Working Hypotheses (MMWH) (different from the Analysis of Competing Hypotheses, or ACH). Basically, the available evidence is weighed against each hypothesis in isolation to determine a subjective level of internal support (e.g., internal = human comfort level with asserting the hypothesis as true). No consideration is given to the complementary hypotheses, nor are two or more hypotheses considered in tandem to assess synergies, subadditivities, or independence. In practice, though, it is often very difficult to look at anything in isolation, so I suspect the judgments were shaped, perhaps only in a small way, by how each expert viewed the alternatives. Unfortunately, while this is good for ACH, it isn’t necessarily good for MMWH.
Results of this Analysis
Despite their apparent lack of concern for page length, unfortunately the authors did not provide a single page summary of the results. Rather, the authors simply summarized the results up front (bottom line up front, or BLUF style) and provided the raw opinion matrices (such as shown above) for each individual hypothesis in Appendix B. The outcomes from this study were expressed in terms of the “most compelling hypotheses” (A, B, I, L, S, V, W, Y, CC) and “unpersuasive hypotheses” (F, P). (I am not sure I like the word persuasive, as it implies that the analysis must be good enough to persuade someone to see past their preinclinations, or rather, their anchors).
Additionally, the analysis did summarize areas of uncertainty that might shape future analytic efforts aimed at better understanding Al-Qaeda (pp. 27-28). In addition, the report offers several “independent counterterrorism strategies” as explained on pages 29-30.
Critique on this Study
Based on my understanding and experience as an intelligence community methodologist, I feel compelled to offer the following critiques.
The opinion matrices are very confusing to me, a person who has thought extensively about confidence and likeliness and many other uncertainty analysis issues. Typically, one speaks of likelihood (or what I refer to as likeliness) in the context of uncertainty about future events (what will happen?), not for questions of fact (what happened? why?). In general, it is sufficient to ask the simple question “is hypothesis xxx a reason for why we haven’t had another attack?” The answer then is YES or NO (as it should be since it is a question of fact). The confidence level assigned to this judgment then discounts the opinion in accordance with how much information, background knowledge, etc. was available to support this judgment. For example, if you believe the answer is YES, then the choice of confidence level places the probability that you are right somewhere between 50% and 100% (where obviously words with higher (lower) intent place you closer to 100%(50%)). Using two scales basically asks the experts to make as assessment of confidence on their assessment of uncertainty, or as the information science people like to call it, “second-order” uncertainty. For questions of future events, this is ok since your judgment (with confidence) expresses a probability distribution over alternative futures. But for questions of fact where the answer is YES or NO, it makes no sense to say “even chance YES” with “medium confidence” (basically, this statement says the expert doesn’t know with less than perfect confidence; does this mean the residual confidence is applied to something other than “even chance”?).
No guidance was described on how the experts arrived at judgments of confidence, but rather the facilitators left the details of confidence assessment open to individual interpretation. (Actually, the first paragraph on page 148 described an “intellectual dilemma” concerning confidence as it related to validity). This individual interpretation problem is such a no-no that government explicitly addressed it in the Intelligence Reform and Prevention of Terrorism Act (IRPTA), subsequently followed up by the Director of National Intelligence in Intelligence Community Directive 203 (ICD 203). That is, analysts must “properly caveat and express uncertainties or confidence in analytic judgments.” Subjective confidence judgments, as the psychologists call them, are extremely sensitive to bias, mood, stress, and a variety of other factors separate from sound reasoning. In fact, much of the intelligence community’s training efforts are focused on providing a variety of tools and techniques aimed at mitigating the effects of irrelevant (non-analytic) contributors to confidence. I, personally, spent my entire time at DIA constructing alternative methods for expressing analytic confidence that strived, to the maximum extent possible, to remove all subjectivity out of confidence assessments. According to my scheme (which I am writing up now), I won’t believe a single thing you say unless you show in writing your reasoning and source analysis. Some of Kris Wheaton’s (Intelligence Studies Program at Mercyhurst College) students also studied this problem; one even went so far as to construct a tool to help in this area (read about it here or download the thesis). But despite all this, the DTRA study resorted to the old confusing way of doing business <heavy sigh>. But then again, DTRA is not part of the intelligence community, so ICD 203 and IRPTA I suppose does not apply to them.
While the experts were treated to a lot of information and assumptions all with citations to some published document, were the experts advised to consider the quality of the underlying sources? Or were the sources vetted using some scheme to determine whether they should be included as part of a hypothesis write-up? This is unclear to me, but very important as I would not want to trust any analysis that hasn’t at least performed some sort of source analysis (and I don’t want to hear that an article must be good since it was published in The Guardian).
What I find funny is that despite some of the hypotheses judged by a majority as “almost certainly” valid with “high confidence,” the authors (perhaps guided by the experts) hedge the analysis by claiming that “the most unassailable conclusion of the study is that we simply do not know why the United States has not been successfully attacked again” (p. 26). Well of course we do not know, but the way this statement was phrased suggests to me that while the experts found the exercise less than perfectly credible, perhaps because things weren’t clear (e.g., confidence), the structure was too limiting (e.g., you can offer hypotheses, but we won’t assess them), or time was too short (e.g., too much to do, so little time). Or, perhaps this hedge isn’t a hedge at all, but an otherwise obvious analytical caveat explaining to readers that we can’t ever really know ground truth, so regardless of what we say with high or low confidence, we will never know with complete certainty why there hasn’t been another attack.
Final Thoughts
In general, this report was very interesting to read and actually gave me a lot to think about, both conceptually and methodologically. The most interesting parts of this report are Appendix B (where the methodology is explained), the Introduction chapter that explains the conference structure and key findings, and the immense detail provided for each of the 29 hypotheses. Actually, the quality of the information supporting each hypothesis is such that I can strip out the materials from 4 or 5 hypotheses and use it as a basis for an in-class ACH (or MMH) exercise. I plan to do this sometime this week - should be fun.
