Thanks to the Penn State Engineering Library, I now have pristine scanned copies of two classic literature reviews focused on psychological deterrents to nuclear theft.  These are:

• Meguire, P. G. and Kramer, J. J. (1976). “Psychological Deterrents to Nuclear Theft: A Preliminary Literature Review and Bibliography.” NSBIR 76-1007, prepared for the Defense Nuclear Agency by the Law Enforcement Standards Laboratory, National Bureau of Standards [Scribd link]

A review of the unclassified literature dealing with psychological deterrents was conducted for the Defense Nuclear Agency (DNA). Its purpose was to identify techniques that might be useful in the DNA’s Forced-Entry Deterrent Systems (FEDS) Program for psychologically deterring nuclear weapon theft. The review indicates that while human psychological processes (sensory, perceptual and cognitive) can be manipulated by various means, definitive empirical data are lacking which relate directly to deterring nuclear weapon theft. Behavioral impact research should be undertaken by DNA to (1) ascertain the deterrence values of the many techniques identified and (2) test the hypotheses implicit in the FEDS concept.

• Lapinsky, G. W. and Goodman, C. (1980). “Psychological Deterrents to Nuclear Theft: An Updated Literature Review and Bibliography.” NSBIR 80-1038, prepared for the Defense Nuclear Agency by the Law Enforcement Standards Laboratory, National Bureau of Standards [Scribd link]

A review of the unclassified literature dealing with psychological deterrents was conducted for the Defense Nuclear Agency (DNA). The review indicates that while human psychological processes (sensory, perceptual and cognitive) can be manipulated by various means, definitive empirical data are lacking which directly relate to deterring nuclear weapon theft. Behavioral impact research should be undertaken by DNA to ascertain the deterrence values of the many techniques identified.

UPDATE: My undergraduate assistant tracked down a number of very useful references extracted from the 1980 report.  These will be posted really soon.

Send article as PDF to

Everyone says that structured analytic techniques are good things to have as part of a “Thinkers Toolkit.”  In the security risk analysis degree program at Penn State, several of my colleagues and I make every attempt to instruct our students in the proper application of and value added of using structured analytic techniques to enhance one’s ability to think clearly, carefully and rigorously through complex problems.  Unfortunately, our situations suffer from a significant setback – most of our students lack “real world” experience doing analysis for problems in the security and intelligence communities (or perhaps doing any real analysis at all for any community).  Accordingly, we often find ourselves searching for carefully constructed case studies that provide the right balance of realism and accessibility to students that may not have sufficient domain knowledge to speak credibly on any particular issue.  We desire case studies that contain enough information to allow students to define the problem, articulate alternative hypotheses, leverage evidence to establish probability distributions over a set of future alternatives and degrees of confidence in analytic judgments, do source analysis, and so on.

To date we have come across several case studies used in the intelligence community, such as those developed by Professor Francis Hughes at the National Defense Intelligence College and several of the cases authored by Thomas Shreeve as part of the Intelligence Community Case Method Program.  And fortunately for us, these case studies have proven to be moderately successful when used as part of our classes.  However, we are still in search of more case studies that walk students through a problem, asking them to apply different structured analytic techniques to enable them to draw defensible inferences from data, make judgments of risk and choose from among alternative strategies for mitigating risk, explore how different ways of communicating analytic results might influence the decision maker, and so on.  And of course, we are also interested in case studies that have a variety of alternative endings, mainly to highlight that the results of the analysis and the way its communicated does have an affect on the outcomes of a situation as well as setting the stage for later analysis.

In my pursuit of fun books to read to my kids before bedtime, I recently came across the Choose Your Own Adventure series of books that many of us enjoyed during our more youthful years.  I tried to recall my experiences reading these books, such as navigating through all the alternative storylines one can follow based on the choices made during the book (one CYOA fan actually took the time to actually develop a map of The Mystery of Chimney Rock by Edward Packard; I must admit that I was tempted to do the same).  Then a thought hit me – would it be possible to develop a CYOA book that resembled a storyline that one might encounter in a professional security or intelligence position?  In addition to providing a compelling story, such a book would, of course, provide greater depth to a problem, provide evidence, and try to be as real as possible so that readers can draw on external resources to aid them in their analysis.  Now here is the kicker – each analysis or decision node would insist that the reader apply a specific structured analytic technique to arrive at the best possible answer or decision.  Once the answer is chosen, the story will then continue.  Some decision nodes would be critical to preserving national security, whereas some others might be less so or even irrelevant to the outcome.  When used as part of a course, the analyst would then prepare written reports along the way outlining the steps they took to arrive at a judgment or decision.

As an attempt to appeal to those individuals having read and enjoyed CYOA books in the past, I decided to label this idea as “Choose Your Own Analytic Adventure” or CYOAA.  See the prototype cover I prepared for the first such book in the series shown above.  I imagine that the analytic training community could create an entire series of such analytic books spanning all aspects of interest, to include terrorism, resource allocation, HUMINT targeting and collection, counter-deception, counter-proliferation, risk analysis, post-blast investigation, cyber security, communicating to decision makers, etc.  What we would need to do this are good writers, good ideas, good researchers, and of course, good artists capable of drawing pretty maps, figures, and sketches (and perhaps permission from the CYOA people to model our books after their likeness).  Just imagine it – we could hand these books out as part of class, and not only would they provide a basis for practicing analysis, but they would also make for a good addition to one’s professional library.  And if they are truly written well, then perhaps they might also make for good recreational reading.

Send article as PDF to

In a recent set of lectures for my SRA 311 course (risk management), I had my students run through a simple risk analysis case study derived from an unclassified 1993 Defense Intelligence College master’s thesis by then Captain David Lawrence Graves (USAF) entitled “Bayesian Analysis for Threat Prediction.”  The case study consisted of a “notional” scenario supported by five items of intelligence.  The case study is provided in the PDF document below.

According to the scenario statement, the Key Risk Questions center on whether it is the intent of the Revolutionary Party to attempt a hostile takeover of the US embassy in Country A, and if so, when such a takeover would be attempted.  Based on the way this scenario was written, I can’t help but feel that takeover was assumed to be guaranteed given an attempt, which is not necessarily true given the nature of typical embassy defenses, less-than-perfect capabilities of adversaries, and so on.  For the purpose of this case study, we did not make this assumption but did focus on “embassy takeover” as the sole outcome of concern. Presumably, the judgment of whether to evacuate depends on the perceived risks of a takeover attempt occurring in the next 72a-hours – if the perceived likeliness of this event (takeover attempted) combined with the severity of the associated consequences (successful takeover of US embassy) is uncomfortably “high,” then the US Ambassador in Country A might feel justified in ordering an evacuation of non-essential (or all) personnel and destruction/removal of all sensitive information.

The objective of this in-class exercise was to do a little but of source analysis (using Schum and Morris’ 25 questions) given the admittedly limited metadata on sources (which is typical), then to synthesize the nuggets of information provided by each “credible” source to order the four hypotheses based on relative likeliness. To do this, I recommended that the students use a suitable structured analytic technique to help them reason through the problem. In particular, I advocated the use of the Analysis of Competing Hypotheses technique, but without labeling it as such (I didn’t have time to offer a formal lecture on ACH, so I just gave them a list of steps). Once the hypotheses were ordered on the basis of likeliness, then, based on group judgment, the hypotheses were again ordered on the basis of vulnerability, or rather, the likeliness of outcome (e.g., takeover) given a particular hypothesis occurs. A complete risk picture examines all likeliness-severity pairs for each hypothesis (i.e., H1 through H4 as shown in the PDF file above). For example, a quick analysis might produce the following ordering of hypotheses in terms of likeliness of event (i.e., Pr(e)) and in terms of likeliness of a successful takeover given attempt (i.e., Pr(o|e)) such as is shown below.

In the end, the final risk analysis product should communicate what can happen (i.e., hypotheses H1 through H4), the relative likeliness of each, and the probability of the stated undesirable consequences. Combined, this information describes the total risk exposure for this problem. A complete risk summary such as this provides the decision maker with much of what he needs to know to make an evacuation decision. Oftentimes, professional analysts are tempted to reduce this complete narrative to a single statement such as the “risk of embassy takeover is high.” Well, perhaps this statement is true, and in some instances it may be appropriate to make such an aggregate judgment to further summarize the complete risk picture (i.e., … therefore, the risk is high). I tend to avoid making such statements as it starts to impose value judgments that are often best done by the decision makers themselves, and forces an aggregation procedure that often draws counter-constructive criticism. For example, is the analyst really in a position to assume a cut-off value of likeliness above which the decision maker should be concerned and below which the decision maker should not worry? (think of Cheney’s 1%-doctrine – was it the analyst that suggested a 1% cutoff value, or was it the decision maker?). Or, is the analyst really in a position to judge the severity of “takeover” with respect to the interests of the supported decision maker? Keeping with the goal of producing analysis that is as objective as possible, I stand by my suggestion to provide a complete narrative of risk (appropriately formatted, whether in text or as a table) that provides the decision maker with everything he needs to make his own judgment of risk.

(As an aside, note that I do, however, advocate for producing actionable risk assessments, which include both an assessment of risk combined with knowledge on which variables have the potential for risk reduction.)

The final step of this exercise was for each group to assign an appropriate level of analytic confidence to this assessment using either the DIA guidance (which I helped develop) or the Peterson Method, the latter being more straightforward but largely based on factors that correlate (not causal) with analytic confidence.

Unfortunately, given the limited time I had available for class that week, I could not afford more than 30-45 minutes in all for students to work on this exercise in class. The rest of the time was spent discussing the factors that contribute to analytic confidence, what reasoning is, the questions to aid in assessing the competence and credibility of human sources, as well as some review on measurement scales and formulas for risk analysis (which included a particularly interesting discussion focused on appraising a experimental risk formula and methodology currently in development by a government agency). What I forgot to address in class was the distinction between likeliness of events and confidence in analysis that, while both can be expressed using the same words or same mathematics (e.g., probability theory), mean completely different things. I will make it a point to bring this up in class next time.

Send article as PDF to

I just got word of a report summarizing a recent workshop held in McLean, VA focused on the “simple” question “why have we not been attacked again” since 9/11? The workshop was sponsored by the Defense Threat Reduction Agency (DTRA) in cooperation with Science Applications International Corporation (SAIC).  The cover page of the report is shown in the Scribd window below.  Per the dissemination control markings, this product is available to the public release.

Basically, this report examines a total of 29 well-defined hypotheses that address the question at issue, namely reasons for the non-occurrence of another attack (a challenging question indeed).  These hypotheses were the compiled as part of an extensive literature review centered on publications that offered reasons why he haven’t experienced a successful attack.  The fact that cells were foiled, according to the authors, was taken to mean that attacks were attempted, but were unsuccessful (a summary of failed attacks is provided in Appendix C of the report).  Using the information collected during this literature review, a comprehensive list of critical assumptions, supporting evidence and contradictory evidence was compiled for each hypothesis. The goal for the workshop was to identify the likely reasons for non-occurrence of an attack based on all available evidence and assumptions combined with the opinions of subject matter experts.

The 29 hypotheses were sorted into four bins as follows, two for each of capabilities and motivations (i.e., the two variables that comprise the DoD definition of threat). Note that these hypotheses are NOT mutually exclusive nor are they collectively, and as stated in the report, it is quite possible that one or more (or perhaps none) of these hypothesis contribute to the fact that the US hasn’t experienced a successful attack in the last 7 years.

Bin I: US and Allied Counterterrorism Efforts (Capabilities part of Threat)

• Hypothesis A: US homeland security efforts
• Hypothesis B: US and allied counterterrorism operations
• Hypothesis C: The wars in Iraq and Afghanistan have drawn jihadists away
• Hypothesis D: Reduced state support for terrorism
• Hypothesis E: Crackdowns on private financing of terrorism since 9/11

Bin II: Terrorist Attack Capabilities (Capabilities part of Threat)

• Hypothesis F: Terrorist threat has been massively exaggerated
• Hypothesis G: Time is required to rebuild al-Qaeda’s capabilities
• Hypothesis H: Al-Qaeda is waiting to acquire a CBRN capability
• Hypothesis I: The assimilation of US muslims
• Hypothesis J: A lull is occurring between Iraq and the next generation of al-Qaeda
• Hypothesis K: Non-Salafist groups have lacked the capability

Bin III: Another Attack Ill-Advised (Motivations part of Threat)

• Hypothesis L: Al-Qaeda’s next attack must surpass 9/11
• Hypothesis M: 9/11 was a strategic miscalculation
• Hypothesis N: Al-Qaeda is safeguarding its sanctuary in Pakistan
• Hypothesis O: Striking the US homeland again could rally support for America
• Hypothesis P: Al-Qaeda has become more sensitive to killing American civilians
• Hypothesis Q: Al-Qaeda is warning the US of its intent to attack
• Hypothesis R: Al-Qaeda needs success, resulting in conservative planning
• Hypothesis S: Domestic extremist organizations have lacked the motivation
• Hypothesis T: “Lone Wolf” terrorists have lacked the motivation
• Hypothesis U: Hezbollah has been restrained by Iran and Syria

Bin IV: Other Attack Priorities (Motivations part of Threat)

• Hypothesis V: Opportunities in Iraq have diverted jihadist resources
• Hypothesis W: Al-Qaeda has shifted its focus to Europe
• Hypothesis X: Al-Qaeda’s focus has returned to toppling Middle Eastern regimes
• Hypothesis Y: Regional groups are focusing on regional targets
• Hypothesis Z: Al-Qaeda’s goal is to “bleed” the United States dry economically
• Hypothesis AA: 9/11 was meant to be a one-time attack
• Hypothesis BB: Al-Qaeda is focused on preventing Shia ascendancy
• Hypothesis CC: Non-Salafist groups have lacked the motivation to attack

Methodology

This information on each hypothesis, to include a full description, critical assumptions, supporting and contradictory evidence, was then fed to one of three working groups comprised of national security practitioners (see Appendix A for a full list).  The goal of each working group was to walk through the available information and assumptions, discuss it amongst the group, and then each expert would render an independent judgment on the “likelihood that the hypothesis is valid” and the “confidence in the assessment given the quality of available evidence, knowledge, experience, …”  The aggregate results for each hypothesis produced a matrix as shown below given a five-tier likelihood scale and a three-tier confidence scale (don’t be fooled by the placement of the X’s along the Y-axis – there is no meaning to the relative position of an X in the box.  The editors simply forgot to center them vertically in the box.).

Several other hypotheses were developed during the meeting as follows, though none were assessed for validity (because no evidence was provided, I suppose).  I am suprised, actually, that these seemingly obvious hypotheses didn’t make the original list.

• Al-Qaeda is still coasting on 9/11
• Terrorists are simply waiting for the right conditions to attack
• The US responses to another attack is too uncertain to jeopardize current successes

For those familiar with the analytic techniques taught by Professor Frank Hughes of the National Defense Intelligence College, this exercise is essentially an implementation of Chamberlin’s Method of Multiple Working Hypotheses (MMWH) (different from the Analysis of Competing Hypotheses, or ACH).  Basically, the available evidence is weighed against each hypothesis in isolation to determine a subjective level of internal support (e.g., internal = human comfort level with asserting the hypothesis as true). No consideration is given to the complementary hypotheses, nor are two or more hypotheses considered in tandem to assess synergies, subadditivities, or independence. In practice, though, it is often very difficult to look at anything in isolation, so I suspect the judgments were shaped, perhaps only in a small way, by how each expert viewed the alternatives.  Unfortunately, while this is good for ACH, it isn’t necessarily good for MMWH.

Results of this Analysis

Despite their apparent lack of concern for page length, unfortunately the authors did not provide a single page summary of the results.  Rather, the authors simply summarized the results up front (bottom line up front, or BLUF style) and provided the raw opinion matrices (such as shown above) for each individual hypothesis in Appendix B.  The outcomes from this study were expressed in terms of the “most compelling hypotheses” (A, B, I, L, S, V, W, Y, CC) and “unpersuasive hypotheses” (F, P). (I am not sure I like the word persuasive, as it implies that the analysis must be good enough to persuade someone to see past their preinclinations, or rather, their anchors).

Additionally, the analysis did summarize areas of uncertainty that might shape future analytic efforts aimed at better understanding Al-Qaeda (pp. 27-28).  In addition, the report offers several “independent counterterrorism strategies” as explained on pages 29-30.

Critique on this Study

Based on my understanding and experience as an intelligence community methodologist, I feel compelled to offer the following critiques.

• The opinion matrices are very confusing to me, a person who has thought extensively about confidence and likeliness and many other uncertainty analysis issues.  Typically, one speaks of likelihood (or what I refer to as likeliness) in the context of uncertainty about future events (what will happen?), not for questions of fact (what happened? why?).  In general, it is sufficient to ask the simple question “is hypothesis xxx a reason for why we haven’t had another attack?”  The answer then is YES or NO (as it should be since it is a question of fact).  The confidence level assigned to this judgment then discounts the opinion in accordance with how much information, background knowledge, etc. was available to support this judgment.  For example, if you believe the answer is YES, then the choice of confidence level places the probability that you are right somewhere between 50% and 100% (where obviously words with higher (lower) intent place you closer to 100%(50%)).  Using two scales basically asks the experts to make as assessment of confidence on their assessment of uncertainty, or as the information science people like to call it, “second-order” uncertainty.  For questions of future events, this is ok since your judgment (with confidence) expresses a probability distribution over alternative futures.  But for questions of fact where the answer is YES or NO, it makes no sense to say “even chance YES” with “medium confidence” (basically, this statement says the expert doesn’t know with less than perfect confidence; does this mean the residual confidence is applied to something other than “even chance”?).

• No guidance was described on how the experts arrived at judgments of confidence, but rather the facilitators left the details of confidence assessment open to individual interpretation.  (Actually, the first paragraph on page 148 described an “intellectual dilemma” concerning confidence as it related to validity).  This individual interpretation problem is such a no-no that government explicitly addressed it in the Intelligence Reform and Prevention of Terrorism Act (IRPTA), subsequently followed up by the Director of National Intelligence in Intelligence Community Directive 203 (ICD 203).  That is, analysts must “properly caveat and express uncertainties or confidence in analytic judgments.”  Subjective confidence judgments, as the psychologists call them, are extremely sensitive to bias, mood, stress, and a variety of other factors separate from sound reasoning.  In fact, much of the intelligence community’s training efforts are focused on providing a variety of tools and techniques aimed at mitigating the effects of irrelevant (non-analytic) contributors to confidence.  I, personally, spent my entire time at DIA constructing alternative methods for expressing analytic confidence that strived, to the maximum extent possible, to remove all subjectivity out of confidence assessments.  According to my scheme (which I am writing up now), I won’t believe a single thing you say unless you show in writing your reasoning and source analysis.  Some of Kris Wheaton’s (Intelligence Studies Program at Mercyhurst College) students also studied this problem; one even went so far as to construct a tool to help in this area (read about it here or download the thesis).  But despite all this, the DTRA study resorted to the old confusing way of doing business <heavy sigh>.  But then again, DTRA is not part of the intelligence community, so ICD 203 and IRPTA I suppose does not apply to them.

• While the experts were treated to a lot of information and assumptions all with citations to some published document, were the experts advised to consider the quality of the underlying sources?  Or were the sources vetted using some scheme to determine whether they should be included as part of a hypothesis write-up?  This is unclear to me, but very important as I would not want to trust any analysis that hasn’t at least performed some sort of source analysis (and I don’t want to hear that an article must be good since it was published in The Guardian).

• What I find funny is that despite some of the hypotheses judged by a majority as “almost certainly” valid with “high confidence,” the authors (perhaps guided by the experts) hedge the analysis by claiming that “the most unassailable conclusion of the study is that we simply do not know why the United States has not been successfully attacked again” (p. 26).  Well of course we do not know, but the way this statement was phrased suggests to me that while the experts found the exercise less than perfectly credible, perhaps because things weren’t clear (e.g., confidence), the structure was too limiting (e.g., you can offer hypotheses, but we won’t assess them), or time was too short (e.g., too much to do, so little time).  Or, perhaps this hedge isn’t a hedge at all, but an otherwise obvious analytical caveat explaining to readers that we can’t ever really know ground truth, so regardless of what we say with high or low confidence, we will never know with complete certainty why there hasn’t been another attack.

Final Thoughts

In general, this report was very interesting to read and actually gave me a lot to think about, both conceptually and methodologically.  The most interesting parts of this report are Appendix B (where the methodology is explained), the Introduction chapter that explains the conference structure and key findings, and the immense detail provided for each of the 29 hypotheses.  Actually, the quality of the information supporting each hypothesis is such that I can strip out the materials from 4 or 5 hypotheses and use it as a basis for an in-class ACH (or MMH) exercise.  I plan to do this sometime this week – should be fun.

Send article as PDF to

In his book Political/Military Applications of Bayesian Analysis: Methodological Issues (ISBN: 0-86531-945-5), Dr. Douglas Hunter (a good friend of mine) outlines a set of 57 problems, which I label as “Hunter’s 57 Problems” (much like Heinz’s 57 Varieties), often encountered in the application of Bayesian analysis.  These 57 problems comprise the bulk of Table of Contents of the book from Chapter 4 onward, and is provided in list form below for future reference.  The book gives an example and offers strategies for overcoming each problem. (Note that I have my own strategies for dealing with some of these problems, and I do not necessarily agree with everything here; but I will defer discussion of these for later posts).

1. Hypotheses not mutually exclusive and/or not exhaustive: the problem of hypotheses which are defective because they are not mutually exclusive and/or not exhaustive.
2. A recognizably defective residual hypothesis: the problem of a final hypothesis which is recognizably too inclusive, meaningless, and “catch-all.”
3. “Buried” subhypotheses: the problem of unrecognized subhypotheses in a hypothesis (often, but not always, the residual hypothesis), which render the hypothesis somewhat meaningless.
4. The optimal number of hypotheses: the question of how many hypotheses are optimal to use in a given situation.
5. Multiple factors in hypotheses: the problem of how to develop hypotheses when we are dealing with multiple factors.
6. Time and “sliding windows”: the problem is phrasing hypotheses of whether to use a fixed time limit in the future or a “sliding window,” or even to mention the factor of time.
7. Time periods in “sliding windows”: the problem of the appropriate “sliding window” time period to use in hypotheses about decisions that have been made to do something in the future.
8. What types of questions can Bayes answer? the question as to what sorts of questions Bayes can answer and, in particular, as to whether it is reasonable to try to use Bayes to predict either future decisions by actors (such as states) or future events.
9. How far in the future can Bayes predict? the question of how far in advance can we use Bayes to predict either eventual future decisions or future states of affairs.
10. Incorrect results from using conditional independence: the problem of mathematically invalid results because of the use of the conditionally independent form of Bayes rather than the conditionally dependent form of Bayes.
11. Confusing “conditional independence” and “distinct”: the problem of incorrectly equating the concepts of “conditional independence” and “distinct” items of evidence.
12. Results affected by the order of evidence: the problem that the order in which we consider the items of evidence is itself an important factor in determining the revised probabilities of hypotheses when we use the conditionally dependent form of Bayes.
13. The types of evidence to use: the question of whether to use “all” relevant evidence or “just” indicator relevant evidence.
14. Previous data: the problem of the type of “previous data” with respect to previous indicator settings to use and which prior probabilities of hypotheses to use if we choose to use indicators.
    
15. “Causal” evidence: the problem of how to deal with actions which are are intended to alter behavior.
16. The one-sided view: the problem of the one-sided view of a complex interaction process that results if we do not utilize “causal” evidence external to the environment in our Bayesian analyses.
17. Predicting with “mixed” items of evidence: the problem of trying to predict a future event when evidence is a mixture of (1) items of evidence which indicate the state of policy at present and (2) items of evidence which may be the “causes” of possible policy change in the future.
18. Recognizing invalid “evidence”: the problem of identifying “evidence” not really from the environment.
19. “Statistical” evidence: the problem of evidence not really about an event or person, but actually about the class to which the event or person belongs.
20. “Negative” evidence: the problem of the need to consider items (events) of non-occurrences of events in particular time periods.
21. Developing lists of negative evidence: the problem of how to alert analysts to the need to consider negative evidence and how to develop lists of items of negative evidence for a particular set of hypotheses.
22. Deciding the relevance of data: the problem of deciding which items of data clearly are “relevant,” or “peripheral,” or “irrelevant” with respect to hypotheses.
23. The probability of reports vs. the probability of evidence: the problem of deciding when to ask conditional probability that the report (of certain evidence) will occur rather than asking the usual conditional probability that the evidence (contained in the report) will occur.
24. How many events to abstract? the problem of how many events, i.e., items of evidence, to abstract from a given report if we decide that it is units of evidence for which we wish to assess probabilities rather than reports.
25. “Double counting” evidence: the problem  of overweighting items of evidence by assessing the same item each time it is reported.
26. Different characterizations of the true data state: the problem of assessing the probability of an event when there are multiple reports on the event which do not characterize the event the same, but rather characterize the event in a way that is wither complementary (supplementary) or contradictory in nature; in short, the problem of how to handle different characterizations of the true data state.
27. How often to count “negative” evidence: the problem of how often to input a particular item (event) of “negative” evidence.
28. The absence of data: the problem of how to deal with the absence of data about a particular event for a certain period of time.
29. No observations of an event: the problem of how to deal with the item of evidence that a particular event has not been observed in a particular time period.
30. Source reliability: the problem of how to take into account the reliability of a source of information.
31. Assessing source reliability: the problem of how to assess the Zlotnick exponent.
32. New reports and source reliability: the problem, if Zlotnick’s exponent is used, of what to do when new reports reflecting the true data state are received.
33. Multiple reports and source reliability: the problem, if there are multiple reports on the true data state, of whether to include the source reliability for each report, and, if so, how to do so.
34. Unbelievable evidence: the problem of how to deal with evidence which does not appear believable.
35. Phrasing conditional probability judgments: the problem of how to verbalize the conditional probability of events.
36. “Blow-up”: the problem of an unwarranted rapid increase or decrease in the revised probabilities of some hypotheses.
37. “Error bounding”: the problem of the high sensitivity of the revised probabilities of hypotheses to minor variations in the assigned probabilities.
38. The importance of prior probabilities: the problem of the impact of the prior probabilities of the hypotheses if there are a few items of evidence.
39. The importance of ratios: the problem of the importance of the ratios when assigning the conditional probabilities of events occurring.
40. Probabilities or odds: the problem of whether to assign probabilities or odds.
41. Group assessments of probabilities: the problem of how to deal with a group assessment of probabilities.
42. Difficulties in thinking in terms of probabilities: the problem where some analysts think easily in terms of probabilities, others need to work at it every time, and a few need constant attention and retraining to overcome a distorted or unrealistic feeling for probabilities.
43. Lack of analyst expertise: the problem of the lack of necessary expertise to assess the probability of events for some hypotheses.
44. Logically “distant” events: the problem of an event logically “distant” from one or more hypotheses, which makes it difficult or impossible to assess a meaningful conditionally dependent probability for those hypotheses.
45. Systematic bias: the problem of systematic bias – even among sophisticated and knowledgeable – in the probabilities that are assigned to events.
46. Idiosyncratic and situational biases: the problem of biases that only certain people have because of personal, idiosyncratic factors, or because of the special situation.
47. Conscious manipulation of probabilities: the problem where an analyst may manipulate consciously his(her) assigned probabilities to support a favored hypothesis.
48. Trend line biases: the problem of analyst biases emerging from paying attention to trend lines.
49. Time-pressure difficulties: the problem that the time required to do a Bayesian analysis may make it difficult to use it in a crisis situation.
50. Fluctuating revised probabilities: the problem of revised probabilities of hypotheses fluctuating up and down, as each piece of conflicting data, reflective of inconsistent or “hedging” government policies, is processed.
51. Non-stationarity of hypotheses: the problem of a change in the state of nature.
52. “Life-span” of evidence (or “impact of past information”): the problem of when and why and to what extent the probability judgments for “old” items of evidence should be deleted from the analysis (and whether the old evidence should be considered in making probability judgments for new items of evidence)
53. Revising previously assigned probabilities: the problem of deciding under what circumstances previously assigned probabilities of events should be revised.
54. New prior probabilities: the problem of when and why to abandon the original prior probabilities and start a new analysis with new priors.
55. The “reliability” of Bayes: the problem of the “reliability” of Bayesian analysis as a function of proximity to the actual occurrence of one of the hypothesized events and whether or not we can improve reliability.
56. Differences between intuitive and Bayesian predictions: the problem of how to deal with differences (particularly drastic differences between Bayesian predictions and predictions based on personal experience), when you are “sure” your analysis is better than the one based on Bayes.
57. Consumer bias: the problem of consumer bias, particularly either gullibility or cynicism.

I must add that the first chapter of this book offers a neat and concise introduction to probability (not statistics) for intelligence analysts.  In the third chapter, the Dr. Hunter also offers what he views to be 9 major advantages of employing Bayesian analysis for intelligence:

1. Use of Bayesian analysis overcomes the conservative bias it is said we supposedly have regarding our revision of initial probabilities in light of evidence.
2. More information can be extracted from the body of available data because the technique calls for each piece of evidence to add its weight to the final assessment in a systematic way.
3. The technique compels us to employ an improved system of accounting for the evidence used.
4. If different analysts reach different conclusions, the source of the disagreements can be determined more easily than in a normal, verbal analysis.
5. Using Bayes forces us to consider alternative hypotheses.
6. As a related advantage to No. 5, use of Bayes decouples the analyst’s ego from the probability figures assigned.
7. When using Bayes, we are making deductive judgments when we assess the probability of evidence given the truth of a hypothesis and all previous evidence.  Deductive judgments are easier to make than inductive judgments, i.e., the probability of a hypothesis given evidence.
8. We are required to quantify, i.e., make explicit, and get away from the ambiguity of words, judgments which we do not ordinarily express in numerical terms.
9. The revised probability of the hypotheses can change very quickly.

Now, I am not necessarily an advocate for insisting that all analysts use Bayesian analysis in their thinking (though there are plenty of new tools available for free that greatly facilitate the use of Bayesian analysis, such as Microsoft Bayesian Editor and Toolkit).  But I do encourage analysts, if time and energy permits, to diagram their arguments and assign probabilities to events for the purpose of facilitating thinking and diagnosing errors in reasoning and judgment.  Done well, Bayesian analysis also permits sensitivity analysis on reasoning, which would help identify linchpin items of evidence or evidence that, if strengthened or weakened, would cause significant change in judgment or confidence.  Unfortunately, it takes quite a bit of training to become used to Bayesian analysis, and for this reason it will be part of only the most mathematically inclined intelligence analyst.

If one looks at Dr. Hunter’s book as not a text advocating Bayesian analysis, but as a self-help book for Bayesian thinkers (which we all are to a degree), then the 57 problems identified in this book are relevant regardless of whether one uses mathematical formulas to make explicit their thinking.  I highly recommend finding the book somewhere online or in the library and sitting down with it for a few hours to soak in all it has to say.  After all, at the time of the book’s writing in 1984, Dr. Hunter had already accumulated decades worth of experience successfully applying Bayesian analysis to challenging political and military analysis.  And if you ever had the opportunity to sit in a class with him (which I have when I took a nighttime course in the “Analysis of Competing Hypotheses”), you will come to understand that he has seen it all when it comes to both good and bad Bayesian analysis.

But before I conclude, I must point out the existence of several published reviews of this book, including the less-than-favorable one by Walter W. Hill, Jr. published in The American Political Science Review, Vol. 79, No. 2, pp. 615-616, 1985 (permalink here), and the more favorable review by Dina Zines as published in the Annals of the American Academy of Political and Social Science, Vol. 479, pp. 159-160, 1985 (permalink here).  Note that you need institutional access to JSTOR to read these reviews (or pay $9 per review).

Used copies of Dr. Hunter’s book are often available on Alibris.com for about $50.  If somehow was able to get the publisher to release this title to the public domain, I would readily make available a PDF file of the book for all to view and download (and so would Google books).  I am sure the author wouldn’t mind.

Send article as PDF to