In my search for good books on risk management and intelligence, I came across the edited volume entitled Managing Strategic Surprise: Lessons from Risk Management and Risk Assessment (edited by Professor Paul Bracken of Yale University, Dr. Ian Bremmer of the Eurasia Group, and Dr. David Gordon and the US State Department and former deputy director of the National Intelligence Council, ISBN: 9780521709606).  What a great book!  I haven’t quite finished it yet, but I must highlight that the first two substantive chapters – chapters two and three – really speak to some important issues.

For example, Paul Bracken’s article “How to Build a Warning System” (pp. 16-42) emphasizes that warning is only one piece of an organization’s overall risk management program.  Professor Bracken highlights six general strategies for risk management (the first time I have seen this): [1] isolating uncertainty (e.g., protection), [2] smoothing of uncertainty (e.g., diversification), [3] warning systems, [4] agility (e.g., rapid response), [5] alliances, and [6] environmental shaping.  Professor Bracken highlights warning systems’ role in providing advanced notice of emerging threats while emphasizing that warning can also inform decision makers of emerging opportunities.  Moreover, Professor Bracken emphasizes that there are two dimensions to warning analysis – the analytic component and the organizational component.  Warning analysis can be either informal (as it is most often the case), or highly structured (as national-level warning systems); but in general every individual and organization has some warning analysis capability.  The organizational component is absolutely essential in that without a structure in place to annunciate warning messages, warning is useless (a point emphasized in many intelligence analysis courses).  Professor Bracken suggests a contingency theory for warning:  “there is no one best way to build a warning system; it depends on the dangers” (p. 26).  The nature of the strategic environment and the capacities of an organization to collect, process, and distribute warning shape how any particular warning system functions.  Even within a single organization, multiple warning systems may be necessary to accommodate multitudes of hazards and threats.

The chapter written by former Director of Intelligence for the Israeli Mossad, Professor Uzi Arad’s article “Intelligence Management as Risk Management: The Case of Surprise Attack” (pp. 43-77) generalizes Prof. Brackens claim by suggesting the intelligence analysis is a risk management function.  He defines intelligence as a “national risk management mechanism built to cope with the risk of violent attack” (p. 45).  It should be noted that DNI’s Vision 2015 says that “intelligence helps reduce the degree of uncertainty and risk when critical choices are made” (Ch. 2).  Granted this view is rather limited by its suggestion that an intelligence organization only looks at downside risks.  But I must admit this definition, as intuitive as it is, adds another dimension to the debate over what “intelligence means” (subscribe to the IAFIE listserv to see what I mean).  More interesting is the idea that the intelligence community, perhaps unlike other types of organizations, must actually consider both environmental risks (dominated by external factors) and operational risks (dominated by internal factors) holisitcally rather than separately: external threats seek to exploit the vulnerabilities of an organization’s internal processes to prevent them from properly assessing environmental risks, thus decreasing the target organization’s decision advantage.  I believe this idea is what justifies the existence of counterintelligence and counterdeception analysis – to help mitigate an organization’s vulnerability to surprise.  This begs the question – what is the probability of a surprise afflicting an organization in its particular strategic environment?  Thinking back to Prof. Bracken’s article, an answer to this question requires us to think carefully about the nature of the strategic environment, capabilities of the adversaries, the organization’s internal processes and culture, and so on.  What I would like to see is a generic approach for assessing the risk of strategic surprise.  The remainder of the paper examines each element of the standard intelligence cycle in terms of the factors that contribute to probability of surprise.  This is good stuff.

While I haven’t read them yet, I look forward to reading the remaining chapters.  These include:

• “Nuclear Proliferation Epidemiology: Uncertainty, Surprise and Risk Management” by Ambassador Lewis A. Dunn (DTRA analysts should find this paper interesting)
• “Precaution Against Terrorism” by Dr. Jessica Stern of Harvard’s Kennedy School of Government and Professor Jonathan B. Wiener of the Duke University Law School and current president of the Society for Risk Analysis
• “Defense Planning and Risk Management in the Presence of Deep Uncertainty” by Professor Paul K. Davis of the Pardee RAND Graduate School
  
• “Manging Energy Security Risks in a Changing World” by Professor Coby van der Linde of the University of Groningen
• “What Markets Miss: Political Stability Frameworks and Country Risk” by Dr. Preston Keat of the Eurasia Group
• “The Risk of Failed-State Contagion” by Provost Jeffrey I. Herbst of Miami University of Ohio
• “Conclusion: Managing Strategic Surprise” by the book’s editors Bracken, Bremmer and Gordon

Just for reference, two interesting papers come to mind that are at least partly relevant to this book.  These include the paper “Using Risk Analysis to Inform Intelligence Analysis” (2008) by Dr. Henry H. Willis of RAND and “The Intelligence Cycle as a Model for Political Risk Assessment” (1985) [published in Political Risks in International Business edited by Thomas L. Brewer, ISBN: 0275900665] by Thomas W. Shreeve of the Intelligence Case Methods Program.  Both of these papers relate aspects of risk analysis to intelligence analysis, but neither really get to the heart of the issues as done in Managing Strategic Surprise.

Send article as PDF to

_{[[NOTE: I revised this post on 6 November 2008]]}

A few weeks ago I came across a post on Kris Wheaton’s blog Sources and Methods describing a master of science thesis by Mercyhurst graduate student Bradley E. Perry entitled “Fast and Frugal Conflict Early Warning in Sub-Saharan Africa: The Role of Intelligence Analysis.”  Since then I have been meaning to download the document and give it a careful read.  I am glad that I did – the literature review on early warning systems and risk assessment is very good.  In fact, it provided me with some incentive to read several of the books I purchased recently on the subject (e.g., Preventive Measures by Davies and Gurr (eds.)) and also pointed out some new references I will be sure to check out in the near future (e.g., “Conflict Prognostication” by Verstegen).  I highly recommend this literature review to those individuals working in the warning community – it is a relatively quick read that is well written and packed with good information.  I will be sure to advertise its existence whenever I speak to my colleagues on the subject of warning.

One thing that caught my attention was no reference to any citations that describe warning systems as a risk management tool.  We do warning to manage risk – the sooner we are made aware of an emerging situation, the sooner we can take action to ensure it doesn’t escalate in an unfavorable direction.  A good paper on this subject was written by M. Elisabeth Pate-Cornell in her 1986 article entitled “Warning Systems in Risk Management” published in the journal Risk Analysis, Vol. 6, No. 2, pp. 223-234 (DOI: 10.1111/j.1539-6924.1986.tb00210.x) [note that Professor Pate-Cornell is/was a member of the President’s Foreign Intelligence Advisory Board, the State Department International Security Advisory Board, as well as an active participant in many other very high-profile public service activities).

On the technical side, I am a bit confused with the idea of taking the highest and lowest possible values for the conflict score, 6.03 and 1.77 respectively, and assuming the middle value (3.9) as a cutoff point between conflict being likely and conflict being unlikely (see page 47).  The implication here is that the range of 1.77 to 6.03 is an unnormalized probability scale that, when normalized by subtracting the offset 1.77 and dividing by the resulting maximum 6.03-1.77, produces a scale on the range of 0 to 1.  The middle value in this case corresponds to 0.5, where values 0.5 or greater are taken as likely, and values less than 0.5 is taken as unlikely (check: (3.9-1.77)/(6.03-1.77) = 0.5).  Basically, the assumption here is that the “fast and frugal” model does produce a probability distribution on the finite frame covering the mutually exclusive and collectively exhaustive events “Violent Conflict” and “Not Violent Conflict.”  I am not convinced based on the arguments outlined in the thesis that this assumption is justified.  In fact, there appears to be no clear basis for selecting 3.9 other than it being the median value of possible score combinations (of which there are only 27).

In my original version of this post, I went on to get into the nitty-gritty of the regression, comparing the author’s analysis with that of one of his cited references.  Soon after I published the original post, I found myself delving into logistic regression and attempting to replicate the results of the cited references (which, by the way is less straightforward that one might think).  Then I realized I was getting too obsessed over work that was not my own and did not pay much attention to the bigger picture.  So I stepped back, took a deep breath, and after a careful re-examination of the thesis author’s work, I now think that the model is not bad (perhaps “good enough”, but definitely not without its flaws), is quick to use and does seem to produce reasonably good results.

My final question centers on how the particular model in this thesis informs decision making.  Some of the independent variables described in the thesis are not variables that can be changed easily, that is, they are well beyond any external actors’ ability to control.  For example, it is not that easy to change a country’s political system.  Nor is it straightforward to change the degree of ethnic homogeneity.  But perhaps something can be done to influence income inequity, such as bringing new industry to a country.  In the end I get the point – there very well might exist simple models that enable warning analysts to estimate likeliness of future events in a manner that is good enough.  The question, now, is how to develop such models that not only help predict, but do so in a manner that also offers actionable guidance into what can be feasibly influenced so as to inform strategies to decrease the potential for unfavorable futures.

Send article as PDF to

Everyone says that structured analytic techniques are good things to have as part of a “Thinkers Toolkit.”  In the security risk analysis degree program at Penn State, several of my colleagues and I make every attempt to instruct our students in the proper application of and value added of using structured analytic techniques to enhance one’s ability to think clearly, carefully and rigorously through complex problems.  Unfortunately, our situations suffer from a significant setback – most of our students lack “real world” experience doing analysis for problems in the security and intelligence communities (or perhaps doing any real analysis at all for any community).  Accordingly, we often find ourselves searching for carefully constructed case studies that provide the right balance of realism and accessibility to students that may not have sufficient domain knowledge to speak credibly on any particular issue.  We desire case studies that contain enough information to allow students to define the problem, articulate alternative hypotheses, leverage evidence to establish probability distributions over a set of future alternatives and degrees of confidence in analytic judgments, do source analysis, and so on.

To date we have come across several case studies used in the intelligence community, such as those developed by Professor Francis Hughes at the National Defense Intelligence College and several of the cases authored by Thomas Shreeve as part of the Intelligence Community Case Method Program.  And fortunately for us, these case studies have proven to be moderately successful when used as part of our classes.  However, we are still in search of more case studies that walk students through a problem, asking them to apply different structured analytic techniques to enable them to draw defensible inferences from data, make judgments of risk and choose from among alternative strategies for mitigating risk, explore how different ways of communicating analytic results might influence the decision maker, and so on.  And of course, we are also interested in case studies that have a variety of alternative endings, mainly to highlight that the results of the analysis and the way its communicated does have an affect on the outcomes of a situation as well as setting the stage for later analysis.

In my pursuit of fun books to read to my kids before bedtime, I recently came across the Choose Your Own Adventure series of books that many of us enjoyed during our more youthful years.  I tried to recall my experiences reading these books, such as navigating through all the alternative storylines one can follow based on the choices made during the book (one CYOA fan actually took the time to actually develop a map of The Mystery of Chimney Rock by Edward Packard; I must admit that I was tempted to do the same).  Then a thought hit me – would it be possible to develop a CYOA book that resembled a storyline that one might encounter in a professional security or intelligence position?  In addition to providing a compelling story, such a book would, of course, provide greater depth to a problem, provide evidence, and try to be as real as possible so that readers can draw on external resources to aid them in their analysis.  Now here is the kicker – each analysis or decision node would insist that the reader apply a specific structured analytic technique to arrive at the best possible answer or decision.  Once the answer is chosen, the story will then continue.  Some decision nodes would be critical to preserving national security, whereas some others might be less so or even irrelevant to the outcome.  When used as part of a course, the analyst would then prepare written reports along the way outlining the steps they took to arrive at a judgment or decision.

As an attempt to appeal to those individuals having read and enjoyed CYOA books in the past, I decided to label this idea as “Choose Your Own Analytic Adventure” or CYOAA.  See the prototype cover I prepared for the first such book in the series shown above.  I imagine that the analytic training community could create an entire series of such analytic books spanning all aspects of interest, to include terrorism, resource allocation, HUMINT targeting and collection, counter-deception, counter-proliferation, risk analysis, post-blast investigation, cyber security, communicating to decision makers, etc.  What we would need to do this are good writers, good ideas, good researchers, and of course, good artists capable of drawing pretty maps, figures, and sketches (and perhaps permission from the CYOA people to model our books after their likeness).  Just imagine it – we could hand these books out as part of class, and not only would they provide a basis for practicing analysis, but they would also make for a good addition to one’s professional library.  And if they are truly written well, then perhaps they might also make for good recreational reading.

Send article as PDF to

In a recent set of lectures for my SRA 311 course (risk management), I had my students run through a simple risk analysis case study derived from an unclassified 1993 Defense Intelligence College master’s thesis by then Captain David Lawrence Graves (USAF) entitled “Bayesian Analysis for Threat Prediction.”  The case study consisted of a “notional” scenario supported by five items of intelligence.  The case study is provided in the PDF document below.

According to the scenario statement, the Key Risk Questions center on whether it is the intent of the Revolutionary Party to attempt a hostile takeover of the US embassy in Country A, and if so, when such a takeover would be attempted.  Based on the way this scenario was written, I can’t help but feel that takeover was assumed to be guaranteed given an attempt, which is not necessarily true given the nature of typical embassy defenses, less-than-perfect capabilities of adversaries, and so on.  For the purpose of this case study, we did not make this assumption but did focus on “embassy takeover” as the sole outcome of concern. Presumably, the judgment of whether to evacuate depends on the perceived risks of a takeover attempt occurring in the next 72a-hours – if the perceived likeliness of this event (takeover attempted) combined with the severity of the associated consequences (successful takeover of US embassy) is uncomfortably “high,” then the US Ambassador in Country A might feel justified in ordering an evacuation of non-essential (or all) personnel and destruction/removal of all sensitive information.

The objective of this in-class exercise was to do a little but of source analysis (using Schum and Morris’ 25 questions) given the admittedly limited metadata on sources (which is typical), then to synthesize the nuggets of information provided by each “credible” source to order the four hypotheses based on relative likeliness. To do this, I recommended that the students use a suitable structured analytic technique to help them reason through the problem. In particular, I advocated the use of the Analysis of Competing Hypotheses technique, but without labeling it as such (I didn’t have time to offer a formal lecture on ACH, so I just gave them a list of steps). Once the hypotheses were ordered on the basis of likeliness, then, based on group judgment, the hypotheses were again ordered on the basis of vulnerability, or rather, the likeliness of outcome (e.g., takeover) given a particular hypothesis occurs. A complete risk picture examines all likeliness-severity pairs for each hypothesis (i.e., H1 through H4 as shown in the PDF file above). For example, a quick analysis might produce the following ordering of hypotheses in terms of likeliness of event (i.e., Pr(e)) and in terms of likeliness of a successful takeover given attempt (i.e., Pr(o|e)) such as is shown below.

In the end, the final risk analysis product should communicate what can happen (i.e., hypotheses H1 through H4), the relative likeliness of each, and the probability of the stated undesirable consequences. Combined, this information describes the total risk exposure for this problem. A complete risk summary such as this provides the decision maker with much of what he needs to know to make an evacuation decision. Oftentimes, professional analysts are tempted to reduce this complete narrative to a single statement such as the “risk of embassy takeover is high.” Well, perhaps this statement is true, and in some instances it may be appropriate to make such an aggregate judgment to further summarize the complete risk picture (i.e., … therefore, the risk is high). I tend to avoid making such statements as it starts to impose value judgments that are often best done by the decision makers themselves, and forces an aggregation procedure that often draws counter-constructive criticism. For example, is the analyst really in a position to assume a cut-off value of likeliness above which the decision maker should be concerned and below which the decision maker should not worry? (think of Cheney’s 1%-doctrine – was it the analyst that suggested a 1% cutoff value, or was it the decision maker?). Or, is the analyst really in a position to judge the severity of “takeover” with respect to the interests of the supported decision maker? Keeping with the goal of producing analysis that is as objective as possible, I stand by my suggestion to provide a complete narrative of risk (appropriately formatted, whether in text or as a table) that provides the decision maker with everything he needs to make his own judgment of risk.

(As an aside, note that I do, however, advocate for producing actionable risk assessments, which include both an assessment of risk combined with knowledge on which variables have the potential for risk reduction.)

The final step of this exercise was for each group to assign an appropriate level of analytic confidence to this assessment using either the DIA guidance (which I helped develop) or the Peterson Method, the latter being more straightforward but largely based on factors that correlate (not causal) with analytic confidence.

Unfortunately, given the limited time I had available for class that week, I could not afford more than 30-45 minutes in all for students to work on this exercise in class. The rest of the time was spent discussing the factors that contribute to analytic confidence, what reasoning is, the questions to aid in assessing the competence and credibility of human sources, as well as some review on measurement scales and formulas for risk analysis (which included a particularly interesting discussion focused on appraising a experimental risk formula and methodology currently in development by a government agency). What I forgot to address in class was the distinction between likeliness of events and confidence in analysis that, while both can be expressed using the same words or same mathematics (e.g., probability theory), mean completely different things. I will make it a point to bring this up in class next time.

Send article as PDF to

In their really, REALLY good paper entitled “Assessing the Competence and Credibility of Human Sources of Intelligence Evidence: Contributions from Law and Probability” published in the journal Law Probability and Risk, Vol 6, pp. 247-274 (doi:10.1093/lpr/mgm025), authors David A. Schum (of George Mason University) and Jon R. Morris (of CIA DS&T) identified a set of twenty-five (25) questions whose answers bear on the question of whether a human source of information is competent and credible.  The twenty-five questions are as follows divided into four categories: competence, veracity, objectivity, and observational sensitivity.

Competence (or is the source qualified to provide the information?)

Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s competence; (b) the evidence on this question disfavors this source’s competence; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s competence; or (d) there is no available evidence bearing on this question.

1. Did this source actually make the observation being claimed or have access to the information reported?
2. Does this source have an understanding of what was observed or any knowledge or expertise regarding this observation?
3. Is this source generally a capable observer?
4. Has this source been consistent in his/her motivation to provide us with information?
5. Has this source been responsive to inquiries we have made of him/her?

Veracity (or does the source believe what he/she is saying?)

Leveraging all relevant existing evidence, for each of the ten (10) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s veracity; (b) the evidence on this question disfavors this source’s veracity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s veracity; or (d) there is no available evidence bearing on this question.

1. Has the source told us anything that is inconsistent with what this source has just reported to us?
2. Is this source subject to any outside influences?
3. Could this source have been exploited in any way in this report to us?
4. Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
5. Is there any evidence from other sources that corroborates or confirms with what this source has just reported?
6. What evidence do we have about this source’s character and honesty?
7. What does this source’s reporting track record show about the source’s honesty in reporting to us?
8. Is there evidence that this source tailored this report in a way that this source believes will capture our attention?
9. Are there collateral details in this report that reflect the possibility of this source’s dishonesty?
10. Evidence regarding the demeanor and bearing of this source during the interview?

Objectivity (or was the source’s belief based on the evidence obtained by the source?)

Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s objectivity; (b) the evidence on this question disfavors this source’s objectivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s objectivity; or (d) there is no available evidence bearing on this question.

1. Is there evidence about what this source expected to observe during the reported observation?
2. Is there evidence about what this source wished to observe during the reported observation?
3. Was this source concerned about the consequences of what this source believed during the observation?
4. Is there any evidence concerning possible defects in the source’s memory? Also, how long ago did this source’s observation take place?
5. Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?

Observational Sensitivity (or how good was the evidence obtained by the source?)

Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s observational sensitivity; (b) the evidence on this question disfavors this source’s observational sensitivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s observational sensitivity; or (d) there is no available evidence bearing on this question.

1. The source’s sensory capacity at the time of observation?
2. The conditions under which the observation took place?
3. The source’s track record of accuracy in previous reports?
4. Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
5. Are there collateral details in this report that reflect the possibility of this source’s inaccuracy?

Using the Questions

According to the authors, the twenty-five questions above have been implemented in a system called MACE (or Method for Assessing the Credibility of Evidence) that apparently has been under development for some time (I wonder if MACE was fully funded by CIA; if so, do I hear FOIA request?).  The remainder of the paper describes the MACE system and how it works.  For the purposes of this post, it is sufficient to point out that MACE is an evidence marshalling tool.  That is, MACE provides a structured set of questions that enables the analyst to make sense of the evidence bearing on a particular source’s competence and credibility.

In addition to providing an answer to each of the twenty-five questions, MACE insists that the analyst judge the relative importance of each question involving a particular situation and a particular report.  Morever, MACE asks the following two questions:

1. On balance, does the evidence favor or disfavor the source’s competence, veracity, objectivity, and observational sensitivity, keeping in mind the number of questions that remain unanswered?
2. On balance, how strongly does the accumulated evidence favor or disfavor our believing of the report this source has just given us, keeping in mind the number of questions that remain unanswered?

Why Care?

According to the standards for analytic tradecraft articulated in Intelligence Community Directive 203 (ICD 203), all intelligence products must “properly describe the quality and reliability of underlying sources” (section D.4.e.(1)).  [Note that the standard in section D.4.e.(2) is also very important, that is, "properly caveats and expresses uncertainties or confidence in analytic judgments."  But I will defer this discussion until a bit later.]  What Schum and Morris provide is a means for arriving at meaningful statements of source competence and credibility that simply were not available in a documented form prior to publication of this paper.

And why do I, as a risk (not necessarily intelligence, though I can play the part) professional think this is important?  Well, most (if not all) security risk analyses rely mostly on the opinions of subject matter experts, organizational representatives, etc. (i.e., humans) for the information needed to make a judgment about threat, vulnerability, and risk.  Much like in intelligence analysis, risk analysts must carefully appraise the information used to support analysis in terms of both its content and its source so as to ensure that the product is free of unintended bias and influence.

Send article as PDF to

In his book Political/Military Applications of Bayesian Analysis: Methodological Issues (ISBN: 0-86531-945-5), Dr. Douglas Hunter (a good friend of mine) outlines a set of 57 problems, which I label as “Hunter’s 57 Problems” (much like Heinz’s 57 Varieties), often encountered in the application of Bayesian analysis.  These 57 problems comprise the bulk of Table of Contents of the book from Chapter 4 onward, and is provided in list form below for future reference.  The book gives an example and offers strategies for overcoming each problem. (Note that I have my own strategies for dealing with some of these problems, and I do not necessarily agree with everything here; but I will defer discussion of these for later posts).

1. Hypotheses not mutually exclusive and/or not exhaustive: the problem of hypotheses which are defective because they are not mutually exclusive and/or not exhaustive.
2. A recognizably defective residual hypothesis: the problem of a final hypothesis which is recognizably too inclusive, meaningless, and “catch-all.”
3. “Buried” subhypotheses: the problem of unrecognized subhypotheses in a hypothesis (often, but not always, the residual hypothesis), which render the hypothesis somewhat meaningless.
4. The optimal number of hypotheses: the question of how many hypotheses are optimal to use in a given situation.
5. Multiple factors in hypotheses: the problem of how to develop hypotheses when we are dealing with multiple factors.
6. Time and “sliding windows”: the problem is phrasing hypotheses of whether to use a fixed time limit in the future or a “sliding window,” or even to mention the factor of time.
7. Time periods in “sliding windows”: the problem of the appropriate “sliding window” time period to use in hypotheses about decisions that have been made to do something in the future.
8. What types of questions can Bayes answer? the question as to what sorts of questions Bayes can answer and, in particular, as to whether it is reasonable to try to use Bayes to predict either future decisions by actors (such as states) or future events.
9. How far in the future can Bayes predict? the question of how far in advance can we use Bayes to predict either eventual future decisions or future states of affairs.
10. Incorrect results from using conditional independence: the problem of mathematically invalid results because of the use of the conditionally independent form of Bayes rather than the conditionally dependent form of Bayes.
11. Confusing “conditional independence” and “distinct”: the problem of incorrectly equating the concepts of “conditional independence” and “distinct” items of evidence.
12. Results affected by the order of evidence: the problem that the order in which we consider the items of evidence is itself an important factor in determining the revised probabilities of hypotheses when we use the conditionally dependent form of Bayes.
13. The types of evidence to use: the question of whether to use “all” relevant evidence or “just” indicator relevant evidence.
14. Previous data: the problem of the type of “previous data” with respect to previous indicator settings to use and which prior probabilities of hypotheses to use if we choose to use indicators.
    
15. “Causal” evidence: the problem of how to deal with actions which are are intended to alter behavior.
16. The one-sided view: the problem of the one-sided view of a complex interaction process that results if we do not utilize “causal” evidence external to the environment in our Bayesian analyses.
17. Predicting with “mixed” items of evidence: the problem of trying to predict a future event when evidence is a mixture of (1) items of evidence which indicate the state of policy at present and (2) items of evidence which may be the “causes” of possible policy change in the future.
18. Recognizing invalid “evidence”: the problem of identifying “evidence” not really from the environment.
19. “Statistical” evidence: the problem of evidence not really about an event or person, but actually about the class to which the event or person belongs.
20. “Negative” evidence: the problem of the need to consider items (events) of non-occurrences of events in particular time periods.
21. Developing lists of negative evidence: the problem of how to alert analysts to the need to consider negative evidence and how to develop lists of items of negative evidence for a particular set of hypotheses.
22. Deciding the relevance of data: the problem of deciding which items of data clearly are “relevant,” or “peripheral,” or “irrelevant” with respect to hypotheses.
23. The probability of reports vs. the probability of evidence: the problem of deciding when to ask conditional probability that the report (of certain evidence) will occur rather than asking the usual conditional probability that the evidence (contained in the report) will occur.
24. How many events to abstract? the problem of how many events, i.e., items of evidence, to abstract from a given report if we decide that it is units of evidence for which we wish to assess probabilities rather than reports.
25. “Double counting” evidence: the problem  of overweighting items of evidence by assessing the same item each time it is reported.
26. Different characterizations of the true data state: the problem of assessing the probability of an event when there are multiple reports on the event which do not characterize the event the same, but rather characterize the event in a way that is wither complementary (supplementary) or contradictory in nature; in short, the problem of how to handle different characterizations of the true data state.
27. How often to count “negative” evidence: the problem of how often to input a particular item (event) of “negative” evidence.
28. The absence of data: the problem of how to deal with the absence of data about a particular event for a certain period of time.
29. No observations of an event: the problem of how to deal with the item of evidence that a particular event has not been observed in a particular time period.
30. Source reliability: the problem of how to take into account the reliability of a source of information.
31. Assessing source reliability: the problem of how to assess the Zlotnick exponent.
32. New reports and source reliability: the problem, if Zlotnick’s exponent is used, of what to do when new reports reflecting the true data state are received.
33. Multiple reports and source reliability: the problem, if there are multiple reports on the true data state, of whether to include the source reliability for each report, and, if so, how to do so.
34. Unbelievable evidence: the problem of how to deal with evidence which does not appear believable.
35. Phrasing conditional probability judgments: the problem of how to verbalize the conditional probability of events.
36. “Blow-up”: the problem of an unwarranted rapid increase or decrease in the revised probabilities of some hypotheses.
37. “Error bounding”: the problem of the high sensitivity of the revised probabilities of hypotheses to minor variations in the assigned probabilities.
38. The importance of prior probabilities: the problem of the impact of the prior probabilities of the hypotheses if there are a few items of evidence.
39. The importance of ratios: the problem of the importance of the ratios when assigning the conditional probabilities of events occurring.
40. Probabilities or odds: the problem of whether to assign probabilities or odds.
41. Group assessments of probabilities: the problem of how to deal with a group assessment of probabilities.
42. Difficulties in thinking in terms of probabilities: the problem where some analysts think easily in terms of probabilities, others need to work at it every time, and a few need constant attention and retraining to overcome a distorted or unrealistic feeling for probabilities.
43. Lack of analyst expertise: the problem of the lack of necessary expertise to assess the probability of events for some hypotheses.
44. Logically “distant” events: the problem of an event logically “distant” from one or more hypotheses, which makes it difficult or impossible to assess a meaningful conditionally dependent probability for those hypotheses.
45. Systematic bias: the problem of systematic bias – even among sophisticated and knowledgeable – in the probabilities that are assigned to events.
46. Idiosyncratic and situational biases: the problem of biases that only certain people have because of personal, idiosyncratic factors, or because of the special situation.
47. Conscious manipulation of probabilities: the problem where an analyst may manipulate consciously his(her) assigned probabilities to support a favored hypothesis.
48. Trend line biases: the problem of analyst biases emerging from paying attention to trend lines.
49. Time-pressure difficulties: the problem that the time required to do a Bayesian analysis may make it difficult to use it in a crisis situation.
50. Fluctuating revised probabilities: the problem of revised probabilities of hypotheses fluctuating up and down, as each piece of conflicting data, reflective of inconsistent or “hedging” government policies, is processed.
51. Non-stationarity of hypotheses: the problem of a change in the state of nature.
52. “Life-span” of evidence (or “impact of past information”): the problem of when and why and to what extent the probability judgments for “old” items of evidence should be deleted from the analysis (and whether the old evidence should be considered in making probability judgments for new items of evidence)
53. Revising previously assigned probabilities: the problem of deciding under what circumstances previously assigned probabilities of events should be revised.
54. New prior probabilities: the problem of when and why to abandon the original prior probabilities and start a new analysis with new priors.
55. The “reliability” of Bayes: the problem of the “reliability” of Bayesian analysis as a function of proximity to the actual occurrence of one of the hypothesized events and whether or not we can improve reliability.
56. Differences between intuitive and Bayesian predictions: the problem of how to deal with differences (particularly drastic differences between Bayesian predictions and predictions based on personal experience), when you are “sure” your analysis is better than the one based on Bayes.
57. Consumer bias: the problem of consumer bias, particularly either gullibility or cynicism.

I must add that the first chapter of this book offers a neat and concise introduction to probability (not statistics) for intelligence analysts.  In the third chapter, the Dr. Hunter also offers what he views to be 9 major advantages of employing Bayesian analysis for intelligence:

1. Use of Bayesian analysis overcomes the conservative bias it is said we supposedly have regarding our revision of initial probabilities in light of evidence.
2. More information can be extracted from the body of available data because the technique calls for each piece of evidence to add its weight to the final assessment in a systematic way.
3. The technique compels us to employ an improved system of accounting for the evidence used.
4. If different analysts reach different conclusions, the source of the disagreements can be determined more easily than in a normal, verbal analysis.
5. Using Bayes forces us to consider alternative hypotheses.
6. As a related advantage to No. 5, use of Bayes decouples the analyst’s ego from the probability figures assigned.
7. When using Bayes, we are making deductive judgments when we assess the probability of evidence given the truth of a hypothesis and all previous evidence.  Deductive judgments are easier to make than inductive judgments, i.e., the probability of a hypothesis given evidence.
8. We are required to quantify, i.e., make explicit, and get away from the ambiguity of words, judgments which we do not ordinarily express in numerical terms.
9. The revised probability of the hypotheses can change very quickly.

Now, I am not necessarily an advocate for insisting that all analysts use Bayesian analysis in their thinking (though there are plenty of new tools available for free that greatly facilitate the use of Bayesian analysis, such as Microsoft Bayesian Editor and Toolkit).  But I do encourage analysts, if time and energy permits, to diagram their arguments and assign probabilities to events for the purpose of facilitating thinking and diagnosing errors in reasoning and judgment.  Done well, Bayesian analysis also permits sensitivity analysis on reasoning, which would help identify linchpin items of evidence or evidence that, if strengthened or weakened, would cause significant change in judgment or confidence.  Unfortunately, it takes quite a bit of training to become used to Bayesian analysis, and for this reason it will be part of only the most mathematically inclined intelligence analyst.

If one looks at Dr. Hunter’s book as not a text advocating Bayesian analysis, but as a self-help book for Bayesian thinkers (which we all are to a degree), then the 57 problems identified in this book are relevant regardless of whether one uses mathematical formulas to make explicit their thinking.  I highly recommend finding the book somewhere online or in the library and sitting down with it for a few hours to soak in all it has to say.  After all, at the time of the book’s writing in 1984, Dr. Hunter had already accumulated decades worth of experience successfully applying Bayesian analysis to challenging political and military analysis.  And if you ever had the opportunity to sit in a class with him (which I have when I took a nighttime course in the “Analysis of Competing Hypotheses”), you will come to understand that he has seen it all when it comes to both good and bad Bayesian analysis.

But before I conclude, I must point out the existence of several published reviews of this book, including the less-than-favorable one by Walter W. Hill, Jr. published in The American Political Science Review, Vol. 79, No. 2, pp. 615-616, 1985 (permalink here), and the more favorable review by Dina Zines as published in the Annals of the American Academy of Political and Social Science, Vol. 479, pp. 159-160, 1985 (permalink here).  Note that you need institutional access to JSTOR to read these reviews (or pay $9 per review).

Used copies of Dr. Hunter’s book are often available on Alibris.com for about $50.  If somehow was able to get the publisher to release this title to the public domain, I would readily make available a PDF file of the book for all to view and download (and so would Google books).  I am sure the author wouldn’t mind.

Send article as PDF to

The National Defense Intelligence College, or NDIC (formerly the Joint Military Intelligence College, or JMIC) recently joined the ranks of most other US colleges and universities by creating an identity on the Internet.  Previously, the only way to learn anything about this “secretive” school was to navigate to some meager, uninformative information via the Defense Intelligence Agency website.

The nice thing about NDIC’s new website is the online availability of its research publications via NDIC College Press.  Of course, having print copies (which I have most of) is best, but PDF files are almost as good.  These publications were authored by people of mixed roles, including NDIC faculty, non-NDIC academics, IC employees, government contractors, and NDIC student who happened to write excellent theses.  Having had the opportunity to browse most of these titles in the past, I recommend the following to those interested in reading to build analytic skills and background knowledge for modern security risk management:

• Global War on Terrorism: Analyzing the Strategic Threat (2004) by Anonymous
• Critical Thinking and Intelligence Analysis (2007) by David Moore
• Beneath the Surface: Intelligence Preparation of the Battlespace for Counterterrorism (2004) by Troy Thomas
• Anticipating Surprise: Analysis for Strategic Warning (2002) by Cynthia Grabo (HIGHLY RECOMMENDED)
• Experiences to Go: Teaching With Intelligence Case Studies (2004) by Thomas Shreeve
• Crime Scene Intelligence: An Experiment in Forensic Entomology (2006) by Albert Cruz (note that “entomology” is the study of bugs)
• Warning Analysis for the Information Age: Rethinking the Intelligence Process (2003) by John Bodnar
• Intelligence Essentials for Everyone (1999) by Lisa Krizan

NDIC students, as part of their requirements for graduation, are required to author theses on intelligence-related topics.  A number of high-quality theses are produced each year, and from among these the following unclassified theses are ones that might interest security risk professionals.  Unfortunately, one has to contact NDIC directly to obtain copies.  (It would really be nice, though, if they made all unclassified theses available for online viewing).

• “Bayesian Analysis Methods for Threat Prediction” (1993) by Graves (a massive 500+ page thesis)
• “An Application and Comparison of Bayesian Analysis Techniques” (1992) by Shade
• “Creating Intelligence: Evidence and Inference in the Intelligence Process” (2002) by Moore
• “Exploiting Structured Methodologies to Improve Qualitative Intelligence Analysis” (1999) by Folker

The first two theses above were advised by my good friend Dr. Douglas Hunter, whereas the latter two theses were advised by my other good friend Professor Frank Hughes.

Finally, the NDIC Student Handbook is available online for viewing by the public.  Among other things, this handbook contains a catalog of courses offered by the college.  For those fortunate enough to have the opportunity to take courses at NDIC, either as an enrolled or space-available student, the following courses are excellent.  But as with any course, it really depends on who is teaching the course when it is offered,

• MSI 607: Argumentation, Logic and Reasoning (I took the previous version of the course, ANA 620, with Mr. George Fidas whose is now at the Elliot School of International Affairs at GWU.  A recent paper by Mr. Fidas can be found here)
• MSI 611: Intelligence and National Security Policy (I took it with Prof. Elizabeth Pickering)
• MSI 643: Advanced Analysis (I helped contribute to this course by giving a guest lecture on Uncertainty Analysis to students in its first offering)
• MSI 645: Covert Action (I sat in a few lectures of this largely unclassified course, and I found it to be very interesting)
• MSI 651: Roots of Terrorism (I never took this class, but a good friend of mine did and he said it was excellent)
• MSI 667: Nuclear, Radiological, Explosives, and Chemical Agents (I took this course with Dr. William Williamson when it was ANA 653: S&T Analysis (WMD) – a good course)
• DAD 600: Introduction to Denial and Deception: History, Concepts, Issues and Implications
• DAD 601: Denial and Deception: Psychological/Cultural Aspects and National Security Decision Making
• DAD 602: Denial and Deception: Adversaries, Organizations, Activities, and Countermeasures
• DAD 603: Denial and Deception: Tradecraft, Tools and Methodology

Now, I never got around to finishing my Master of Science in Strategic Intelligence (MSSI) program at NDIC, largely due to the fact that I wasn’t a government employee long enough to finish the required coursework.  You see, besides having the requisite educational background (BS, GREs, etc.), there is one other requirement that limits enrollment in the MSSI program: you must be a federal government employee (not contractor).  Oh well.  But in the short time I spent there at NDIC (and JMIC, when it was called that), I met a number of great people with whom I continue to enjoy friendly relations.

Send article as PDF to