In a recent set of lectures for my SRA 311 course (risk management), I had my students run through a simple risk analysis case study derived from an unclassified 1993 Defense Intelligence College master’s thesis by then Captain David Lawrence Graves (USAF) entitled “Bayesian Analysis for Threat Prediction.”  The case study consisted of a “notional” scenario supported by five items of intelligence.  The case study is provided in the PDF document below.

According to the scenario statement, the Key Risk Questions center on whether it is the intent of the Revolutionary Party to attempt a hostile takeover of the US embassy in Country A, and if so, when such a takeover would be attempted.  Based on the way this scenario was written, I can’t help but feel that takeover was assumed to be guaranteed given an attempt, which is not necessarily true given the nature of typical embassy defenses, less-than-perfect capabilities of adversaries, and so on.  For the purpose of this case study, we did not make this assumption but did focus on “embassy takeover” as the sole outcome of concern. Presumably, the judgment of whether to evacuate depends on the perceived risks of a takeover attempt occurring in the next 72a-hours – if the perceived likeliness of this event (takeover attempted) combined with the severity of the associated consequences (successful takeover of US embassy) is uncomfortably “high,” then the US Ambassador in Country A might feel justified in ordering an evacuation of non-essential (or all) personnel and destruction/removal of all sensitive information.

The objective of this in-class exercise was to do a little but of source analysis (using Schum and Morris’ 25 questions) given the admittedly limited metadata on sources (which is typical), then to synthesize the nuggets of information provided by each “credible” source to order the four hypotheses based on relative likeliness. To do this, I recommended that the students use a suitable structured analytic technique to help them reason through the problem. In particular, I advocated the use of the Analysis of Competing Hypotheses technique, but without labeling it as such (I didn’t have time to offer a formal lecture on ACH, so I just gave them a list of steps). Once the hypotheses were ordered on the basis of likeliness, then, based on group judgment, the hypotheses were again ordered on the basis of vulnerability, or rather, the likeliness of outcome (e.g., takeover) given a particular hypothesis occurs. A complete risk picture examines all likeliness-severity pairs for each hypothesis (i.e., H1 through H4 as shown in the PDF file above). For example, a quick analysis might produce the following ordering of hypotheses in terms of likeliness of event (i.e., Pr(e)) and in terms of likeliness of a successful takeover given attempt (i.e., Pr(o|e)) such as is shown below.

In the end, the final risk analysis product should communicate what can happen (i.e., hypotheses H1 through H4), the relative likeliness of each, and the probability of the stated undesirable consequences. Combined, this information describes the total risk exposure for this problem. A complete risk summary such as this provides the decision maker with much of what he needs to know to make an evacuation decision. Oftentimes, professional analysts are tempted to reduce this complete narrative to a single statement such as the “risk of embassy takeover is high.” Well, perhaps this statement is true, and in some instances it may be appropriate to make such an aggregate judgment to further summarize the complete risk picture (i.e., … therefore, the risk is high). I tend to avoid making such statements as it starts to impose value judgments that are often best done by the decision makers themselves, and forces an aggregation procedure that often draws counter-constructive criticism. For example, is the analyst really in a position to assume a cut-off value of likeliness above which the decision maker should be concerned and below which the decision maker should not worry? (think of Cheney’s 1%-doctrine – was it the analyst that suggested a 1% cutoff value, or was it the decision maker?). Or, is the analyst really in a position to judge the severity of “takeover” with respect to the interests of the supported decision maker? Keeping with the goal of producing analysis that is as objective as possible, I stand by my suggestion to provide a complete narrative of risk (appropriately formatted, whether in text or as a table) that provides the decision maker with everything he needs to make his own judgment of risk.

(As an aside, note that I do, however, advocate for producing actionable risk assessments, which include both an assessment of risk combined with knowledge on which variables have the potential for risk reduction.)

The final step of this exercise was for each group to assign an appropriate level of analytic confidence to this assessment using either the DIA guidance (which I helped develop) or the Peterson Method, the latter being more straightforward but largely based on factors that correlate (not causal) with analytic confidence.

Unfortunately, given the limited time I had available for class that week, I could not afford more than 30-45 minutes in all for students to work on this exercise in class. The rest of the time was spent discussing the factors that contribute to analytic confidence, what reasoning is, the questions to aid in assessing the competence and credibility of human sources, as well as some review on measurement scales and formulas for risk analysis (which included a particularly interesting discussion focused on appraising a experimental risk formula and methodology currently in development by a government agency). What I forgot to address in class was the distinction between likeliness of events and confidence in analysis that, while both can be expressed using the same words or same mathematics (e.g., probability theory), mean completely different things. I will make it a point to bring this up in class next time.

Send article as PDF to

In their really, REALLY good paper entitled “Assessing the Competence and Credibility of Human Sources of Intelligence Evidence: Contributions from Law and Probability” published in the journal Law Probability and Risk, Vol 6, pp. 247-274 (doi:10.1093/lpr/mgm025), authors David A. Schum (of George Mason University) and Jon R. Morris (of CIA DS&T) identified a set of twenty-five (25) questions whose answers bear on the question of whether a human source of information is competent and credible.  The twenty-five questions are as follows divided into four categories: competence, veracity, objectivity, and observational sensitivity.

Competence (or is the source qualified to provide the information?)

Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s competence; (b) the evidence on this question disfavors this source’s competence; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s competence; or (d) there is no available evidence bearing on this question.

1. Did this source actually make the observation being claimed or have access to the information reported?
2. Does this source have an understanding of what was observed or any knowledge or expertise regarding this observation?
3. Is this source generally a capable observer?
4. Has this source been consistent in his/her motivation to provide us with information?
5. Has this source been responsive to inquiries we have made of him/her?

Veracity (or does the source believe what he/she is saying?)

Leveraging all relevant existing evidence, for each of the ten (10) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s veracity; (b) the evidence on this question disfavors this source’s veracity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s veracity; or (d) there is no available evidence bearing on this question.

1. Has the source told us anything that is inconsistent with what this source has just reported to us?
2. Is this source subject to any outside influences?
3. Could this source have been exploited in any way in this report to us?
4. Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
5. Is there any evidence from other sources that corroborates or confirms with what this source has just reported?
6. What evidence do we have about this source’s character and honesty?
7. What does this source’s reporting track record show about the source’s honesty in reporting to us?
8. Is there evidence that this source tailored this report in a way that this source believes will capture our attention?
9. Are there collateral details in this report that reflect the possibility of this source’s dishonesty?
10. Evidence regarding the demeanor and bearing of this source during the interview?

Objectivity (or was the source’s belief based on the evidence obtained by the source?)

Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s objectivity; (b) the evidence on this question disfavors this source’s objectivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s objectivity; or (d) there is no available evidence bearing on this question.

1. Is there evidence about what this source expected to observe during the reported observation?
2. Is there evidence about what this source wished to observe during the reported observation?
3. Was this source concerned about the consequences of what this source believed during the observation?
4. Is there any evidence concerning possible defects in the source’s memory? Also, how long ago did this source’s observation take place?
5. Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?

Observational Sensitivity (or how good was the evidence obtained by the source?)

Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s observational sensitivity; (b) the evidence on this question disfavors this source’s observational sensitivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s observational sensitivity; or (d) there is no available evidence bearing on this question.

1. The source’s sensory capacity at the time of observation?
2. The conditions under which the observation took place?
3. The source’s track record of accuracy in previous reports?
4. Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
5. Are there collateral details in this report that reflect the possibility of this source’s inaccuracy?

Using the Questions

According to the authors, the twenty-five questions above have been implemented in a system called MACE (or Method for Assessing the Credibility of Evidence) that apparently has been under development for some time (I wonder if MACE was fully funded by CIA; if so, do I hear FOIA request?).  The remainder of the paper describes the MACE system and how it works.  For the purposes of this post, it is sufficient to point out that MACE is an evidence marshalling tool.  That is, MACE provides a structured set of questions that enables the analyst to make sense of the evidence bearing on a particular source’s competence and credibility.

In addition to providing an answer to each of the twenty-five questions, MACE insists that the analyst judge the relative importance of each question involving a particular situation and a particular report.  Morever, MACE asks the following two questions:

1. On balance, does the evidence favor or disfavor the source’s competence, veracity, objectivity, and observational sensitivity, keeping in mind the number of questions that remain unanswered?
2. On balance, how strongly does the accumulated evidence favor or disfavor our believing of the report this source has just given us, keeping in mind the number of questions that remain unanswered?

Why Care?

According to the standards for analytic tradecraft articulated in Intelligence Community Directive 203 (ICD 203), all intelligence products must “properly describe the quality and reliability of underlying sources” (section D.4.e.(1)).  [Note that the standard in section D.4.e.(2) is also very important, that is, "properly caveats and expresses uncertainties or confidence in analytic judgments."  But I will defer this discussion until a bit later.]  What Schum and Morris provide is a means for arriving at meaningful statements of source competence and credibility that simply were not available in a documented form prior to publication of this paper.

And why do I, as a risk (not necessarily intelligence, though I can play the part) professional think this is important?  Well, most (if not all) security risk analyses rely mostly on the opinions of subject matter experts, organizational representatives, etc. (i.e., humans) for the information needed to make a judgment about threat, vulnerability, and risk.  Much like in intelligence analysis, risk analysts must carefully appraise the information used to support analysis in terms of both its content and its source so as to ensure that the product is free of unintended bias and influence.

Send article as PDF to