In their really, REALLY good paper entitled “Assessing the Competence and Credibility of Human Sources of Intelligence Evidence: Contributions from Law and Probability” published in the journal Law Probability and Risk, Vol 6, pp. 247-274 (doi:10.1093/lpr/mgm025), authors David A. Schum (of George Mason University) and Jon R. Morris (of CIA DS&T) identified a set of twenty-five (25) questions whose answers bear on the question of whether a human source of information is competent and credible. The twenty-five questions are as follows divided into four categories: competence, veracity, objectivity, and observational sensitivity.
Competence (or is the source qualified to provide the information?)
Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s competence; (b) the evidence on this question disfavors this source’s competence; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s competence; or (d) there is no available evidence bearing on this question.
- Did this source actually make the observation being claimed or have access to the information reported?
- Does this source have an understanding of what was observed or any knowledge or expertise regarding this observation?
- Is this source generally a capable observer?
- Has this source been consistent in his/her motivation to provide us with information?
- Has this source been responsive to inquiries we have made of him/her?
Veracity (or does the source believe what he/she is saying?)
Leveraging all relevant existing evidence, for each of the ten (10) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s veracity; (b) the evidence on this question disfavors this source’s veracity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s veracity; or (d) there is no available evidence bearing on this question.
- Has the source told us anything that is inconsistent with what this source has just reported to us?
- Is this source subject to any outside influences?
- Could this source have been exploited in any way in this report to us?
- Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
- Is there any evidence from other sources that corroborates or confirms with what this source has just reported?
- What evidence do we have about this source’s character and honesty?
- What does this source’s reporting track record show about the source’s honesty in reporting to us?
- Is there evidence that this source tailored this report in a way that this source believes will capture our attention?
- Are there collateral details in this report that reflect the possibility of this source’s dishonesty?
- Evidence regarding the demeanor and bearing of this source during the interview?
Objectivity (or was the source’s belief based on the evidence obtained by the source?)
Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s objectivity; (b) the evidence on this question disfavors this source’s objectivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s objectivity; or (d) there is no available evidence bearing on this question.
- Is there evidence about what this source expected to observe during the reported observation?
- Is there evidence about what this source wished to observe during the reported observation?
- Was this source concerned about the consequences of what this source believed during the observation?
- Is there any evidence concerning possible defects in the source’s memory? Also, how long ago did this source’s observation take place?
- Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
Observational Sensitivity (or how good was the evidence obtained by the source?)
Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s observational sensitivity; (b) the evidence on this question disfavors this source’s observational sensitivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s observational sensitivity; or (d) there is no available evidence bearing on this question.
- The source’s sensory capacity at the time of observation?
- The conditions under which the observation took place?
- The source’s track record of accuracy in previous reports?
- Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
- Are there collateral details in this report that reflect the possibility of this source’s inaccuracy?
Using the Questions
According to the authors, the twenty-five questions above have been implemented in a system called MACE (or Method for Assessing the Credibility of Evidence) that apparently has been under development for some time (I wonder if MACE was fully funded by CIA; if so, do I hear FOIA request?). The remainder of the paper describes the MACE system and how it works. For the purposes of this post, it is sufficient to point out that MACE is an evidence marshalling tool. That is, MACE provides a structured set of questions that enables the analyst to make sense of the evidence bearing on a particular source’s competence and credibility.
In addition to providing an answer to each of the twenty-five questions, MACE insists that the analyst judge the relative importance of each question involving a particular situation and a particular report. Morever, MACE asks the following two questions:
- On balance, does the evidence favor or disfavor the source’s competence, veracity, objectivity, and observational sensitivity, keeping in mind the number of questions that remain unanswered?
- On balance, how strongly does the accumulated evidence favor or disfavor our believing of the report this source has just given us, keeping in mind the number of questions that remain unanswered?
Why Care?
According to the standards for analytic tradecraft articulated in Intelligence Community Directive 203 (ICD 203), all intelligence products must “properly describe the quality and reliability of underlying sources” (section D.4.e.(1)). [Note that the standard in section D.4.e.(2) is also very important, that is, "properly caveats and expresses uncertainties or confidence in analytic judgments." But I will defer this discussion until a bit later.] What Schum and Morris provide is a means for arriving at meaningful statements of source competence and credibility that simply were not available in a documented form prior to publication of this paper.
And why do I, as a risk (not necessarily intelligence, though I can play the part) professional think this is important? Well, most (if not all) security risk analyses rely mostly on the opinions of subject matter experts, organizational representatives, etc. (i.e., humans) for the information needed to make a judgment about threat, vulnerability, and risk. Much like in intelligence analysis, risk analysts must carefully appraise the information used to support analysis in terms of both its content and its source so as to ensure that the product is free of unintended bias and influence.
