Source Analysis and the Twenty-Five Questions of Schum and Morris
Written by Will McGill on September 20th, 2008In their really, REALLY good paper entitled “Assessing the Competence and Credibility of Human Sources of Intelligence Evidence: Contributions from Law and Probability” published in the journal Law Probability and Risk, Vol 6, pp. 247-274 (doi:10.1093/lpr/mgm025), authors David A. Schum (of George Mason University) and Jon R. Morris (of CIA DS&T) identified a set of twenty-five (25) questions whose answers bear on the question of whether a human source of information is competent and credible. The twenty-five questions are as follows divided into four categories: competence, veracity, objectivity, and observational sensitivity.
Competence (or is the source qualified to provide the information?)
Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s competence; (b) the evidence on this question disfavors this source’s competence; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s competence; or (d) there is no available evidence bearing on this question.
- Did this source actually make the observation being claimed or have access to the information reported?
- Does this source have an understanding of what was observed or any knowledge or expertise regarding this observation?
- Is this source generally a capable observer?
- Has this source been consistent in his/her motivation to provide us with information?
- Has this source been responsive to inquiries we have made of him/her?
Veracity (or does the source believe what he/she is saying?)
Leveraging all relevant existing evidence, for each of the ten (10) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s veracity; (b) the evidence on this question disfavors this source’s veracity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s veracity; or (d) there is no available evidence bearing on this question.
- Has the source told us anything that is inconsistent with what this source has just reported to us?
- Is this source subject to any outside influences?
- Could this source have been exploited in any way in this report to us?
- Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
- Is there any evidence from other sources that corroborates or confirms with what this source has just reported?
- What evidence do we have about this source’s character and honesty?
- What does this source’s reporting track record show about the source’s honesty in reporting to us?
- Is there evidence that this source tailored this report in a way that this source believes will capture our attention?
- Are there collateral details in this report that reflect the possibility of this source’s dishonesty?
- Evidence regarding the demeanor and bearing of this source during the interview?
Objectivity (or was the source’s belief based on the evidence obtained by the source?)
Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s objectivity; (b) the evidence on this question disfavors this source’s objectivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s objectivity; or (d) there is no available evidence bearing on this question.
- Is there evidence about what this source expected to observe during the reported observation?
- Is there evidence about what this source wished to observe during the reported observation?
- Was this source concerned about the consequences of what this source believed during the observation?
- Is there any evidence concerning possible defects in the source’s memory? Also, how long ago did this source’s observation take place?
- Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
Observational Sensitivity (or how good was the evidence obtained by the source?)
Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s observational sensitivity; (b) the evidence on this question disfavors this source’s observational sensitivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s observational sensitivity; or (d) there is no available evidence bearing on this question.
- The source’s sensory capacity at the time of observation?
- The conditions under which the observation took place?
- The source’s track record of accuracy in previous reports?
- Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
- Are there collateral details in this report that reflect the possibility of this source’s inaccuracy?
Using the Questions
According to the authors, the twenty-five questions above have been implemented in a system called MACE (or Method for Assessing the Credibility of Evidence) that apparently has been under development for some time (I wonder if MACE was fully funded by CIA; if so, do I hear FOIA request?). The remainder of the paper describes the MACE system and how it works. For the purposes of this post, it is sufficient to point out that MACE is an evidence marshalling tool. That is, MACE provides a structured set of questions that enables the analyst to make sense of the evidence bearing on a particular source’s competence and credibility.
In addition to providing an answer to each of the twenty-five questions, MACE insists that the analyst judge the relative importance of each question involving a particular situation and a particular report. Morever, MACE asks the following two questions:
- On balance, does the evidence favor or disfavor the source’s competence, veracity, objectivity, and observational sensitivity, keeping in mind the number of questions that remain unanswered?
- On balance, how strongly does the accumulated evidence favor or disfavor our believing of the report this source has just given us, keeping in mind the number of questions that remain unanswered?
Why Care?
According to the standards for analytic tradecraft articulated in Intelligence Community Directive 203 (ICD 203), all intelligence products must “properly describe the quality and reliability of underlying sources” (section D.4.e.(1)). [Note that the standard in section D.4.e.(2) is also very important, that is, "properly caveats and expresses uncertainties or confidence in analytic judgments." But I will defer this discussion until a bit later.] What Schum and Morris provide is a means for arriving at meaningful statements of source competence and credibility that simply were not available in a documented form prior to publication of this paper.
And why do I, as a risk (not necessarily intelligence, though I can play the part) professional think this is important? Well, most (if not all) security risk analyses rely mostly on the opinions of subject matter experts, organizational representatives, etc. (i.e., humans) for the information needed to make a judgment about threat, vulnerability, and risk. Much like in intelligence analysis, risk analysts must carefully appraise the information used to support analysis in terms of both its content and its source so as to ensure that the product is free of unintended bias and influence.
2
PM
Will,
Very good summary of source vetting.
On another subject, your last paragraph raises an issue of relevance to another discussion.
You clearly differentiate between “risk analysis,” and intelligence analysis.”
What does “intelligence” mean?
This is not a trick question. I’m just curious as to your definition of intelligence, based on the clear differentiation that you made.
Thanks.
Kent
kent@cameronhalifax.com
4
AM
I can’t help but link to Kris Wheaton’s blog post on assessing the reliability of internet sources. In particular, I would draw all reader’s attention to Dax Norman’s thesis on the subject (at the National Defense Intelligence College) and his checklist for assessing web source reliability.
4
PM
Kent: I can’t say what intelligence means. But I can say what risk analysis means (see my other blog posts). From my experience, intelligence people sometimes do risk analysis work (whether they realize it or not), but at other times they do something other than risk analysis (e.g., order of battle). So when I say I can play the intelligence part, I mean that I know how to do analysis to answer questions in general, but I am better known for doing it in the context of risk analysis.
I always viewed “intelligence analysis” as the business of answering questions of import to decision makers. I suppose I might differentiate intelligence analysis from engineering analysis in that intelligence analysis does what it does in pursuit of achieving some level of favorable informational asymmetry over an opponent, whereas engineering analysis is in support of a design decision for which there is no opponent, at least not one that dominates the decision process at the engineering level. Political risk analysis has elements of intelligence analysis too, but again there is no direct opponent here save for competitors interested in similar analyses to inform their investment decisions. Now if the competitors were in direct competition, perhaps then it might be labeled as intelligence analysis.
But in the end, I just want to do good work for my client. So regardless of whether there is a clear definition or not, I still will do my darndest to produce actionable knowledge for decision makers. And if they also want me to take action to (ethically) screw up an opponent’s decision process (i.e.,create asymmetry by increasing opponent uncertainty), then that makes my job all the more interesting…
2
AM
re: ….So regardless of whether there is a clear definition or not, I still will do my darndest to produce actionable knowledge for decision makers. And if they also want me to take action to (ethically) screw up an opponent’s decision process (i.e.,create asymmetry by increasing opponent uncertainty), then that makes my job all the more interesting…
Brilliantly put! I am currently dealing with a small scale assymetry by just plain idiots: idiots in emergency services (LEO) how they can impact a community by volatile pre-emptive announcements—with mistatement and error. What is even more interesting (!) is how some folks KNOW this person —she has titled authority and is a sworn LEO— apparently cries “Wolf!” alot. She keeps her job, others in authority stay mum and her critical announcements that they know SOUND very urgent at the outset, are not critical nor urgent. And, when the announcements are revisited weeks later (AGAIN, for publication to the community)—and, at the last moment when corrections cannot be done to ‘fix’ the prior misinformation and the havoc it has caused, the basic facts are changed. The changes are so huge, the result is a recognizible hoodwink. GOOD people are hurt by this misinformation tactic. But because Idiots Lead Idiots, YEARS of crap continue on and on.
YOU are an amazing writer. I wish your logic and clarity in critical thinking could be an everyday classroom challenge: say, a tip a day. (Seriously, from age 3 on up. I did this with my son and he is a debating champion!! His veracity is uncommon, as well. Lots of street cred.)
I have taught numerous EMS and MDs cardiology, for example, and critical thinkng is an element MANY folks just are not receiving anymore in life-saving training, let alone, daily life and its ins/outs.
Thanks–You rock! MH
8
PM
Useful and very applicable to anyone service receiving broad or specific information from random and known sources. The Link to from Will McGill is also useful for those sourcing internet information.
Keep up the good work. Hindering or stopping the opponent is a great source of satisfaction. I agree.