In his book Political/Military Applications of Bayesian Analysis: Methodological Issues (ISBN: 0-86531-945-5), Dr. Douglas Hunter (a good friend of mine) outlines a set of 57 problems, which I label as “Hunter’s 57 Problems” (much like Heinz’s 57 Varieties), often encountered in the application of Bayesian analysis.  These 57 problems comprise the bulk of Table of Contents of the book from Chapter 4 onward, and is provided in list form below for future reference.  The book gives an example and offers strategies for overcoming each problem. (Note that I have my own strategies for dealing with some of these problems, and I do not necessarily agree with everything here; but I will defer discussion of these for later posts).

1. Hypotheses not mutually exclusive and/or not exhaustive: the problem of hypotheses which are defective because they are not mutually exclusive and/or not exhaustive.
2. A recognizably defective residual hypothesis: the problem of a final hypothesis which is recognizably too inclusive, meaningless, and “catch-all.”
3. “Buried” subhypotheses: the problem of unrecognized subhypotheses in a hypothesis (often, but not always, the residual hypothesis), which render the hypothesis somewhat meaningless.
4. The optimal number of hypotheses: the question of how many hypotheses are optimal to use in a given situation.
5. Multiple factors in hypotheses: the problem of how to develop hypotheses when we are dealing with multiple factors.
6. Time and “sliding windows”: the problem is phrasing hypotheses of whether to use a fixed time limit in the future or a “sliding window,” or even to mention the factor of time.
7. Time periods in “sliding windows”: the problem of the appropriate “sliding window” time period to use in hypotheses about decisions that have been made to do something in the future.
8. What types of questions can Bayes answer? the question as to what sorts of questions Bayes can answer and, in particular, as to whether it is reasonable to try to use Bayes to predict either future decisions by actors (such as states) or future events.
9. How far in the future can Bayes predict? the question of how far in advance can we use Bayes to predict either eventual future decisions or future states of affairs.
10. Incorrect results from using conditional independence: the problem of mathematically invalid results because of the use of the conditionally independent form of Bayes rather than the conditionally dependent form of Bayes.
11. Confusing “conditional independence” and “distinct”: the problem of incorrectly equating the concepts of “conditional independence” and “distinct” items of evidence.
12. Results affected by the order of evidence: the problem that the order in which we consider the items of evidence is itself an important factor in determining the revised probabilities of hypotheses when we use the conditionally dependent form of Bayes.
13. The types of evidence to use: the question of whether to use “all” relevant evidence or “just” indicator relevant evidence.
14. Previous data: the problem of the type of “previous data” with respect to previous indicator settings to use and which prior probabilities of hypotheses to use if we choose to use indicators.
    
15. “Causal” evidence: the problem of how to deal with actions which are are intended to alter behavior.
16. The one-sided view: the problem of the one-sided view of a complex interaction process that results if we do not utilize “causal” evidence external to the environment in our Bayesian analyses.
17. Predicting with “mixed” items of evidence: the problem of trying to predict a future event when evidence is a mixture of (1) items of evidence which indicate the state of policy at present and (2) items of evidence which may be the “causes” of possible policy change in the future.
18. Recognizing invalid “evidence”: the problem of identifying “evidence” not really from the environment.
19. “Statistical” evidence: the problem of evidence not really about an event or person, but actually about the class to which the event or person belongs.
20. “Negative” evidence: the problem of the need to consider items (events) of non-occurrences of events in particular time periods.
21. Developing lists of negative evidence: the problem of how to alert analysts to the need to consider negative evidence and how to develop lists of items of negative evidence for a particular set of hypotheses.
22. Deciding the relevance of data: the problem of deciding which items of data clearly are “relevant,” or “peripheral,” or “irrelevant” with respect to hypotheses.
23. The probability of reports vs. the probability of evidence: the problem of deciding when to ask conditional probability that the report (of certain evidence) will occur rather than asking the usual conditional probability that the evidence (contained in the report) will occur.
24. How many events to abstract? the problem of how many events, i.e., items of evidence, to abstract from a given report if we decide that it is units of evidence for which we wish to assess probabilities rather than reports.
25. “Double counting” evidence: the problem  of overweighting items of evidence by assessing the same item each time it is reported.
26. Different characterizations of the true data state: the problem of assessing the probability of an event when there are multiple reports on the event which do not characterize the event the same, but rather characterize the event in a way that is wither complementary (supplementary) or contradictory in nature; in short, the problem of how to handle different characterizations of the true data state.
27. How often to count “negative” evidence: the problem of how often to input a particular item (event) of “negative” evidence.
28. The absence of data: the problem of how to deal with the absence of data about a particular event for a certain period of time.
29. No observations of an event: the problem of how to deal with the item of evidence that a particular event has not been observed in a particular time period.
30. Source reliability: the problem of how to take into account the reliability of a source of information.
31. Assessing source reliability: the problem of how to assess the Zlotnick exponent.
32. New reports and source reliability: the problem, if Zlotnick’s exponent is used, of what to do when new reports reflecting the true data state are received.
33. Multiple reports and source reliability: the problem, if there are multiple reports on the true data state, of whether to include the source reliability for each report, and, if so, how to do so.
34. Unbelievable evidence: the problem of how to deal with evidence which does not appear believable.
35. Phrasing conditional probability judgments: the problem of how to verbalize the conditional probability of events.
36. “Blow-up”: the problem of an unwarranted rapid increase or decrease in the revised probabilities of some hypotheses.
37. “Error bounding”: the problem of the high sensitivity of the revised probabilities of hypotheses to minor variations in the assigned probabilities.
38. The importance of prior probabilities: the problem of the impact of the prior probabilities of the hypotheses if there are a few items of evidence.
39. The importance of ratios: the problem of the importance of the ratios when assigning the conditional probabilities of events occurring.
40. Probabilities or odds: the problem of whether to assign probabilities or odds.
41. Group assessments of probabilities: the problem of how to deal with a group assessment of probabilities.
42. Difficulties in thinking in terms of probabilities: the problem where some analysts think easily in terms of probabilities, others need to work at it every time, and a few need constant attention and retraining to overcome a distorted or unrealistic feeling for probabilities.
43. Lack of analyst expertise: the problem of the lack of necessary expertise to assess the probability of events for some hypotheses.
44. Logically “distant” events: the problem of an event logically “distant” from one or more hypotheses, which makes it difficult or impossible to assess a meaningful conditionally dependent probability for those hypotheses.
45. Systematic bias: the problem of systematic bias – even among sophisticated and knowledgeable – in the probabilities that are assigned to events.
46. Idiosyncratic and situational biases: the problem of biases that only certain people have because of personal, idiosyncratic factors, or because of the special situation.
47. Conscious manipulation of probabilities: the problem where an analyst may manipulate consciously his(her) assigned probabilities to support a favored hypothesis.
48. Trend line biases: the problem of analyst biases emerging from paying attention to trend lines.
49. Time-pressure difficulties: the problem that the time required to do a Bayesian analysis may make it difficult to use it in a crisis situation.
50. Fluctuating revised probabilities: the problem of revised probabilities of hypotheses fluctuating up and down, as each piece of conflicting data, reflective of inconsistent or “hedging” government policies, is processed.
51. Non-stationarity of hypotheses: the problem of a change in the state of nature.
52. “Life-span” of evidence (or “impact of past information”): the problem of when and why and to what extent the probability judgments for “old” items of evidence should be deleted from the analysis (and whether the old evidence should be considered in making probability judgments for new items of evidence)
53. Revising previously assigned probabilities: the problem of deciding under what circumstances previously assigned probabilities of events should be revised.
54. New prior probabilities: the problem of when and why to abandon the original prior probabilities and start a new analysis with new priors.
55. The “reliability” of Bayes: the problem of the “reliability” of Bayesian analysis as a function of proximity to the actual occurrence of one of the hypothesized events and whether or not we can improve reliability.
56. Differences between intuitive and Bayesian predictions: the problem of how to deal with differences (particularly drastic differences between Bayesian predictions and predictions based on personal experience), when you are “sure” your analysis is better than the one based on Bayes.
57. Consumer bias: the problem of consumer bias, particularly either gullibility or cynicism.

I must add that the first chapter of this book offers a neat and concise introduction to probability (not statistics) for intelligence analysts.  In the third chapter, the Dr. Hunter also offers what he views to be 9 major advantages of employing Bayesian analysis for intelligence:

1. Use of Bayesian analysis overcomes the conservative bias it is said we supposedly have regarding our revision of initial probabilities in light of evidence.
2. More information can be extracted from the body of available data because the technique calls for each piece of evidence to add its weight to the final assessment in a systematic way.
3. The technique compels us to employ an improved system of accounting for the evidence used.
4. If different analysts reach different conclusions, the source of the disagreements can be determined more easily than in a normal, verbal analysis.
5. Using Bayes forces us to consider alternative hypotheses.
6. As a related advantage to No. 5, use of Bayes decouples the analyst’s ego from the probability figures assigned.
7. When using Bayes, we are making deductive judgments when we assess the probability of evidence given the truth of a hypothesis and all previous evidence.  Deductive judgments are easier to make than inductive judgments, i.e., the probability of a hypothesis given evidence.
8. We are required to quantify, i.e., make explicit, and get away from the ambiguity of words, judgments which we do not ordinarily express in numerical terms.
9. The revised probability of the hypotheses can change very quickly.

Now, I am not necessarily an advocate for insisting that all analysts use Bayesian analysis in their thinking (though there are plenty of new tools available for free that greatly facilitate the use of Bayesian analysis, such as Microsoft Bayesian Editor and Toolkit).  But I do encourage analysts, if time and energy permits, to diagram their arguments and assign probabilities to events for the purpose of facilitating thinking and diagnosing errors in reasoning and judgment.  Done well, Bayesian analysis also permits sensitivity analysis on reasoning, which would help identify linchpin items of evidence or evidence that, if strengthened or weakened, would cause significant change in judgment or confidence.  Unfortunately, it takes quite a bit of training to become used to Bayesian analysis, and for this reason it will be part of only the most mathematically inclined intelligence analyst.

If one looks at Dr. Hunter’s book as not a text advocating Bayesian analysis, but as a self-help book for Bayesian thinkers (which we all are to a degree), then the 57 problems identified in this book are relevant regardless of whether one uses mathematical formulas to make explicit their thinking.  I highly recommend finding the book somewhere online or in the library and sitting down with it for a few hours to soak in all it has to say.  After all, at the time of the book’s writing in 1984, Dr. Hunter had already accumulated decades worth of experience successfully applying Bayesian analysis to challenging political and military analysis.  And if you ever had the opportunity to sit in a class with him (which I have when I took a nighttime course in the “Analysis of Competing Hypotheses”), you will come to understand that he has seen it all when it comes to both good and bad Bayesian analysis.

But before I conclude, I must point out the existence of several published reviews of this book, including the less-than-favorable one by Walter W. Hill, Jr. published in The American Political Science Review, Vol. 79, No. 2, pp. 615-616, 1985 (permalink here), and the more favorable review by Dina Zines as published in the Annals of the American Academy of Political and Social Science, Vol. 479, pp. 159-160, 1985 (permalink here).  Note that you need institutional access to JSTOR to read these reviews (or pay $9 per review).

Used copies of Dr. Hunter’s book are often available on Alibris.com for about $50.  If somehow was able to get the publisher to release this title to the public domain, I would readily make available a PDF file of the book for all to view and download (and so would Google books).  I am sure the author wouldn’t mind.

Send article as PDF to

This post is focused on a neat, and FREE, Windows-based tool called Constructor which is available for download at http://www.ramas.com/constructor.htm.  Constructor was developed by Applied Biomathematics, Inc., (AB) a risk and uncertainty modeling and analysis firm located in Setauket, NY (Suffolk County, Long Island).  One of my good friends Scott Ferson, an author of numerous highly-cited publications in the field of risk analysis and information science and an employee of AB, introduced me to this software a few months ago, and since then I found it quite useful for my dissertation, research, and so on.

Constructor is a Windows-based tool for constructing probability boxes or p-boxes for a given random variable according to a set of constraints informed by a sample of data, expert judgment, or what have you.  For those unfamiliar with the concept of a p-box, think of it as a region on the X-Y axis bounded by two cumulative distribution functions (“CDFs” or “S-curves”) that contains an entire family of CDFs with similar properties.  That is, rather than trying to specify a precise CDF with, say a given mean and variance (e.g., using maximum entropy arguments), a p-box contains the complete family of CDF curves defined according to specified constraints (e.g., same mean, variance).  For example, consider the following set of constraints for a unitless random variable X:

• Mean: 500
• Variance: 10000
• Limits: [0 2000]

These constraints yield the following p-box, that rigorously contains a complete family (and more, of course) of CDFs whose corresponding probability density functions have a mean of 500, coefficient of variation of 20%, and is bounded by 0 and 2000.  In the figure below, the green curve represents the lower limit on the CDF-value at a given value X (e.g., X=600), and the yellow curve represents the upper value.  If this p-box represented the visualization of a Dempster-Shafer structure, the green curve would be the cumulative belief function and the yellow curve would be the cumulative plausibility function.  The assumption here is that, if the constraints imposed on the CDF are valid, the p-box is sure to contain the “true” distribution of a random variable, even if we can never know exactly what this true distribution is.

The really nice thing about this tool, and p-boxes in general, is that they rigorously express, in a useful quantitative manner, what is known about a variable, even if it is very little.  A p-box can be propagated through any mathematical function as if it were a probability distribution.  Moreover, p-box mathematics accomodates uncertainty in the dependency between two or more random variables combined via some arithmetic operation.  For example, if Z = X + Y, where X and Y are random variables represented by probability distributions, p-boxes, Dempster-Shafer structures, etc., p-boxes can be used to calculate Z regardless of whether the nature of dependency between X and Y is known.  And the best thing about p-boxes is that is makes no false assumptions of the true form of a distribution, such as might be the case is one accepts maximum entropy arguments and leverages a maximum entropy distribution shape, or if one defaults to assuming independence between two or more variables when there is no firm basis for settling on any statement of dependence.

The only problem I have with Constructor is its quirky nature when entering incompatible constraints.  A bad constraint relative to others previously entered has the potential to cause an excessive amount of computer beeping (similar to holding the left arrow down too long when using MS Excel) with annoying pop-up boxes containing messages that make little sense to the casual user.

But that problem aside, Constructor is a cool tool for getting acquainted with p-boxes and for generating ascii data files specifying p-boxes for use with other mathematics software, such as MATLAB and Excel.  I encourage you to download the software and check it out (that is, if the concept of a CDF makes sense to you; if it does not, then you are probably better off spending your time reading a book on probability and statistics).

For more information on probability boxes, consult the following references:

• Constructing Probability Boxes and Dempster-Shafer Structures (SAND 2002-4015)
• Dependence in Probabilistic Modeling, Dempster-Shafer Theory, and Probability Bounds Analysis (SAND 2004-3072)
• Constructor: Synthesizing Information and Uncertain Variables (SAND 2005-3769)
• Sensitivity in Risk Analysis with Uncertain Numbers (SAND 2006-2801)
• Experimental Uncertainty Estimation and Statistics for Data Having Interval Uncertainty (SAND 2007-0939)

Send article as PDF to