The theme for the day was set theory, and the material was largely taken from lectures 5 and 6 last semester (see: “Set Theory in Disguise“).   The main addition to last semester’s content was a more detailed discussion on the fundamentals of set theory, to include a formal definition of a sample space and set operations.  My intent this semester is to have my students practice some of these concepts for security risk analysis problems.  Unfortunately, it is really hard to find any good security risk analysis exercises – I can say this with confidence since I have every known security risk management book on my bookshelf.  The one exception are the excellent articles in Philip Purpura’s book Security and Loss Prevention (ISBN: 978-0123725257).  Despite this resource, I still find myself having to come up with a number of good homework examples.

There were two books of the day for today, both by Kevin Mitnick (reknown convicted hacker-turned-security consultant).  The titles are The Art of Deception (2003, ISBN: 978-0764542800) and the more recent The Art of Intrusion (2005, ISBN: 978-0471782667).  Both of these books provide excellent accounts of how adversaries actually think and innovate to achieve their goals.  Reading these books changed the way I think about security, and I am sure that it will have the same effect on my students.  I recommend both highly to all security professionals and students.

Next week I will delve into a few structured analytic techniques to help in the “imagination” phase of risk assessment (or as my colleague Jim Matschulat calls it, risk recognition).  This includes pre-mortem analysis (originally proposed by Gary Klein; see this article for a discussion) complemented by some divergent-convergent thinking.

Send article as PDF to

In a recent issue of my most favorite news magazine The Week, there was a short review on a book with the intruiging title Guesstimation (by Lawrence Weinstein and John Adams at Old Dominion University, ISBN 9780691129495).  In the engineering world, the best engineers are the ones skilled in “guesstimation,” or the art of quantitative approximation.  Unfortunately, few other communities cultivate professionals to become skilled guesstimators (save for actuaries, underwriters, traders, investors, and other people in the finance world).

This book is designed to teach, through example, people of all professions and backgrounds to become apprentice guesstimators (of course, you are not an expert guesstimator until you do it for problems that matter).  Moreover, this book attempts to illustrate the value of approximation.  In fact, this book does not insist at all on being accurate; rather, the book emphasizes that most practical problems require answers accurate only to within a factor of ten (read for yourself in Chapter 1).  More precision is often unnecessary, unhelpful, and frequently impossible to achieve in the amount of time and resources available to collect data and do analysis.

In my view, this book is a very good attempt to debunk misconceptions about quantification (and let me tell you, the Intelligence and Risk Analysis communities really need a few “mythbusters” to come in and rid these analytic worlds of their aversion to numbers).  In addition to teaching analysts how to employ structured analytic techniques, we should also improve their ability to guesstimate and to appreciate the value in doing so.  For one, the order of battle analysis community would benefit from improved guesstimation abilities.  My position has always been that numbers help with reasoning (such as in decision trees, event trees, etc.) even if they are only approximations or guesstimates.

What is missing from this book are many more examples that demonstrate the power of guesstimation in probabilistic analysis or risk analysis (the authors touch on four examples in their eleventh and final chapter).  For example, one could backcalculate the implicit subjective probability of occurrence for a particular threat based on the benefits and costs of a strategy aimed at, say, decreasing the risks due to a biological weapons attack (I have seen such an example in the past when I took a course on science and technology intelligence analysis, or S&TI, but can’t remember where).  We could also use guesstimation to estimate the maximum amount of money that could be spent to achieve a particular risk reduction objective in order to maintain a benefit-cost ratio of, say, one.  Perhaps this will be the goal of the authors’ second book (nudge nudge).  I will send them an email to put in such a request.

Send article as PDF to

You have to hand it to good old Ben Franklin, the nation’s first decision analyst, or at least the first decision analyst advocating the use of structured analytic techniques.  In a letter dated 1772 to his friend Joseph Priestly, Ben Franklin spoke about his “moral algebra” or “prudential algebra,” or what is perhaps one of the simplest, and most effective structured analytic technique for aiding decision making:

Dear Sir:

In the affair of so much importance to you, wherin you ask my advice, I cannot, for want of sufficient premises, advise you what to determine; but, if you please, I will tell you how.  When these difficult cases occur, they are difficult, chiefly because, while we have them under consideration, all the reasons pro and con are not present to the mind at the same time; but sometimes one set present themselves, and at others times another, the first being out of sight.  Hence the various purposes or inclinations that alternately prevail, and the uncertainty that perplexes us.

To get over this, my way is to divide half a sheet of paperby a line into two columns; writing over the one pro, and over the other con; then during three or four days’ consideration, I put down under the heads short hints of the different motives that at different times occur to me, for or against the measure.  When I have thus got them all together in one view, I endeavor to estimate their respective weights; and, where I find two (one on each side) that seem equal, I strike them both out.  If i find a reason pro equal to some two reasons con, I strike out the three.  If I judge some two reasons con, equal to some three reasons pro, I strike out the five; and thus proceeding I find at length where the balance lies; and if, after a day or two of further consideration, nothing new that is of importance occurs on either side, I come to a determination accordingly.  And through the weight of reasons cannot be taken with the precision of algebraic quantities, yet, when each is thus considered separately and comparatively, and the whole lies before me, I think I can judge better, and am less likely to make a rash step; and in fact I have found great advantage from this kind of equation, in what may be called moral or prudential algebra.

Wishing sincerely that you may determine for the best, I am ever, my dear friend, yours most affectionately.

B. Franklin

For a moment let’s assume that the “Sir” in the salutation was not directed toward Joseph Priestly, but to future decision makers in the United States government.  How would they take it?  I, for one, would heed Dr. Franklin’s advice – he is after all a noted philosopher, mathematician, scientist, artist, statesman, etc., etc.  More importantly, he played a major role (if not the largest role) as one of the founding fathers of the United States government.  So with all this, would he steer us wrong?

From the point of view of one interested in the mechanics of Franklin’s “moral algebra,” let’s examine what he advises more closely.  First, he clearly points out the cognitive limitations of most humans to not be able to entertain the whole picture in their minds at all times.  Sometimes a decision maker might be in an optimistic mood and thus see a decision option for its potential gains, and at other times a decision maker might take the pessimistic view and focus only on the potential losses or harm.  Sometimes a decision maker emphasizes reasons for, and at other times stresses reasons against a particular proposal.  The point is that, as Franklin observes, it is very difficult to be mindful of all reasons pro and con at once.  Hence the need for the use of structured analytic techniques to help decision makers reason through their decision challenges.

Second, the methodology put forth is simple and structured in that it forces each reason for or against a decision into one of two columns, either pro (for the decision) or con (against the decision) (see below for an image of such a thing that I picked up at the National Archives in Washington, DC). Dr. Franklin stresses deliberation in his method, adding items to each column “during three or four days’ consideration.”  This element of time allows one’s conscious and subconscious to continuously (and hopefully critically) evaluate his or her thinking on the particular decision at issue.  I’d imagine that Dr. Franklin would not balk at taking more time to ensure that all majors reasons pro and con are accounted for, but I would also suggest that his pragmatic nature would be mindful of the “80-20 rule.”  Once the major reasons are on paper, the decision maker then does a comparative analysis of all pros and cons, appropriately weighing each individually or in groups against items from the opposite column to see on what side of the pro-con spectrum the balance of reasons lie.  This task, of course, insists on a subjective valuation of each reason along some dimension of personal utility (which inherently includes whatever types of uncertainty, ambiguity, probability, etc. are associated with each reason).  The net result offers a tentative answer to the decision problem; as Dr. Franklin advises, the decision maker should take one or two days to sit on the results to see if anything new comes of importance on either side, such as new reasons, revised weightings, and so on.

Pro Con Paper

I beileve that Dr. Franklin doesn’t really insist that this tool (or any tool) make the answer for the decision maker.  After all, the extra time afforded to final deliberation allows the decision maker to consider the tentative result, and if displeased, seek additional arguments in opposition to the stated advice.  At the very least, this tool helps decision makers get to the heart of (or rather make explicit) their reasons for or against a particular decision.  Having the reasons explicitly stated now allows for objetive analysis as to their merit and value, and may very well help the decision maker identify the harmful biases that may be influencing their gut feelings.  In the end, though, decision makers ultimately make the decision regardless of what the tool says.  As Dr. Franklin believes, users of this tool “can judge better, and [are] less likely to make a rash step.”  In this complex world we live in, that’s all we can really hope for.

Now can this tool also help analysts in general (e.g., risk analysts, intelligence analysts), say, argue for or against a particular hypothesis?  Can this tool be applied to a finite set of competing decisions or hypotheses in the same manner, or does it insist that all decision problems be restated as yes/no questions?  At first glance and without giving it much thought, I would say “moral algebra” bears a striking resemblance to Heuer’s Analysis of Competing Hypotheses (ACH) or Chamberlin’s Method of Multiple Working Hypotheses (MMWH) with the exception that any particular column corresponds to one pro-con table (one column asks the question is a particular hypothesis true or false, and each row element states a reason where a “+” sides with true, “-” sides with false, and “0,” “NA,” or [blank] indicates no bearing either way).  In fact, some argue that the true value of ACH or MMWH is in getting all evidence that in support or against each among a large set of hypotheses on the table for the analyst to view holistically.  Of course, as a scientist, I must ask whether there is any evidence that does not refute the assertion that such a technique as moral algebra actually improves decision making.  But if I were to judge based only on Ben Franklin’s advice, I would probably take him at his word.

Send article as PDF to

Everyone says that structured analytic techniques are good things to have as part of a “Thinkers Toolkit.”  In the security risk analysis degree program at Penn State, several of my colleagues and I make every attempt to instruct our students in the proper application of and value added of using structured analytic techniques to enhance one’s ability to think clearly, carefully and rigorously through complex problems.  Unfortunately, our situations suffer from a significant setback – most of our students lack “real world” experience doing analysis for problems in the security and intelligence communities (or perhaps doing any real analysis at all for any community).  Accordingly, we often find ourselves searching for carefully constructed case studies that provide the right balance of realism and accessibility to students that may not have sufficient domain knowledge to speak credibly on any particular issue.  We desire case studies that contain enough information to allow students to define the problem, articulate alternative hypotheses, leverage evidence to establish probability distributions over a set of future alternatives and degrees of confidence in analytic judgments, do source analysis, and so on.

To date we have come across several case studies used in the intelligence community, such as those developed by Professor Francis Hughes at the National Defense Intelligence College and several of the cases authored by Thomas Shreeve as part of the Intelligence Community Case Method Program.  And fortunately for us, these case studies have proven to be moderately successful when used as part of our classes.  However, we are still in search of more case studies that walk students through a problem, asking them to apply different structured analytic techniques to enable them to draw defensible inferences from data, make judgments of risk and choose from among alternative strategies for mitigating risk, explore how different ways of communicating analytic results might influence the decision maker, and so on.  And of course, we are also interested in case studies that have a variety of alternative endings, mainly to highlight that the results of the analysis and the way its communicated does have an affect on the outcomes of a situation as well as setting the stage for later analysis.

In my pursuit of fun books to read to my kids before bedtime, I recently came across the Choose Your Own Adventure series of books that many of us enjoyed during our more youthful years.  I tried to recall my experiences reading these books, such as navigating through all the alternative storylines one can follow based on the choices made during the book (one CYOA fan actually took the time to actually develop a map of The Mystery of Chimney Rock by Edward Packard; I must admit that I was tempted to do the same).  Then a thought hit me – would it be possible to develop a CYOA book that resembled a storyline that one might encounter in a professional security or intelligence position?  In addition to providing a compelling story, such a book would, of course, provide greater depth to a problem, provide evidence, and try to be as real as possible so that readers can draw on external resources to aid them in their analysis.  Now here is the kicker – each analysis or decision node would insist that the reader apply a specific structured analytic technique to arrive at the best possible answer or decision.  Once the answer is chosen, the story will then continue.  Some decision nodes would be critical to preserving national security, whereas some others might be less so or even irrelevant to the outcome.  When used as part of a course, the analyst would then prepare written reports along the way outlining the steps they took to arrive at a judgment or decision.

As an attempt to appeal to those individuals having read and enjoyed CYOA books in the past, I decided to label this idea as “Choose Your Own Analytic Adventure” or CYOAA.  See the prototype cover I prepared for the first such book in the series shown above.  I imagine that the analytic training community could create an entire series of such analytic books spanning all aspects of interest, to include terrorism, resource allocation, HUMINT targeting and collection, counter-deception, counter-proliferation, risk analysis, post-blast investigation, cyber security, communicating to decision makers, etc.  What we would need to do this are good writers, good ideas, good researchers, and of course, good artists capable of drawing pretty maps, figures, and sketches (and perhaps permission from the CYOA people to model our books after their likeness).  Just imagine it – we could hand these books out as part of class, and not only would they provide a basis for practicing analysis, but they would also make for a good addition to one’s professional library.  And if they are truly written well, then perhaps they might also make for good recreational reading.

Send article as PDF to