Lecture 12 gave the students an opportunity to apply the concepts of probability covered previously in lecture 10. Basically, I developed a spreadsheet with fictional flight information spanning two years. We considered four daily flights from Dulles International Airport (Washington, DC) into State College, PA. For each day spanning two calendar years, I noted whether the flight took off on-time (blank cell), was delayed, or was cancelled (i.e., infinitely delayed). Given this information, I asked the students to calculate such things as the probability of delay, probability of cancellation, probability of cancellation of a particular flight during Winter, and various other examples that tested students’ ability to apply different concepts from probability theory (Bayes’ rules, conditional probability, theorem of total probability, and so on). I also insisted that the students use Miscrosoft Excel for their calculations.
Overall, the exercise went well. Out of 8 problems assigned, many students completed the first five. Unfortunately, I made the largely incorrect assumption that my students were fluent in MS Excel. Next time I run this exercise, I will be sure to be more mindful of what my students actually do know about spreadsheets. Perhaps I will make this lecture into an “applied probability and introduction to Excel” lecture. What I did do was allow the students to finish the assignment at home and turn it in for extra credit during the next class. 4 of 39 students ended up turning something in for extra credit. I wish this figure was higher, but what can I do other than make it mandatory? (which was not an option given all the work students already have to do for me).
In my view, lecture 9 was one of the best so far this semester. We began class with a short (10 minute) quiz covering the assigned readings and topics from lecture 7. The reading for the day was “Performing a Project Pre-Mortem” by Gary Klein (published in the September 2007 issue of Harvard Business Review, pp. 18-19). On the quiz I asked students to describe a process for conducting a pre-mortem analysis. Additionally, I asked the students to described the information used by the author to make his case for the use of pre-mortem analysis (ANSWER: scientific findings and anecdotes). The multiple-choice portion covered such things as cardinality, power sets, cross products, and generalization/specialization. I think it went well.
The book of the day (which I just ordered just before lecture, actually) was a longer piece by Gary Klein entitled The Power of Intuition: How to Use Gut Feelings to Make Better Decisions at Work (2007, ISBN: 978-0385502894). This book was cited by the pre-mortem article, and without knowing much more about it, I assume that this book covers the idea of “prospective hindsight” in greater depth (for more on “prosepective hindsight,” check out this article if you have access to it). If not, the book is still a Gary Klein book, which at the very least suggests that it would be a good and worthwhile read.
The only really teaching I did in this lecture centered on one slide where I showed the following equation for risk:
I used this equation as a basis for reviewing how much we already learned about risk, and to talk about the techniques we learned to assess parts of it. Personally, I was surprised at how much we already covered. The goal for lecture 9 was to talk more about vulnerability analysis, in particular the use of pre-mortem analysis to understand the weaknesses present in a system that might cause undesirable impacts. The backdrop the in-class exercise was campus security. We began by reading a case study on the 2007 Virginia Tech shootings prepared pro bono by my friend Thomas Shreeve of the Intelligence Community Case Methods Program. This case study was used to create a frame of reference for a pre-mortem analysis centered on the same event occurring at Penn State. In particular, I asked each group to articulate an appropriate scenario (pairing of outcome and event) and use this as the basis for brainstorming reasons why this scenario occurred at Penn State. These reasons, if correct, reveal system weaknesses, or rather vulnerabilities, that contribute to the potential for the scenario occurring.
Lecture 5 was one of my favorites. The topic was structured brainstorming, in particular the divergent/convergent thinking technique described in both the CIA and DIA analytic tradecracft primers (both of which are unclassified, and can be obtained by joining IAFIE, contacting the public relations offices of the respective agencies, etc.). I teach structured brainstorming in my risk analysis course because, as often cited by risk scholars, the first step in any risk analysis is to imagine (read “brainstorm”) answers to the question “what can go wrong?”
But before we got into the meat of lecture 5, we began class with a short quiz and a discussion of the day’s reading. The paper for today was entitled “The Case for ‘Risk Awareness’” by Stevyn Gibson (Security Journal, Volume 16, pp. 55-64, doi: 10.1057/palgrave.sj.8340140). As one might tell from the title and my preface to this post, the theme for the week is combating ignorance-induced vulnerability (which I argue is one of the biggest contributors to a person’s risk exposure). The quiz asked for the purpose of Gibson’s article (“purpose” being one of the eight elements of though) and sought answers to five multiple choice questions focused on relevant aspects from set theory (e.g., what the word “possible means,” Venn diagrams, conditional exhaustiveness, and the distinction between open and closed-world assumptions).
Moreover, consistent with this week’s theme of creating risk awareness, I showed off an “interesting” book that took the idea of creating risk awareness to the extreme. The title of the book was An Introduction to Planetary Defense: A Study of Modern Warfare Applied to Extra-Terrestrial Invasion by Travis S. Taylor (a.k.a. “Doc” Travis) and collaborators (2006, ISBN: 978-1581124477). An interesting book, indeed, though it is not without its flaws (some small, one or two VERY big – check out the one-star reviews on Amazon.com to see what I mean).
Now onto the meat of the lecture. The focus of lecture 5 was on a generic building security risk analysis question adapted from problem 8E of Philip P. Purpura’s text Security and Loss Prevention, 5th edition (2007, ISBN: 978-0123725257 ). The problem is shown in the SCRIBD window below.
Building on the materials from lecture 4, the aim of this class was to apply structured brainstorming to identify a complete set of security events that might take advantage of one or more observed facility weaknesses. The only technology we used for this in-class exercise was sticky notes (Office Depot brand) and empty wall space, window space, or an unused chalkboard. My strategy for this exercise was to allow 20 minutes or so for unassisted team divergence, followed by me and my teaching intern walking around the room with our own pads of sticky notes interjecting random ideas to help spark creative thinking. The activity finished with 10-15 minutes of convergence where each group was advised to settle on 5-6 broad classes of initiating security events.Of course, the event sets that the students came up with was by no means complete. However, as I advised, this is ok so long as the students articulate what events they are leaving out and for what reasons. This is the essence of a conditionally exhaustive set.
The only bad thing about this lecture was that it was the first lecture I gave at Penn State where I did not have my tablet PC available. Unfortunately, I spilled hot coffee on my tablet, and now it doesn’t work at all. The warranty doesn’t cover such damages either. This “black swan” event totally forced me to reshape how I can go about delivering future lectures. I suppose I have to use the white board and black boards more often now!
In a recent set of lectures for my SRA 311 course (risk management), I had my students run through a simple risk analysis case study derived from an unclassified 1993 Defense Intelligence College master’s thesis by then Captain David Lawrence Graves (USAF) entitled “Bayesian Analysis for Threat Prediction.” The case study consisted of a “notional” scenario supported by five items of intelligence. The case study is provided in the PDF document below.
According to the scenario statement, the Key Risk Questions center on whether it is the intent of the Revolutionary Party to attempt a hostile takeover of the US embassy in Country A, and if so, when such a takeover would be attempted. Based on the way this scenario was written, I can’t help but feel that takeover was assumed to be guaranteed given an attempt, which is not necessarily true given the nature of typical embassy defenses, less-than-perfect capabilities of adversaries, and so on. For the purpose of this case study, we did not make this assumption but did focus on “embassy takeover” as the sole outcome of concern. Presumably, the judgment of whether to evacuate depends on the perceived risks of a takeover attempt occurring in the next 72a-hours – if the perceived likeliness of this event (takeover attempted) combined with the severity of the associated consequences (successful takeover of US embassy) is uncomfortably “high,” then the US Ambassador in Country A might feel justified in ordering an evacuation of non-essential (or all) personnel and destruction/removal of all sensitive information.
The objective of this in-class exercise was to do a little but of source analysis (using Schum and Morris’ 25 questions) given the admittedly limited metadata on sources (which is typical), then to synthesize the nuggets of information provided by each “credible” source to order the four hypotheses based on relative likeliness. To do this, I recommended that the students use a suitable structured analytic technique to help them reason through the problem. In particular, I advocated the use of the Analysis of Competing Hypotheses technique, but without labeling it as such (I didn’t have time to offer a formal lecture on ACH, so I just gave them a list of steps). Once the hypotheses were ordered on the basis of likeliness, then, based on group judgment, the hypotheses were again ordered on the basis of vulnerability, or rather, the likeliness of outcome (e.g., takeover) given a particular hypothesis occurs. A complete risk picture examines all likeliness-severity pairs for each hypothesis (i.e., H1 through H4 as shown in the PDF file above). For example, a quick analysis might produce the following ordering of hypotheses in terms of likeliness of event (i.e., Pr(e)) and in terms of likeliness of a successful takeover given attempt (i.e., Pr(o|e)) such as is shown below.
In the end, the final risk analysis product should communicate what can happen (i.e., hypotheses H1 through H4), the relative likeliness of each, and the probability of the stated undesirable consequences. Combined, this information describes the total risk exposure for this problem. A complete risk summary such as this provides the decision maker with much of what he needs to know to make an evacuation decision. Oftentimes, professional analysts are tempted to reduce this complete narrative to a single statement such as the “risk of embassy takeover is high.” Well, perhaps this statement is true, and in some instances it may be appropriate to make such an aggregate judgment to further summarize the complete risk picture (i.e., … therefore, the risk is high). I tend to avoid making such statements as it starts to impose value judgments that are often best done by the decision makers themselves, and forces an aggregation procedure that often draws counter-constructive criticism. For example, is the analyst really in a position to assume a cut-off value of likeliness above which the decision maker should be concerned and below which the decision maker should not worry? (think of Cheney’s 1%-doctrine – was it the analyst that suggested a 1% cutoff value, or was it the decision maker?). Or, is the analyst really in a position to judge the severity of “takeover” with respect to the interests of the supported decision maker? Keeping with the goal of producing analysis that is as objective as possible, I stand by my suggestion to provide a complete narrative of risk (appropriately formatted, whether in text or as a table) that provides the decision maker with everything he needs to make his own judgment of risk.
(As an aside, note that I do, however, advocate for producing actionable risk assessments, which include both an assessment of risk combined with knowledge on which variables have the potential for risk reduction.)
The final step of this exercise was for each group to assign an appropriate level of analytic confidence to this assessment using either the DIA guidance (which I helped develop) or the Peterson Method, the latter being more straightforward but largely based on factors that correlate (not causal) with analytic confidence.
Unfortunately, given the limited time I had available for class that week, I could not afford more than 30-45 minutes in all for students to work on this exercise in class. The rest of the time was spent discussing the factors that contribute to analytic confidence, what reasoning is, the questions to aid in assessing the competence and credibility of human sources, as well as some review on measurement scales and formulas for risk analysis (which included a particularly interesting discussion focused on appraising a experimental risk formula and methodology currently in development by a government agency). What I forgot to address in class was the distinction between likeliness of events and confidence in analysis that, while both can be expressed using the same words or same mathematics (e.g., probability theory), mean completely different things. I will make it a point to bring this up in class next time.
