I recently received some interesting reports from and published by Mike Pugh (a member of the RISKANAL email newsgroup) that talk about the use of probabilistic analysis to inform nuclear safety decisions. Both of these predate WASH-1400 by several years. The citations are below with clickable links to download the full documents:
Pugh, M. C. (1969). “Probabilistic Approach to Safety Analysis.” TRG Report 1949 (R), The Reactor Group, United Kingdom Atomic Energy Authority. DOWNLOAD HERE.
This paper gives a brief and simple description of some of the methods employed by the Steam Generating Heavy Water Reactor (SGHWR) Design Office in the application of probability theory to achieve a safe and economic design. Although this paper is specifically related to the design of SGHWRs the methods employed are of a general nature and could be equally applied to other reactor systems.
Pugh, M. C. (1971). “The Use of Probability Techniques in a Reactor Design Office.” SRS/GR/5, Safety and Reliability Directorate, United Kingdom Atomic Energy Authority. DOWNLOAD HERE.
The paper briefly describes how probability techniques were used in the Steam Generating Heavy Water Reactor (SGHWR) Design Office of the UKAEA to achieve a safe economic reactor design.
Probability techniques have proved to be a very useful design tool and generally promote a better understanding of a design. The basic techniques are of value to all members of a design team who can use them in the same way that stressing and heat transfer techniques are used, i.e., to analyse simple problems and to place the more complex problems into the right perspective so that detailed analysis can be performed by a specialist.
Lecture 12 gave the students an opportunity to apply the concepts of probability covered previously in lecture 10. Basically, I developed a spreadsheet with fictional flight information spanning two years. We considered four daily flights from Dulles International Airport (Washington, DC) into State College, PA. For each day spanning two calendar years, I noted whether the flight took off on-time (blank cell), was delayed, or was cancelled (i.e., infinitely delayed). Given this information, I asked the students to calculate such things as the probability of delay, probability of cancellation, probability of cancellation of a particular flight during Winter, and various other examples that tested students’ ability to apply different concepts from probability theory (Bayes’ rules, conditional probability, theorem of total probability, and so on). I also insisted that the students use Miscrosoft Excel for their calculations.
Overall, the exercise went well. Out of 8 problems assigned, many students completed the first five. Unfortunately, I made the largely incorrect assumption that my students were fluent in MS Excel. Next time I run this exercise, I will be sure to be more mindful of what my students actually do know about spreadsheets. Perhaps I will make this lecture into an “applied probability and introduction to Excel” lecture. What I did do was allow the students to finish the assignment at home and turn it in for extra credit during the next class. 4 of 39 students ended up turning something in for extra credit. I wish this figure was higher, but what can I do other than make it mandatory? (which was not an option given all the work students already have to do for me).
Lecture 10 was all about probability theory. I began this lecture by reintroducing the Venn Diagram and sets, and described probability of an event as the fraction of space consumed by the set in a particular Venn Diagram (however defined). I often prefer to teach probability the “geometric way” since I think it makes it easier to explain Bayes’ theorem and the Theorem of Total Probability. Next, I presented the three axioms of probability and associated corollaries (see the Wikipedia entry for more details – it is a good reference for this topic), talked about conditional probability, and proceeded into a geometric derivation of Bayes’ Theorem and the Theorem of Total Probability.
Unfortunately due to a recent coffee mishap, I did not have a suitable computer to use in support of this lecture. So, I delivered my lecture the old-fashioned way (definitely a challenge!) – I used a white board with markers. I think the lecture went ok, but perhaps it was a bit too much for my students to handle given their preparation. But alas, the concepts are important, and over time I will figure out the best way to communicate the concepts to an IST audience.
The book of the day for this lecture was Probability for Risk Managementby Hassett and Stewart (1999, ISBN: 1566983479). This book is a key reference chosen to help future actuaries prepared for their first actuary exam, Exam-P. This book is really good and worthwhile to have on any risk professional’s bookshelf.
Lecture 7 was an intense lecture (and not to mention a new one for SRA 311). The lecture began with a short quiz focused on the assigned reading “Bogus!” by Michael Pecht and Sanjay Tiku (published in the May 2006 Issue of IEEE Spectrum). This quiz sought to leverage the context of the readings (i.e., counterfeit electronics) to discuss topics from previous lectures (e.g., security context, scoping a risk study, and risk awareness). In addition, I also quizzed the students on extracting a single point of view from the paper, suggesting an alternative point of view, and then generating a question that these alternative views attempt to answer.
In attempt to provide students with advice on where to go for additional information on set theory (and probability theory, which will come up in a few lectures), the book of the day was the ever-popular Discrete Mathematics with Applications, Third Edition, by Susana Epp (2003, ISBN: 978-0534359454). This book is commonly required for the IST 230 course on discrete mathematics and Penn State. I, too, was a proud user of this book when I taught discrete mathematics at Howard Community College in Columbia, MD in Fall 2006.
The meat of lecture 7 began with a review of the probabilistic expression for objective risk and introduced the “security bow-tie.” Then we went on to discuss set inclusion, cardinality of a set, the power set, generalization and specialization, the cross product of two sets, and what I call the susceptibility function (i.e., the function that returns a zero for a particular pairing of initiating event and outcome that doesn’t make sense in the current context, or returns a non-zero value, usually 1, when such a pairing is plausible; for example, SQL injection causing physical damage to the building structure is an example of a non-sensical pairing of event and outcome in many contexts). The content from this lecture was essential for students to finish homework #2, which was a continuation of the in-class exercise began in lecture 5.
To make the topic as interesting and engaging as it could possibly get, I decided to hand out a sheet containing identified elements of an attempted event set and outcome set (see below). My goal was to apply the concepts to these example sets real-time so as to help students learn these relatively simple concepts through practice. In addition, and perhaps without the students realizing it, this in-class example was designed to get the students to start thinking about vulnerability, in particular through the use of the susceptibility function. In general, I think this approach worked ok. However, I admit that it could have gone MUCH better had I actually afforded students time to work the problems on their own. Or at the very least I should have exercised a bit more patience and gone slower through the material. Hopefully the students will use a future opportunity to question me about any material that did not come across clearly.
Hooray! As of about 6:30PM on Saturday, 20 Dec 08, I am done with SRA 311 (Risk Management) for the Fall 2008 semester! I have been thinking for several weeks now about what worked in SRA 311, what didn’t work, and what could be better from both student and instructor (and teaching assistant) perspectives. And now that class is over, I decided to take a few hours to write-about how I plan to do things differently for iteration 2 of SRA 311 in Spring 2009. For reference, I include my Fall 2008 syllabus below:
In the following I will just highlight a few of the changes I plan to make in the second iteration of SRA 311.
Revised Course Content for Iteration Two of SRA 311
Probability Theory: Looking back, I think I overstressed some less important topics, and forgot to include others I am now finding to be quite important. The most important thing I should have done was go over the basics of probability theory in detail rather than assume that my students were fully equipped to think probabilistically (I should have known better given that, at best, my students have had only a few lectures on the subject prior to taking this course). So, for the first part of the course, I plan to get serious about teaching probability theory from first principles. But of course, I will discuss this subject with respect to its place in security risk management. (btw: the book I plan to draw from is Introduction to Applied Probability by Pfeiffer and Schum, 1973, ISBN: 0125531508).
The Six Questions of Risk, Risk Triplet, and Definitions: As I did last semester, I plan to stress the six questions of risk assessment and risk management over and over again. The same holds for my repeated mention of the risk triplet and the definition of risk. This time, however, I will emphasize the risk triplet as being the set of scenario, s (which is the pair of initiating event and outcome, or s = (e,o)), probability of the scenario, p, and utility associated with the scenario, u. That is, risk is the set of all relevant ordered triples {<s,p,u>}. As for definitions, I am going to largely focus on risk as the potential for harm or loss, and thus save the more generic definition (i.e., risk as uncertainty about future events) for graduate discussions. Also, I will also stress the need for common definitions issue less as I found that such talk either goes over my students heads at best, or confuses them at worst.
Set Theory and Open vs. Closed Worlds: As I did last semester, I will spend about a week talking about sets (mutually exclusive, collectively exhaustive, conditionally exhaustive, etc.), as well as talk quite a bit about the difference between open and closed world thinking (w/ residual hypothesis). I still think that talking about open worlds (i.e., admitting the possibility of a residual hypothesis) should be introduced at the very beginning of a student’s exposure to risk and uncertainty. See my previous post on the topic. I plan to keep this lecture pretty much intact, but I do think I might add a few more security-oriented examples.
Utility Theory: To accommodate a full discussion of risk, I will be sure to spend a full lecture on the basics of utility theory, to cover what utility is, aspects of multicriteria decision analysis, risk attitudes, and so on. Last semester I spoke about utility for only 10 or so minutes, and consequently my students could not speak to the topic on the final. Though I a lot of utility theory was already covered in the prerequisite class SRA 231 (Decision Analysis), a little review couldn’t hurt.
Support Theory, Possibility Theory and Surprise: Integrated into my discussion of probability theory will be a discussion of support theory (the descriptive side of humans and probability), mention of possibility theory and its axioms, and also mention of Shackle’s theory of potential surprise. I am not clear yet how just where these discussions will occur, but they will surely happen somewhere during the first part (fundamentals) of the course.
Talk About Uncertainty: The discussion of uncertainty (aleatory and epistemic) and all its types will be presented up front with or soon after the introduction of the concept of risk. If all goes well, this will happen during lecture 2. We will also include the discussion of different types of ignorance at this time (unlike last time when it felt out of place in lecture 11).
Security Context and the Eight Elements of Thought: As I did last semester, I will introduce Liner and Paul’s Eight Elements of Thought and Intellectual Standards in great detail during the first lecture. Then, as before, the students will apply the Eight Elements and Intellectual Standards to their first Critical Article Review (CAR) assignment of Manunta’s paper “What is Security?” (published in a 1999 issue of Security Journal). The second lecture will be spent going over the Eight Elements and the Intellectual Standards as they apply to Manunta’s article, discussing the concept of a security context, and then proceed to do an in class example identifying and articulating different security contexts. This semester I want the students to be confident about the Eight Elements and Intellectual Standards by the end of their first week.
Accreditation: As part of the discussion on risk acceptance in Part III of the course, I will include a discussion of accreditation, or the practice of acknowledging that the risk associated with a protected asset is acceptable with respect to its value and purpose. I didn’t do this last semester, but now that SRA 311 is an essential part of the NSA Certificate in Risk Analysis, I figured I ought to start talking about accredidation. I will also include a discussion on standards, whether implicit (e.g., what would a normal security manager do) or explicit (e.g., contractually).
Life-Cycle Cost: When I talk about risk management this semester, I will be sure to include a discussion of life cycle cost of a risk mitigation strategy, to include maintenance costs, replacement costs, operational and procurement costs, and more interestingly, the implicit costs of decreased performance as adversary’s adapt and learn to overcome the countermeasure.
The Insurance Game: My good friend Professor Bilal Ayyub at the University of Maryland recently pointed out to me an interesting pedagogical exercise aimed at teaching undergraduates how to appreciate the role insurance plays in risk management. This semester, I plan to try out this game in the classroom to see how it works (and perhaps spend some money making cool game props, such as custom cards and so on).
Expert Elicitation and Probability Calibration: Last semester I spoke about probability, but did not talk at all about how to elicit probabilistic information when needed. This means that I also did not talk about how to calibrate personal probability judgments. Though I had every intention of talking about this during my fact finding discussion (which I also skipped over), this semester I will be sure to spend a whole lecture on the subject. I will call this lecture “Expert Elicitation and Fact Finding.”
Analytic Confidence: The discussion of analytic confidence will take place sometime in the first 5 weeks of class, probably after my discussion of conditional probability and possibility theory. Last semester I spoke about this all-too-important subject during lecture 20 – by then it was already too late for the concepts to sink in. I won’t make this mistake again.
Influence Diagrams: I am kicking myself for not discussing influence diagrams in class this past semester. Next semester I plan to not only talk about influence diagrams, but also have students use one or more software tools to draw and quantitatively analyze influence diagrams. This should be fun.
Decision Advantage: While risk analysis does promote decision advantage, I think that I will abandon this awkward phrase next semester. Instead, I will simply stick with “risk analysis informs decision making.”
Metrics and Formulas for Risk: As I did last semester, this semester I plan to cover all the relevant measurement scales and formula types one might encounter in a risk analysis methodology. But this time I will do it all in one lecture (or maybe a lecture and a half). I will also have some references to draw on this time around.
Pre-Mortem Analysis, Root Cause Analysis, and Convergent/Divergent Thinking: Last semester I ran an interesting case study focused on the 2007 shooting incident at the Virginia Tech campus. A few lectures after running this case study I figured out how to relate pre-mortem analysis and convergent/divergent thinking ot the case-study. But by the time I did this, it was already too late to solidify the connection in the student’s minds. So, this semester I plan to spend a week covering pre-mortem analysis and covergent/divergent thinking (and introduce the similar topic of root cause analysis) concurrently with running the case study. But unlike last semester’s VT case study, this semester’s focus will be on Aum Shimrikyo and the mid-1990’s sarin gas attack on the Tokyo subway. Other options might be a case study on the Khobar Towers bombing or one focused on the bombing of the Marine barracks in Beirut.
More Information Security and Crime, Less Terrorism: While terrorism is a hot topic these days (though becoming less so), I want to be sure that my course on security risk management (i.e., security in general) covers more than notional terrorists with bombs. This semester I plan to spend more time thinking about information security problems, routine criminal problems, and perhaps a little bit of personnel security/executive prevention. I will also talk about loss prevention as an idea, and also spend some time examining how safety balances and sometimes interferes with security.
CORAS, the McCumber Cube Model, and others: This semester I will start talking about a number of established security concepts and processes, to include CORAS, the McCumber Cube model, and OCTAVE, in addition to reviewing the basic concepts of the security bow tie and the swiss cheese model. But since these topics are no fun to hear about on their own, I still need to figure out a strategy for integrating them into the standard flow of course ideas. I think I figured out a way…
Certifications, Professional Societies, and Ethics: This time around, I will emphasize all the different certifications and security professional societies all throughout the semester. I plan to also integrate ethics into the curriculum in two ways – first by highlighting ethical issues as a matter of course during the semester, and to cap the course off with an ethics story-telling exercise on the last day of class (as I did this past semester, but this time it will be more structured).
Real Questions from Real Certification Exams: This semester I plan to integrate real risk management questions from either the CISSP exam, CPP or PSP exam, perhaps even the CAS Exam P (for actuaries). The goal here is to highlight that everything I teach in my class is relevant to things that matter in the professional world. I anticipate that no less than 25% of the final exam will consist of questions taken from professional exam study guides.
Assignments and Policies
Established Groups at the Beginning of the Semester: On day one I will assign all students to work in groups of my crafting. They are free to make individual trades among themselves long as the class is evenly divided into groups. These groups will work on all in-class exercises, homework, and projects together.
More Quizzes: Attendance was a big problem for me last semester. Without having the data in front of me, I estimate that, on average, only 60% of students showed up for any given lecture. So, this semester, in attempt to better prepare my class for the multiple-choice final exam, to review course material in a fast and effective way, and to take attendance, I plan to give frequent in-class multiple choice quizzes on either the assigned readings (CAR-style questions) or previous lecture’s material. Quizzes will be my means of taking attendance as well as gauging student performance.
More Organized CARs: Unlike last semester where I divided up the class so that 10-14 CARs were due at each lecture, this semester I plan to arrange the schedule such that all students work on CARs at the same time. This means five CARs, each due for all students at the same time. No make-ups. The format for these CAR assignments will be exactly the same as it was last semester.
Critical Book Reviews: Like last semester there will be two required book reviews. But unlike last semester, I will prescribe both books. The first book is Against the Gods: The Remarkable Story of Risk by Peter Bernstein, and the second book is Risk Intelligence by David Apgar. The format for these assignments will be exactly the same as before.
Homework: Ah, there will be homework assignments this semester. Homework will largely consist of preparatory exercises for quizzes. But there will be times when I ask for, say, an influence diagram, some worked problems, etc. All homework will be done in groups.
Final Course Project: This semester, the focus of the final project will be focused on building a risk assessment tool for exploring a particular risk problem of interest to real decision makers. That is, the tool is primary, and will be supported with some multi-media presentation (e.g., reports, poster, auto slide show, You Tube, etc.). While the topics have yet to be determined, I will make available 5-10 topics that groups may choose from. I am tentatively thinking about one or two on maritime piracy, one or two on lab site security, one or two on online communities, one on social engineering, one on party security, and so on. I am still eliciting ideas from people, and hope to have a list in hand by the second week of class.
Methodology Appraisal: There will be no formal methodology appraisal this semester. Rather, I will integrate a methodology review into one or two of the five CAR assignments.
Final Exam: There will be a final exam that, for the most part, will assume the same form as the exam from the Fall semester. There is some question in my mind whether to keep the CAR in the exam or if I should make the entire exam one big multiple choice test. I think I will keep my options open for the next few weeks.
Extra Credit: I always give extra credit, but never anything that amounts to more than 5% on top of a student’s final grade. This semester, extra credit was very helpful for those students who, for some reason or another, didn’t do well on the first assignments of the semester. Although I will not guarantee extra credit opportunities, I suspect that something will come up toward the middle-end of the semester. After all, it helps out those who did well on previous assignments, but not well enough to meet my cutoff values for certain letter grades. But in a perfect world there would be no need for extra credit since everyone would have already done superb routine work…
Attendance: Attendance in required. I will take attendance most of the time this semester, but not always. My policy for attendance is that I don’t give points to students for showing up to class. Rather, I take away points for not showing up to class. My plan for Spring is to implement an attendance policy that is tolerant of up to two (2) random absences, and then for apply a reduction factor to the final exam grade that is in proportion to the number of classes missed. If a student misses all classes, that student will get a zero on the final regardless of whether he or she actually takes it [in math speak, the final exam grade = actual score * (attendance days - missed days)/attendance days]. This is a hard core policy, but perfectly reasonable.
Course Materials: This semester three books will be required – two for the book reviews (see above) and one newer version of a book covering the Eight Elements of Thought. However, this semester I will also insist on using a variety of online soft-copy materials that will all be posted to the PSU course management system (i.e., ANGEL).
Office Hours: Despite having official office hours posted, either no one comes or they try to schedule a different time with me. So, my intentions this year are to have office hours by appointment only. But I also plan to do it in different environments, such as Second Life, PS3 Home, Skype, etc. It is high time I become more IST-ized.
No TA, But One Grader and One TI: Unfortunately, next semester I will not have a TA to help me along with my class. Instead I will have one undergraduate grader working for SRA 311 10 hours per week, and one teaching intern who will take part in class activities and maybe an occasional afternoon or evening event. I, personally, don’t know how I will function without a TA, but I suspose things should be ok if my grader and TI are good (which I suspect they will be).
Better Class Time and Better Room: I am by no means a morning person. This is why I am happy about having a class that begins at 11:15am instead of 9:45am. On top of that, I am pleased to find that I am moving my class from the worst room in the IST building (IST 205 with annoying tabletop Macintosh computers) to the best room in the building (IST 206 with PC laptops). I suppose that, in some may, the later class time and better room make up for the college taking away my teaching assistant.
My Blog About the Class: Next semester I intend to make more efficient use of my blog for recapping course content. The way I am going to do this, though, is to point to relevant reference materials to support learning instead of writing lecture notes from scratch after each class session (my tendency to write a lot about each lecture acted more as a deterrent to writing than I intended). I will also start to tweet about the class and integrate some other types of web communication technology (RSS feeds?).
Future Challenges
Two Sections of SRA 311: In Spring 2009, there will be two offerings of SRA 311. One of these (the larger one) will be taught by me on Tuesday/Thursday mornings. A smaller section of SRA 311 will be taught Tuesday/Thursday afternoons by Professor Dave Mudgett of IST 230-fame. What this means is that we have to coordinate our class schedules, or at least align the learning objectives for our courses.
Cybertorium in Fall 2009: Beginning Fall 2009, my understanding is that SRA 311 will be moving to the infamous IST Cybertorium, a 150+ person computer-ridden ampitheatre not at all designed for fTf (face to face). Fortunately (and by my request) the schedule should be such that the class will meet twice per week for 50-minutes in the Cybertorium, and one more time per week in smaller groups at a location somewhere away from the IST building. The astute reader will see here that I am making lecture more of just that – a lecture. My intent is to move all in-class activities to the recitation sections where students can spend an entire hour applying the things they learned in the lessons prior. To accomodate this move to the cybertorium, I will be, in a small way, treating my Spring 2008 course as a cybertorium class, focusing mostly on lectures with fewer in-class exercises. But when in-class exercises do occur, they will be extensive.
Guest Speakers: For some reason, I feel some pressure to recruit a guest speaker or two this semester. A challenge for me is to identify who would be a good speaker that can (a) entertain the students, (b) convey useful real-world insights, and (c) align his message with the learning objectives I would otherwise have to address were I giving the lecture. Any thoughts?
Epilogue
I invite interested readers to make suggestions regarding what to include, what to stress, what to omit, and what to test. I will be posting a revised syllabus to this blog within the next two weeks. Note that I reserve the right to add more to this post (either directly or via comment) as I things come to mind.
Risk, or the potential for harm or loss (a generic definition suitable for the purpose of this posting), can be characterized by the risk triplet as follows (see SRA 311 Lecture 4 Overview for an expanded discussion):
R = <e,p,o> (1)
which can be articulated as “risk is the combination of an initiating event (e), outcome of concern (o), and likeliness (p) that both will occur.” This middle likeliness term is technically the joint probability of e and o, or:
p = Pr(e,o) (2)
The joint probability in Eq. 2 is the measure of risk (where p is assessed with respect to a specified combination of e and o), whereas the risk triplet in Eq. 1 defines the notion of risk. Statements of risk are typically of the form: “the risk of outcome o due to e is p.” As an example, one might say “the risk of dying in a car accident is low,” where “low” is a linguistic phrase describing the likeliness of the joint event “car accident” and “death,” that may be equal to, say, some value between 0 and 1/3.
According to the basic rules of probability theory, the expression for p in Eq. 2 can be expanded in one of two ways, both of which are mathematically equivalent:
Pr(e,o) = Pr(e)Pr(o|e) (3)
and
Pr(e,o) = Pr(o)Pr(e|o) (4)
Despite their mathematical “equivalence,” Eqs. 3 and 4 yield different interpretations of risk. The structure of a risk assessment according to Eq. 3 follows a sequence of events starting with the initiating event, then followed by an outcome given the event has occurred. Equation 3 is the more common of the two, and conforms (or perhaps is the basis for) the prototypical Risk = Threat x Vulnerability x Consequence model. When viewed as a control variable in risk management, the vulnerability Pr(o|e) in Eq. 3 has a antiterrorism flare, that is, it measures the performance of defensive measures that seek to minimize the potential of undesirable outcomes given the occurrence of an attack of fixed description. Countermeasures aimed at reducing vulnerability seek to favorably modify the structure of the target system to render it less susceptible to loss or harm in the event of an attack.
Now let’s look at the expression in Eq. 4. This equation starts with the outcome, then proceeds to speculate on the cause of this outcome. This is essentially a backwards event-tree view of risk (from consequence to cause). The conditional probability in Eq. 4 essentially asks “given that we experienced a specific outcome, what was its cause?” When viewed as a control variable, the objective is to lessen the likeliness that the cause of loss was an attack given that losses of a significant level were realized. If the ultimate goal is to deny terrorist’s the opportunity to cause o (or convert them from being able or interested in causing o) as in counterterrorism, then Pr(e|o) is a variable that measures the likeliness of a terrorist attack relative to all other hazards that would prompt loss of a given magnitude. The goal is to deny terrorists the capability or influence their motivations, which ultimately lessens Pr(e|o) relative to other plausible events.
In summary, Eq. 3 can be taken as the antiterrorism (AT) view of risk and Eq. 4 can be taken as the counterterrorism (CT) view of risk, where the conditional probabilities in both equations are a measure of non-performance relative to AT(CT) goals. But since we think as strong antiterrorism (counterterrorism) capabilities as being a good thing, it seems more sensible to recast the conditional probabilities in a manner such that higher values (i.e., closer to one) are preferred. Thus, appealing to the second axiom of probability we have the following pair of performance measures (EAT = effectiveness of antiterrorism capabilities; ECT = effectiveness of counterterrorism capabilities):
EAT = 1 – Pr(o|e)
ECT = 1 – Pr(e|o)
As interesting as this discussion may be, whether right or wrong, the challenge of assessing any particular value is quite difficult, even at a conceptual level in terms of reasoned narratives. While in theory we are simply asking for a single number or, more sensibly, an interval within the range of zero to one, the process of reasoning from data to such a value requires the analyst to possess a significant amount of knowledge and experience.
