What I Would Do Differently? Reflections on the First Offering of SRA 311
Sunday, December 21st, 2008Hooray! As of about 6:30PM on Saturday, 20 Dec 08, I am done with SRA 311 (Risk Management) for the Fall 2008 semester! I have been thinking for several weeks now about what worked in SRA 311, what didn’t work, and what could be better from both student and instructor (and teaching assistant) perspectives. And now that class is over, I decided to take a few hours to write-about how I plan to do things differently for iteration 2 of SRA 311 in Spring 2009. For reference, I include my Fall 2008 syllabus below:
Sra311 Syllabus Fall2008 Final
In the following I will just highlight a few of the changes I plan to make in the second iteration of SRA 311.
Revised Course Content for Iteration Two of SRA 311
Probability Theory: Looking back, I think I overstressed some less important topics, and forgot to include others I am now finding to be quite important. The most important thing I should have done was go over the basics of probability theory in detail rather than assume that my students were fully equipped to think probabilistically (I should have known better given that, at best, my students have had only a few lectures on the subject prior to taking this course). So, for the first part of the course, I plan to get serious about teaching probability theory from first principles. But of course, I will discuss this subject with respect to its place in security risk management. (btw: the book I plan to draw from is Introduction to Applied Probability by Pfeiffer and Schum, 1973, ISBN: 0125531508).
The Six Questions of Risk, Risk Triplet, and Definitions: As I did last semester, I plan to stress the six questions of risk assessment and risk management over and over again. The same holds for my repeated mention of the risk triplet and the definition of risk. This time, however, I will emphasize the risk triplet as being the set of scenario, s (which is the pair of initiating event and outcome, or s = (e,o)), probability of the scenario, p, and utility associated with the scenario, u. That is, risk is the set of all relevant ordered triples {<s,p,u>}. As for definitions, I am going to largely focus on risk as the potential for harm or loss, and thus save the more generic definition (i.e., risk as uncertainty about future events) for graduate discussions. Also, I will also stress the need for common definitions issue less as I found that such talk either goes over my students heads at best, or confuses them at worst.
Set Theory and Open vs. Closed Worlds: As I did last semester, I will spend about a week talking about sets (mutually exclusive, collectively exhaustive, conditionally exhaustive, etc.), as well as talk quite a bit about the difference between open and closed world thinking (w/ residual hypothesis). I still think that talking about open worlds (i.e., admitting the possibility of a residual hypothesis) should be introduced at the very beginning of a student’s exposure to risk and uncertainty. See my previous post on the topic. I plan to keep this lecture pretty much intact, but I do think I might add a few more security-oriented examples.
Utility Theory: To accommodate a full discussion of risk, I will be sure to spend a full lecture on the basics of utility theory, to cover what utility is, aspects of multicriteria decision analysis, risk attitudes, and so on. Last semester I spoke about utility for only 10 or so minutes, and consequently my students could not speak to the topic on the final. Though I a lot of utility theory was already covered in the prerequisite class SRA 231 (Decision Analysis), a little review couldn’t hurt.
Support Theory, Possibility Theory and Surprise: Integrated into my discussion of probability theory will be a discussion of support theory (the descriptive side of humans and probability), mention of possibility theory and its axioms, and also mention of Shackle’s theory of potential surprise. I am not clear yet how just where these discussions will occur, but they will surely happen somewhere during the first part (fundamentals) of the course.
Talk About Uncertainty: The discussion of uncertainty (aleatory and epistemic) and all its types will be presented up front with or soon after the introduction of the concept of risk. If all goes well, this will happen during lecture 2. We will also include the discussion of different types of ignorance at this time (unlike last time when it felt out of place in lecture 11).
Security Context and the Eight Elements of Thought: As I did last semester, I will introduce Liner and Paul’s Eight Elements of Thought and Intellectual Standards in great detail during the first lecture. Then, as before, the students will apply the Eight Elements and Intellectual Standards to their first Critical Article Review (CAR) assignment of Manunta’s paper “What is Security?” (published in a 1999 issue of Security Journal). The second lecture will be spent going over the Eight Elements and the Intellectual Standards as they apply to Manunta’s article, discussing the concept of a security context, and then proceed to do an in class example identifying and articulating different security contexts. This semester I want the students to be confident about the Eight Elements and Intellectual Standards by the end of their first week.
Accreditation: As part of the discussion on risk acceptance in Part III of the course, I will include a discussion of accreditation, or the practice of acknowledging that the risk associated with a protected asset is acceptable with respect to its value and purpose. I didn’t do this last semester, but now that SRA 311 is an essential part of the NSA Certificate in Risk Analysis, I figured I ought to start talking about accredidation. I will also include a discussion on standards, whether implicit (e.g., what would a normal security manager do) or explicit (e.g., contractually).
Life-Cycle Cost: When I talk about risk management this semester, I will be sure to include a discussion of life cycle cost of a risk mitigation strategy, to include maintenance costs, replacement costs, operational and procurement costs, and more interestingly, the implicit costs of decreased performance as adversary’s adapt and learn to overcome the countermeasure.
The Insurance Game: My good friend Professor Bilal Ayyub at the University of Maryland recently pointed out to me an interesting pedagogical exercise aimed at teaching undergraduates how to appreciate the role insurance plays in risk management. This semester, I plan to try out this game in the classroom to see how it works (and perhaps spend some money making cool game props, such as custom cards and so on).
Expert Elicitation and Probability Calibration: Last semester I spoke about probability, but did not talk at all about how to elicit probabilistic information when needed. This means that I also did not talk about how to calibrate personal probability judgments. Though I had every intention of talking about this during my fact finding discussion (which I also skipped over), this semester I will be sure to spend a whole lecture on the subject. I will call this lecture “Expert Elicitation and Fact Finding.”
Analytic Confidence: The discussion of analytic confidence will take place sometime in the first 5 weeks of class, probably after my discussion of conditional probability and possibility theory. Last semester I spoke about this all-too-important subject during lecture 20 - by then it was already too late for the concepts to sink in. I won’t make this mistake again.
Influence Diagrams: I am kicking myself for not discussing influence diagrams in class this past semester. Next semester I plan to not only talk about influence diagrams, but also have students use one or more software tools to draw and quantitatively analyze influence diagrams. This should be fun.
Decision Advantage: While risk analysis does promote decision advantage, I think that I will abandon this awkward phrase next semester. Instead, I will simply stick with “risk analysis informs decision making.”
Metrics and Formulas for Risk: As I did last semester, this semester I plan to cover all the relevant measurement scales and formula types one might encounter in a risk analysis methodology. But this time I will do it all in one lecture (or maybe a lecture and a half). I will also have some references to draw on this time around.
Pre-Mortem Analysis, Root Cause Analysis, and Convergent/Divergent Thinking: Last semester I ran an interesting case study focused on the 2007 shooting incident at the Virginia Tech campus. A few lectures after running this case study I figured out how to relate pre-mortem analysis and convergent/divergent thinking ot the case-study. But by the time I did this, it was already too late to solidify the connection in the student’s minds. So, this semester I plan to spend a week covering pre-mortem analysis and covergent/divergent thinking (and introduce the similar topic of root cause analysis) concurrently with running the case study. But unlike last semester’s VT case study, this semester’s focus will be on Aum Shimrikyo and the mid-1990’s sarin gas attack on the Tokyo subway. Other options might be a case study on the Khobar Towers bombing or one focused on the bombing of the Marine barracks in Beirut.
More Information Security and Crime, Less Terrorism: While terrorism is a hot topic these days (though becoming less so), I want to be sure that my course on security risk management (i.e., security in general) covers more than notional terrorists with bombs. This semester I plan to spend more time thinking about information security problems, routine criminal problems, and perhaps a little bit of personnel security/executive prevention. I will also talk about loss prevention as an idea, and also spend some time examining how safety balances and sometimes interferes with security.
CORAS, the McCumber Cube Model, and others: This semester I will start talking about a number of established security concepts and processes, to include CORAS, the McCumber Cube model, and OCTAVE, in addition to reviewing the basic concepts of the security bow tie and the swiss cheese model. But since these topics are no fun to hear about on their own, I still need to figure out a strategy for integrating them into the standard flow of course ideas. I think I figured out a way…
Certifications, Professional Societies, and Ethics: This time around, I will emphasize all the different certifications and security professional societies all throughout the semester. I plan to also integrate ethics into the curriculum in two ways - first by highlighting ethical issues as a matter of course during the semester, and to cap the course off with an ethics story-telling exercise on the last day of class (as I did this past semester, but this time it will be more structured).
Real Questions from Real Certification Exams: This semester I plan to integrate real risk management questions from either the CISSP exam, CPP or PSP exam, perhaps even the CAS Exam P (for actuaries). The goal here is to highlight that everything I teach in my class is relevant to things that matter in the professional world. I anticipate that no less than 25% of the final exam will consist of questions taken from professional exam study guides.
Assignments and Policies
Established Groups at the Beginning of the Semester: On day one I will assign all students to work in groups of my crafting. They are free to make individual trades among themselves long as the class is evenly divided into groups. These groups will work on all in-class exercises, homework, and projects together.
More Quizzes: Attendance was a big problem for me last semester. Without having the data in front of me, I estimate that, on average, only 60% of students showed up for any given lecture. So, this semester, in attempt to better prepare my class for the multiple-choice final exam, to review course material in a fast and effective way, and to take attendance, I plan to give frequent in-class multiple choice quizzes on either the assigned readings (CAR-style questions) or previous lecture’s material. Quizzes will be my means of taking attendance as well as gauging student performance.
More Organized CARs: Unlike last semester where I divided up the class so that 10-14 CARs were due at each lecture, this semester I plan to arrange the schedule such that all students work on CARs at the same time. This means five CARs, each due for all students at the same time. No make-ups. The format for these CAR assignments will be exactly the same as it was last semester.
Critical Book Reviews: Like last semester there will be two required book reviews. But unlike last semester, I will prescribe both books. The first book is Against the Gods: The Remarkable Story of Risk by Peter Bernstein, and the second book is Risk Intelligence by David Apgar. The format for these assignments will be exactly the same as before.
Homework: Ah, there will be homework assignments this semester. Homework will largely consist of preparatory exercises for quizzes. But there will be times when I ask for, say, an influence diagram, some worked problems, etc. All homework will be done in groups.
Final Course Project: This semester, the focus of the final project will be focused on building a risk assessment tool for exploring a particular risk problem of interest to real decision makers. That is, the tool is primary, and will be supported with some multi-media presentation (e.g., reports, poster, auto slide show, You Tube, etc.). While the topics have yet to be determined, I will make available 5-10 topics that groups may choose from. I am tentatively thinking about one or two on maritime piracy, one or two on lab site security, one or two on online communities, one on social engineering, one on party security, and so on. I am still eliciting ideas from people, and hope to have a list in hand by the second week of class.
Methodology Appraisal: There will be no formal methodology appraisal this semester. Rather, I will integrate a methodology review into one or two of the five CAR assignments.
Final Exam: There will be a final exam that, for the most part, will assume the same form as the exam from the Fall semester. There is some question in my mind whether to keep the CAR in the exam or if I should make the entire exam one big multiple choice test. I think I will keep my options open for the next few weeks.
Extra Credit: I always give extra credit, but never anything that amounts to more than 5% on top of a student’s final grade. This semester, extra credit was very helpful for those students who, for some reason or another, didn’t do well on the first assignments of the semester. Although I will not guarantee extra credit opportunities, I suspect that something will come up toward the middle-end of the semester. After all, it helps out those who did well on previous assignments, but not well enough to meet my cutoff values for certain letter grades. But in a perfect world there would be no need for extra credit since everyone would have already done superb routine work…
Attendance: Attendance in required. I will take attendance most of the time this semester, but not always. My policy for attendance is that I don’t give points to students for showing up to class. Rather, I take away points for not showing up to class. My plan for Spring is to implement an attendance policy that is tolerant of up to two (2) random absences, and then for apply a reduction factor to the final exam grade that is in proportion to the number of classes missed. If a student misses all classes, that student will get a zero on the final regardless of whether he or she actually takes it [in math speak, the final exam grade = actual score * (attendance days - missed days)/attendance days]. This is a hard core policy, but perfectly reasonable.
Course Materials: This semester three books will be required - two for the book reviews (see above) and one newer version of a book covering the Eight Elements of Thought. However, this semester I will also insist on using a variety of online soft-copy materials that will all be posted to the PSU course management system (i.e., ANGEL).
Office Hours: Despite having official office hours posted, either no one comes or they try to schedule a different time with me. So, my intentions this year are to have office hours by appointment only. But I also plan to do it in different environments, such as Second Life, PS3 Home, Skype, etc. It is high time I become more IST-ized.
No TA, But One Grader and One TI: Unfortunately, next semester I will not have a TA to help me along with my class. Instead I will have one undergraduate grader working for SRA 311 10 hours per week, and one teaching intern who will take part in class activities and maybe an occasional afternoon or evening event. I, personally, don’t know how I will function without a TA, but I suspose things should be ok if my grader and TI are good (which I suspect they will be).
Better Class Time and Better Room: I am by no means a morning person. This is why I am happy about having a class that begins at 11:15am instead of 9:45am. On top of that, I am pleased to find that I am moving my class from the worst room in the IST building (IST 205 with annoying tabletop Macintosh computers) to the best room in the building (IST 206 with PC laptops). I suppose that, in some may, the later class time and better room make up for the college taking away my teaching assistant.
My Blog About the Class: Next semester I intend to make more efficient use of my blog for recapping course content. The way I am going to do this, though, is to point to relevant reference materials to support learning instead of writing lecture notes from scratch after each class session (my tendency to write a lot about each lecture acted more as a deterrent to writing than I intended). I will also start to tweet about the class and integrate some other types of web communication technology (RSS feeds?).
Future Challenges
Two Sections of SRA 311: In Spring 2009, there will be two offerings of SRA 311. One of these (the larger one) will be taught by me on Tuesday/Thursday mornings. A smaller section of SRA 311 will be taught Tuesday/Thursday afternoons by Professor Dave Mudgett of IST 230-fame. What this means is that we have to coordinate our class schedules, or at least align the learning objectives for our courses.
Cybertorium in Fall 2009: Beginning Fall 2009, my understanding is that SRA 311 will be moving to the infamous IST Cybertorium, a 150+ person computer-ridden ampitheatre not at all designed for fTf (face to face). Fortunately (and by my request) the schedule should be such that the class will meet twice per week for 50-minutes in the Cybertorium, and one more time per week in smaller groups at a location somewhere away from the IST building. The astute reader will see here that I am making lecture more of just that - a lecture. My intent is to move all in-class activities to the recitation sections where students can spend an entire hour applying the things they learned in the lessons prior. To accomodate this move to the cybertorium, I will be, in a small way, treating my Spring 2008 course as a cybertorium class, focusing mostly on lectures with fewer in-class exercises. But when in-class exercises do occur, they will be extensive.
Guest Speakers: For some reason, I feel some pressure to recruit a guest speaker or two this semester. A challenge for me is to identify who would be a good speaker that can (a) entertain the students, (b) convey useful real-world insights, and (c) align his message with the learning objectives I would otherwise have to address were I giving the lecture. Any thoughts?
Epilogue
I invite interested readers to make suggestions regarding what to include, what to stress, what to omit, and what to test. I will be posting a revised syllabus to this blog within the next two weeks. Note that I reserve the right to add more to this post (either directly or via comment) as I things come to mind.
