In my view, lecture 9 was one of the best so far this semester.  We began class with a short (10 minute) quiz covering the assigned readings and topics from lecture 7.  The reading for the day was “Performing a Project Pre-Mortem” by Gary Klein (published in the September 2007 issue of Harvard Business Review, pp. 18-19).  On the quiz I asked students to describe a process for conducting a pre-mortem analysis.  Additionally, I asked the students to described the information used by the author to make his case for the use of pre-mortem analysis (ANSWER: scientific findings and anecdotes).  The multiple-choice portion covered such things as cardinality, power sets, cross products, and generalization/specialization.  I think it went well.

The book of the day (which I just ordered just before lecture, actually) was a longer piece by Gary Klein entitled The Power of Intuition: How to Use Gut Feelings to Make Better Decisions at Work (2007, ISBN: 978-0385502894).  This book was cited by the pre-mortem article, and without knowing much more about it, I assume that this book covers the idea of “prospective hindsight” in greater depth (for more on “prosepective hindsight,” check out this article if you have access to it).  If not, the book is still a Gary Klein book, which at the very least suggests that it would be a good and worthwhile read.

The only really teaching I did in this lecture centered on one slide where I showed the following equation for risk:

I used this equation as a basis for reviewing how much we already learned about risk, and to talk about the techniques we learned to assess parts of it.  Personally, I was surprised at how much we already covered.  The goal for lecture 9 was to talk more about vulnerability analysis, in particular the use of pre-mortem analysis to understand the weaknesses present in a system that might cause undesirable impacts.  The backdrop the in-class exercise was campus security.  We began by reading a case study on the 2007 Virginia Tech shootings prepared pro bono by my friend Thomas Shreeve of the Intelligence Community Case Methods Program.  This case study was used to create a frame of reference for a pre-mortem analysis centered on the same event occurring at Penn State.  In particular, I asked each group to articulate an appropriate scenario (pairing of outcome and event) and use this as the basis for brainstorming reasons why this scenario occurred at Penn State.  These reasons, if correct, reveal system weaknesses, or rather vulnerabilities, that contribute to the potential for the scenario occurring.

Next time: the axioms of probability (with a slew of in-class exercises)

Send article as PDF to

A few weeks ago I came across an excellent book from 1977 entitled An Anatomy of Risk by William Rowe, Sr. (ISBN: 0471019941).  This book provides a thorough technical summary of the state of the art in risk analysis through the mid-1970s.  This includes some of the ground breaking work on risk perception, risk assessment for nuclear power, risk communication, etc.  I believe that this book is one of the first authoritative texts on quantitative risk analysis ever published.  However, since the book was written at a time when risk analysis was a relatively new academic discipline, it was not intended for undergraduate audiences looking to learn the basics of risk.  For me, I intend to use this text as my gateway to the classic research works on risk analysis.

An Anatomy of Risk was previously reviewed by a number of scholars as cited below.  Note that in most cases you must have a subscription to view the actual review.  I also noted the tone of the review on a five-tier scale (SCATHING, UNFAVORABLE, NEUTRAL, FAVORABLE, PRAISING).

• A PRAISING review by P. K. M’Pherson in Cybernetics and Systems, Vol. 8, Nos. 3 & 4, pp. 352-354 (1977) (permalink)
• A FAVORABLE review by L. E.Hill in Technology and Culture, Vol. 19, No. 4, pp. 788-790 (1978) (permalink)
• A PRAISING review by A. R. Unwin in The Journal of the Operational Research Society, Vol. 29, No. 8, pp. 825-826 (1978) (permalink)
• A FAVORABLE review in ACM SIGSIM Simulation Digest, Vol. 10, No. 4, p. 70 (1979) (permalink)
• A SCATHING review by R. G. Easterling in Technometrics, Vol. 22, No. 2, pp. 278-279 (1980) (permalink)
• A FAVORABLE review by M. L. Randolph in Ecology, Vol. 62, No. 4, pp. 1133-1134 (1981) (permalink)

On balance, I would say that the overall take on Dr. Rowe’s book was FAVORABLE++.  I personally recommend that all emerging risk researchers add this title to their Christmas book wish list.

An Anatomy of Risk is no longer available NEW, and can only be purchased used via a used book outlet such as Alibris.com (see here).

Send article as PDF to

In their really, REALLY good paper entitled “Assessing the Competence and Credibility of Human Sources of Intelligence Evidence: Contributions from Law and Probability” published in the journal Law Probability and Risk, Vol 6, pp. 247-274 (doi:10.1093/lpr/mgm025), authors David A. Schum (of George Mason University) and Jon R. Morris (of CIA DS&T) identified a set of twenty-five (25) questions whose answers bear on the question of whether a human source of information is competent and credible.  The twenty-five questions are as follows divided into four categories: competence, veracity, objectivity, and observational sensitivity.

Competence (or is the source qualified to provide the information?)

Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s competence; (b) the evidence on this question disfavors this source’s competence; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s competence; or (d) there is no available evidence bearing on this question.

1. Did this source actually make the observation being claimed or have access to the information reported?
2. Does this source have an understanding of what was observed or any knowledge or expertise regarding this observation?
3. Is this source generally a capable observer?
4. Has this source been consistent in his/her motivation to provide us with information?
5. Has this source been responsive to inquiries we have made of him/her?

Veracity (or does the source believe what he/she is saying?)

Leveraging all relevant existing evidence, for each of the ten (10) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s veracity; (b) the evidence on this question disfavors this source’s veracity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s veracity; or (d) there is no available evidence bearing on this question.

1. Has the source told us anything that is inconsistent with what this source has just reported to us?
2. Is this source subject to any outside influences?
3. Could this source have been exploited in any way in this report to us?
4. Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
5. Is there any evidence from other sources that corroborates or confirms with what this source has just reported?
6. What evidence do we have about this source’s character and honesty?
7. What does this source’s reporting track record show about the source’s honesty in reporting to us?
8. Is there evidence that this source tailored this report in a way that this source believes will capture our attention?
9. Are there collateral details in this report that reflect the possibility of this source’s dishonesty?
10. Evidence regarding the demeanor and bearing of this source during the interview?

Objectivity (or was the source’s belief based on the evidence obtained by the source?)

Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s objectivity; (b) the evidence on this question disfavors this source’s objectivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s objectivity; or (d) there is no available evidence bearing on this question.

1. Is there evidence about what this source expected to observe during the reported observation?
2. Is there evidence about what this source wished to observe during the reported observation?
3. Was this source concerned about the consequences of what this source believed during the observation?
4. Is there any evidence concerning possible defects in the source’s memory? Also, how long ago did this source’s observation take place?
5. Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?

Observational Sensitivity (or how good was the evidence obtained by the source?)

Leveraging all relevant existing evidence, for each of the five (5) questions below, respond with one of the following four answers: (a) the evidence on this question favors this source’s observational sensitivity; (b) the evidence on this question disfavors this source’s observational sensitivity; (c) I cannot decide whether the evidence on this question favors or disfavors the source’s observational sensitivity; or (d) there is no available evidence bearing on this question.

1. The source’s sensory capacity at the time of observation?
2. The conditions under which the observation took place?
3. The source’s track record of accuracy in previous reports?
4. Is there any other evidence from other sources that contradicts or conflicts with what this source has just reported?
5. Are there collateral details in this report that reflect the possibility of this source’s inaccuracy?

Using the Questions

According to the authors, the twenty-five questions above have been implemented in a system called MACE (or Method for Assessing the Credibility of Evidence) that apparently has been under development for some time (I wonder if MACE was fully funded by CIA; if so, do I hear FOIA request?).  The remainder of the paper describes the MACE system and how it works.  For the purposes of this post, it is sufficient to point out that MACE is an evidence marshalling tool.  That is, MACE provides a structured set of questions that enables the analyst to make sense of the evidence bearing on a particular source’s competence and credibility.

In addition to providing an answer to each of the twenty-five questions, MACE insists that the analyst judge the relative importance of each question involving a particular situation and a particular report.  Morever, MACE asks the following two questions:

1. On balance, does the evidence favor or disfavor the source’s competence, veracity, objectivity, and observational sensitivity, keeping in mind the number of questions that remain unanswered?
2. On balance, how strongly does the accumulated evidence favor or disfavor our believing of the report this source has just given us, keeping in mind the number of questions that remain unanswered?

Why Care?

According to the standards for analytic tradecraft articulated in Intelligence Community Directive 203 (ICD 203), all intelligence products must “properly describe the quality and reliability of underlying sources” (section D.4.e.(1)).  [Note that the standard in section D.4.e.(2) is also very important, that is, "properly caveats and expresses uncertainties or confidence in analytic judgments."  But I will defer this discussion until a bit later.]  What Schum and Morris provide is a means for arriving at meaningful statements of source competence and credibility that simply were not available in a documented form prior to publication of this paper.

And why do I, as a risk (not necessarily intelligence, though I can play the part) professional think this is important?  Well, most (if not all) security risk analyses rely mostly on the opinions of subject matter experts, organizational representatives, etc. (i.e., humans) for the information needed to make a judgment about threat, vulnerability, and risk.  Much like in intelligence analysis, risk analysts must carefully appraise the information used to support analysis in terms of both its content and its source so as to ensure that the product is free of unintended bias and influence.

Send article as PDF to

How do vulnerability assessors actually assess vulnerability?  This is an interesting question that I have been thinking about recently, and below are some of my initial thoughts on the issue.  Let’s begin by recalling the following expression of risk:

p = Pr(e,o)   (1)

where the joint probability of initiating event e and outcome o can be expressed in one of two ways:

Pr(e,o) = Pr(e)Pr(o|e)   (2a)

Pr(e,o) = Pr(o)Pr(e|o)   (2b)

From my experience, the more common of these two expressions is Eq. 2a as it really conforms to the more intuitive event tree view of risk (consequence following cause).  The latter expression Eq. 2b is much less commonly used, if it is even used at all.  Yet, Eq. 2b is as much an expression of risk as Eq. 2a.  I actually use part of Eq. 2b later on in this post, which is the reason why I mentioned both equations.

Now why the math?  My first hypothesis is that regardless of whether one can speak the language of probabilistic mathematics, all people think about vulnerability analysis in the same basic way, whether it be as part of one’s profession or routine risk-taking decision making.

Colloquially, when one thinks of vulnerability, one might say something to the effect of “I am vulnerable to outcome o due to event e” (where e and o is defined as before).  More common are statements such as “I am vulnerable [with respect] to e,” where the outcome is implied by the context of discussion.  For example, in the course of discussing an organization’s information systems, the statement “I am vulnerable to attack” made by the organization’s IT security manager most likely refers to a attack directed against IT infrastructure, with the outcomes being loss of confidentiality, integrity, availability, or non-repudiation. This same statement said by a pedestrian walking in downtown Los Angeles might be in reference to a physical assault against his or her person where the outcomes are injury and loss of property.  In both cases, however, these terms are in reference to the conditional assertion that the individual will suffer some type of loss should an “attack” occur.  That is, there is no assessment of the likeliness of event; only the assessment of likeliness of an adverse outcome given event.

Now, refer to Eq. 2a.  In this expression, Pr(e) is the probability of an initiating event and the conditional probability Pr(o|e) is the probability of a particular outcome given an initiating event were to occur.  In the security context, Pr(e) is viewed as a measure for likeliness of attack (i.e., the initiating event), and Pr(o|e) is the measure for conditional likeliness to a particular outcome given an attack were to occur.  I choose to label this latter parameter the “vulnerability to o from e” as it is conceptually equivalent to the manner in which statements of vulnerability are made in everyday language.  Accordingly, in terms of subjective probability, statements of vulnerability express the degree of belief held by an individual in the outcomes that will occur when confronted by a particular challenge.

Back to my original question.  How does a vulnerability assessor do a vulnerability assessment?  Ultimately, the answer to this question should take the form of a descriptive model of human reasoning.  So, as a first step in my quest toward a descriptive model of vulnerability assessment, I decided to contemplate how I, personally, would perform a vulnerability assessment.  The resulting model from this inquiry is what I will tentatively call the “McGill Descriptive Vulnerability Assessment Model“:

Step 1: Soak in the subject environment.  Without looking for anything in particular and without reference to any particular type of attack, explore the subject environment in a thorough, careful and curious manner.  Over time your brain will pick up on both glaring and subtle environmental cues suggestive of strength and weakness.

Step 2: Hypothesize outcomes of particular concern that are relevant to the problem at hand.  These outcomes “o_j” (j = 1, 2, …) can be vaguely defined as “a lot of people hurt” or “significant property damage” or “damage to reputation.”  There is no need to be crisp about the outcomes of concern at this stage.

Step 3: In a very non-quantitative way, attempt to make a judgment about Pr(o_j|E), where the set “E” (big-E) represents the union of all plausible events e_i (i = 1, 2, …) of a particular type.  That is, attempt to make a judgment about the likeliness of one or more of the “bad” outcomes identified in Step 2 assuming that some sort of vaguely-defined event (e.g., “terrorist attack,” “assault,” and so on) occurs (i.e., “E”).

Step 4: For those outcomes where the likeliness is viewed to be sufficiently “strong” (high or intense), assume that these outcome have been realized but the cause is unknown.  This step attempts to hypothesize what the most likely cause of these outcomes were.  This is a sort of pre-mortem analysis.  A list of causes (or initiating events) e_i can be developed in this manner, where the list is ranked in order of decreasing (or increasing if you prefer) likeliness.  If there is no strong feeling of vulnerability, then use this step to try to explain why and attempt to challenge yourself using alternative analysis techniques (e.g., Devil’s advocacy).

Step 5: For each e_i identified in Step 4, assess your subjective degree of belief that undesirable outcome o_j (for each j) will follow from event e_i.  This is a more refined vulnerability assessment of Pr(o_j|e_i) than Step 3 in that we are looking at specific “e_i“’s instead of the whole collection “E.”

[NOTE: you can cycle through steps 2 through 5 over and over again, each time refining the definition of e, adding o's, and so on.]

Step 6: Express your opinion of vulnerability to OUTCOME given EVENT.  A four-tier symmetric linguistic vulnerability scale of the following type can be used (as an example) to aid in expressing vulnerability where the bracketed values express lower and upper probability limits for the phrase:

• Highly Vulnerable … Pr(o|e) = [0.75, 1.00] … (odds are heavily in favor of the adversary)
• Vulnerable … Pr(o|e) = [0.50, 0.75] … (odds are in favor of the adversary)
• Invulnerable … Pr(o|e) = [0.25, 0.50] … (odds are in favor of the defender)
• Highly Invulnerable … Pr(o|e) = [0.00, 0.25] … (odd are heavily in favor of the defender)

[Note that while it may appear that step 6 departs from what one might otherwise think was part of a normative model and not a descriptive one, this is actually how I think.  So it is, in fact, descriptive, but with respect to how I think about vulnerability.]

Let’s see how this descriptive model works.  Suppose I am tasked to assess the vulnerability of my house in Maryland to damage resulting from naturally-occurring events (ignoring that I have insurance).  I admit here that nature is my assumed adversary, and perhaps is my only adversary aside from the occasional disgruntled student.  As I walk around my house, I notice a slightly lopsided roof, sturdy brick exterior, clogged gutters, new windows, canopies of trees (that seem to be on their last leg) blocking the sun, empty garbage cans in the yard, lawn junk (e.g., garden gnome) on the neighbor’s property, curbside lunch trash leftover by contractors than tend to take breaks in front of my house, loose television antennas on neighborhood rooves, etc.  I begin to think that a bad day for me would be when my roof caves in or many of my windows break, since both would cause a significant amount of property damage.  All things considered, I think my vulnerability to many broken windows is quite low, but the roof collapse worries me.  I proceed to consider a variety of causes of roof collapse, to include (in order of decreasing likeliness) tree limbs crashing down from above and excessive rain and autumn leaves weakening the integrity of my roof structure.  Returning to the outcome of concern and leveraging my structural engineering background, I now can make the following judgments:

• My roof is right now vulnerable to collapse due to falling tree limbs (any cause).
• My roof is right now highly invulnerable to roof collapse due to buildup of leaves and rain.
• My roof is right now highly invulnerable to roof collapse due to most other natural causes.
• My windows are right now highly invulnerable to significant damage due to most natural causes.

Notice the underlined words that caveat my vulnerability judgments.

• right now means the vulnerability assessment is valid only for the system in its present state and normal deviations.  If things change (e.g., adding solar panels to the roof, aged roof and windows), then the vulnerability assessment may change as well.
• most other and most are used to allow flexibility for the residual hypothesis I am not considering in my mind.  While I would be hard-pressed to articulate all of the events floating around in my head that might prompt damage, using the word most allows for me forgetting to include a few. (yes, I know this is a cop out, but supposedly more experience = more hypothesized events).

Finally, I must point out that nowhere here do I make any judgment about event likeliness.  That is, what I have here is a method for vulnerability assessment, not threat assessment.  Had I gone on to asset threat as well, the combination of threat and vulnerability (for a given pair of event and outcome) would produce a statement of risk a la Eq. 2a.

I wonder how well does this model matches that of practicing vulnerability assessors in DoD and industry, or with those focused on computers, physical sites, or the fabric of society?  Regardless of how long my model remains unrefuted (which may be a day or much longer), I will continue to seek out ways to discredit it in hopes of converging on a robust descriptive model for vulnerability assessment.

Send article as PDF to

A few days ago I had the privilege of shopping at Ollie’s Bargain Outlet in State College, PA.  While perusing the numerous books, games, and such on sale at what I thought were extremely discounted prices, I came across an interesting miniature guide on the weather entitled The Rough Guide to Weather by Robert Henson (ISBN: 1-85828-827-4) sitting adjacent to the Discovery Channel videos.  The book was only $10, so I figured why not buy it and add it to my reference collection on naturally-occurring events.  And now having had an opportunity to flip through the book and absorb its contents, I am very glad I made I made this purchase.

The Rough Guide is divided into six sections as follows:

• “The Ingredients” section describes all the “actors” and associated phenomenologies on the weather stage, to include the sun, wind, atmosphere, ocean, earth, climate zones, and so on.  This section has very interesting factoids about how weather affected humans from before the “age of observing,” explanations of what the atmosphere is, how light is diffracted to form rainbows, “where the wind goes,” and so on.  Basically, in fewer than 50 pages the author provides a layman’s summary of how the Earth’s weather system works.
• “The Wild Stuff” section describes in detail weather hazards (which by the way are only hazards because humans label them as such; without humans there to suffer loss, these “weather hazards” are merely “weather events”).  The events considered include rain, snow, freezing rain and sleet, fog, thunderstorms, hail, tornadoes, hurricanes and tropical cyclones, coastal storms, other windstorms, floods, drought, El Niño and La Niña.  The author provides a thorough, yet accessible, description of each event type supplemented with informative pictures and tone box asides.
• The “Forecasts and How to Read Them” section provides a brief description of the role of weather forecasts in society and how its role and supporting technologies evolved over the past few centuries.  In addition, this section offers advice on how to interpret weather forecasts and warnings with some detail on the how weather predictions come about.
• “A Primer on Global Climate Change” provides a short, but sweet, discussion on how rising temperatures will affect weather and people.  Nothing too significant here that you couldn’t get in better detail elsewhere.
• The “Weather from Around the World” section is perhaps the most useful part of this whole book.  This section provides a narrative account of the weather for 200 destinations worldwide.  But this book provides more than facts and figures; the author provides a concise and highly informative meteorological story for each destination that may include reference to its history, geography, demographics, size, climate, and so on.  Basically, this section of the book makes for a handy reference to a country intelligence or risk analyst interested in environmental hazards for their target.  I will provide three short country story excerpts to show you what I mean:

Syria. Syria epitomizes Middle Eastern climate, with a Mediterranean wet-winter regime on the coast and a desert that spans most of the country east of a narrow strip of coastal mountains. Damascus is part of the Fertile Crescent, a transition zone near the Lebanon border that just gets enough moisture to keep the desert at arm’s length.  Winters are chilly in the mountains and adjacent plateau, with a bout of rain once or twice a week and a day or two of snow possible in Damascus.  Snows are more heavy and frequent a high elevations.  The rains stop from June through August, which are hot by day but relatively cool by night – except across the deserts, where even the nights sizzle in midsummer.  Winter in the deserts can bring a few days of light rain and even an occasional dusting of snow. (pp. 288-289)

Nigeria. Africa’s most populous nation, Nigeria has a climate regime similar to that of the Guinea-coast nations to its west, as the ITCZ brings wet weather northward from March to May and southward from October to December.  Only the extreme north is semi-arid in classic Sahel style, while parts of the lower Niger delta are drenched with more than 3000mm/118in of rain each year.  The Lagos area is the least sodden trench of coastline, though its humidity – and the heat radiating from urban sprawl – help maintain a sticky atmosphere year round.  The immediate coast experiences a major dry period in winter with heaviest rains in late spring and early summer, followed by a brief dry spell around August and a second pulse of rain into October.  Harmattan dust storms are a perennial winter problem, especially toward the north. (p. 307)

Maldives. Perhaps the most telling aspect of the Maldive’s climate isn’t local but global.  The slow worldwide rise in sea level threatens to inundate much of this island nation within the next century.  There’s little high ground here: the tallest atolls barely top 2m/6ft.  Since the Maldives straddle the equator, wet seasons vary, but they tend toward summer half of the year.  Temperatures hardly vary at all, staying sultry even for a tropical locale.  Only the northernmost Maldives are at risk from tropical cyclones. (p. 336)

• Finally, the “Resources” section provides a list of classic (e.g., Aristotle) and modern references for the budding meteorologist, a list of websites for meteorological organizations and worldwide weather agencies, a discussion of the impact of weather on health (infectious disease, sunburn, etc.), and a few tables that aid in converting pressure, temperature, and windspeed between different measurement systems.

I must say that this book is not bad for $10.  But if you want this book, I wouldn’t run to Ollie’s just yet.  A quick search on Amazon revealed to me at least one seller willing to part with this book for a quarter ($0.25).  On Alibris one could order this book for $1.99 (+ shipping).  Of course these prices are for used copies of the book, though they still may be in pristine condition (I am doubtful about the 25-cent copy, though).

As far as credibility goes, the “Acknowledgments” section on the front-side of the third sheet of paper (not numbered) states that this book has been reviewed by quite a large number (say >50, but I didn’t count the names listed) of scientists and experts in weather science from around the globe.  However, the books lacks any actual bibliographic citations and offers minimal information on the author’s credentials save for a very uninformative blurb on the rear cover telling the reader than “Robert Henson is a meteorologist.”  Thus, we are left having to accept the fact that the book is published in the wake of expert criticism as evidence of its credibility.  But was the decision to publish contingent on favorable reviews from all, or perhaps even a majority of experts?  Or was it up to the author to simply appeal to experts on his own, selectively choose weather to accept the feedback, and then submit a final draft that may or may not be accurate?  While I have no reason to suggest that the book is inaccurate, additional meta-data on the author and publication processes would enable me to increase my subjective confidence in the credibility of this reference.

Oh, and if you are as excited as I am about meteorology, check out the meteorology certificate program at the USDA Graduate School.  For the cost of 25 credit-hours of class (and all that entails in terms of time and lost opportunities, and price per credit hour), anyone can acquire the intuition for meteorological events.  The good news for engineers, scientists, and many other college course takers (current or retired) is that the credit-burden is only 21-credits if you have already completed a course in differential and integral calculus.  The remaining courses in physical, satellite, dynamic, and synoptic meteorology, however, are probably ones we all have to take regardless of background (meteorology majors excluded).

Send article as PDF to

Many, if not most, upper-level undergraduate and first-year graduate engineering students are familiar with the famous text entitled Advanced Engineering Mathematics by Erwin Kreyszig (now in its ninth edition).  If you are not familiar with this book and you desire a single source for the body of practical mathematical concepts that enable engineering analysis, then I strongly advise that you become acquainted with “Kreyszig.”  This book covers the practical elements of calculus, differential equations, linear algebra, numerical analysis, optimization, and probability and statistics, all in 1248 pages!  I will forever keep this book handy.

Recently I encountered a book that, in my mind, rivals Kreyszig in terms of comprehensiveness and thoroughness.  The title is Actuarial Mathematics by Bowers, Gerber, Hickman, Jones, and Nesbitt (second edition, ISBN: 0938959468).  But unlike the Kreyszig text, Actuarial Mathematics is all about the mathematics of risk.  Topics covered in this book include probability models, survivorship functions, insurance pricing, regression, and so on.  Though the title may sound dry, this book is sufficiently lively in tone to keep my mind occupied during an otherwise boring meeting.  This book is absolutely amazing, and for that reason I call it the “Kreyszig of Risk.”  But I would argue that the text advocates mathematical practice that, despite being the accepted standard of practice in the world of professional actuaries, is primitive relative to modern uncertainty modeling approaches (e.g., probability boxes).  I think there is potential for quite a lot of research work focused on applying modern mathematical theory to actuarial problems.

Now despite its mathematical allure, Actuarial Mathematics does not help security risk professionals do their job any better given their inherent relucatance to quantify things without supporting data.  But this did not stop me from buying the book and enjoying every minute of it.  Actually, I believe (as of late) that there is much for a security risk professional to learn from other disciplines where risk analysis is routinely used (e.g., political risk assessment, actuarial science).  So picking this book up for me was my first attempt at understanding the requisite mathematical body of knowledge to become an actuary (see the American Academy of Actuaries website for more information on what an actuary does and what it takes to become one).

Send article as PDF to

The other day I ran an exercise in my risk management class where I asked students to fill in the blank for “______ risk” and describe for me what could be meant by the resulting phrase in terms of who would care, events of concern, and outcomes of concern.  In light of recent frustrating events, I feel compelled to offer yet another example as follows:

California EIT License Risk

(Note that EIT = Engineer in Training).

Backstory

In October 2000, I, like many other 4th-year engineering undergraduate students and professionals throughout the nation, had the painful experience of taking the EIT exam.  Since I was a engineering student at the University of Southern California, I naturally took my exam in California.  Consequently, my success led to my EIT “license” (if you could call it a license) being granted by the California Board for Professional Engineers.

Soon thereafter I graduated and left California to pursue a variety of career options in the Washington, DC area with absolutely no intentions to return to California for work.  Accordingly, from May 2001 onward, I was no longer a California resident.  In fact, since that time I spent no more than one calendar month (cumulative) in California visiting family, vacationing, etc.  With several years engineering experience + a masters degree, I successfully earned my professional engineer license in the State of Maryland, which effectively trumps the EIT (you see, EIT is a necessary stepping stone toward a PE in the engineering world, but once you earn your PE the EIT becomes irrelevant).

Now spring forward to 2005/2006 (I forget the exact month and date).  One day I received a strange letter from the State of California Franchise Tax Board demanding that I file a CA-state income tax return.  Basically, since I held an “active license” with the State of California, the State felt that I must be earning money as either a CA resident or as a person doing business in CA under a CA-license.  Well, I can see this being a plausible inference for most licenses, but for the EIT?  Technically, the EIT entitles you to nothing save for the “right” to make progress toward a PE.  And it has no expiration date and no means of termination except in cases of ethical or criminal misconduct; this means that once you are an EIT, you are effectively always an EIT regardless of where you live.  Moreover, since the EIT is given based on results from a nationally-accredited examination, States recognize EITs granted in other states; this fact leaves little incentive to transfer an EIT between states, if even one could do such a thing.  Fortunately, after a short, but hard-fought battle with a live CA tax representative, I convinced them that they were in error and asked them to ensure that I don’t receive such a letter again.  For two years this worked.

Now in 2008 (about a week ago), I received yet another letter from CA demanding I file a tax return with the state.  (apparently, CA’s aggresive pursuit of tax $$ has led them to develop a filing enforcement program).  Again, CA stated that since I hold an “active license” with the Board for Professional Engineers in the State of California, I must be earning money there, and thus am required to file a tax return.  This time, they hedged a little bit and offered a way out for people who were issued the demand in error.  But this wiggle room came at a cost: unlike my previous experience, finding a live person to talk to was hard to do, so hard in fact that I failed to find an approriate mechanism to speak with a CA tax person on this issue.  The only means available for me to correct this error was to fill out a form they provided, state under penalty of perjury my 2006 family taxable income, and pay for the stamp to return the form to CA.  And if I don’t return the form by mid-Sep, then bad things could happen, such as CA sending me a bill for unpaid taxes on money they estimate I should have earned + interest for two years.  Or worse, I could enter in a very costly multi-year legal battle with CA over an issue in which they never really had jurisdiction.

Me being the risk averse person that I am with such matters, fronted the $0.42 to mail back the form, but I decided to not include my taxable income.  After all, I did nothing wrong, so technically I am not required to tell CA anything at all.  Who gave them the right to ask and then demand that I answer less they fine me, or worse, threaten legal action?  Can they arbitrarily claim jurisdiction over me despite me not having resided in the state for over 7 years?  In the grand scheme of things, I provided sufficient other information on the form to make it clear that I owe nothing, and figured that $0.42 is a very small price to pay to avoid having to engage in a never-ending battle with the California bureaucracy.  But man was I furious.

California EIT Risk

So back to my original purpose.  It seems that for whatever reason, CA recognizes CA-EIT holders as holders of a professional license (although again, though it is important, it isn’t really a license).  That said:

Who Cares? Holders of a CA EIT that moved, or are planning to move, away from CA on a long term basis, and without any intention to do work for a CA-based business.

Events of Concern? California not paying attention to this fact, and thus sending you threatening “Demand for Tax Return Letters” at your new non-CA home (wherever that is).  Note: Since $$ is at stake, California will find you.

Outcomes of Concern? Having to bow down to CA-state pressure to prove your innocence less having to face the wrath of the CA bureaucracy.  Of course, I have no evidence for what CA would really do if one doesn’t respond, but who really wants to find that out.

Admittedly, I am taking a narrow view on this, and one could consider a slew of other events and outcomes of concern.  One, for example, might be that CA misplaces or deletes any record of you having earned an EIT, thus making it more difficult when the time comes to apply for your PE.  But I figured I would keep it simple by focusing on the lesser known events/outcomes that should be on the minds on the thousands of EIT engineers originally “licensed” in CA.

Send article as PDF to

Today I gave a lecture to my risk management class at Penn State (SRA 311, Risk Management: Assessment and Mitigation) focused on the words of risk analysis (lecture 2 of 31).  As anyone who provides services to any type of client knows, one of the first things you have to do on day one is ensure a common understanding of key words and phrases.  This was part one of my lecture, that is, explaining that people don’t necessarily assign the same meanings to certain words as others, even if they are in the same field.  The remaining parts focused on two words in particular – “security” and “risk” – and sought to explain what “risk” is and how it fits into security activities.  This lecture was fun for me to deliver, but in hindsight, it was probably a bit too densely packed with ideas for students with less background knowledge.  All in all, I think it went ok.

Class Summary

As a backdrop for discussion, I had my students read two articles.  The first article was entitled “Same Words, Different Meanings: The Need for Uniformity of Language and Lexicon in Security Analysis and Management” by Andrew Harter (a good friend of mine) published by the Critical Infrastructure Protection Program of the George Mason University School of Law in the monograph entitled Critical Infrastructure Protection: Elements of Risk (prepared by Liz Jackson, another good friend of mine).  Basically, this article is a call to action in the security analysis and risk management community for establishing a common lexicon through voluntary consensus standards.  For those unfamiliar with this issue, Mr. Harter’s article addresses the question “why is a common lexicon needed?” and “what can be done to make progress toward this goal?”   Though one might argue that alternative viewpoints (e.g., a common lexicon is not needed) were not addressed in this article (which is a “hit” on fairness), the point surely rings true to anyone who plays the security risk analysis game.  Imagine how difficult it is to communicate on risk matters when your definition of risk (e.g., potential for harm) doesn’t match well with mine (e.g, loss following an event).  I’ve experienced hours of time wasted due to a simple misinterpretation of language, and nothing is worse than arguing semantics when other more important issues have yet to be resolved.

Some might argue that definitions don’t matter so much.  After all, risk analysis is a decision support activity, and really all that matters is whether we have empowered the decision maker with “decision advantage.” [I borrow this phrase from the Jennifer Sims at Georgetown University as it is applicable to ALL areas where analysis is done, risk and intelligence in particular].  Accordingly, one might accept the definition of risk as “whatever is appropriate for the decision maker at the time.”  But as the author of my second paper, Giovanni Manunta, might argue, while such a vague definition might be useful in the client-analyst context, it is not helpful if one desires to treat risk as a science and methodically study all the different subtopics that fall under the heading of risk analysis (see the very first text block on the Society for Risk Analysis homepage for their definition of what “risk analysis” entails).  A common understanding of the various “words of risk analysis” is needed in order to speak sensibly about the subject within the community of educators, scholars, and practitioners.  (as an aside, see Professor Kristan Wheaton’s blog for an interesting and related discussion entitled “What is Intelligence?“)

The second paper discussed in my class was entitled “What is Security?” by Dr. Giovanni Manunta and published in the Security Journal, Volume 12, Issue 3, pp. 57-66 (http://dx.doi.org/10.1057/palgrave.sj.8340030).  I chose this paper for three reasons.  First, for me it was a great read and why not share with my students papers I find worthwhile.  In fact, many of Dr. Manunta’s monographs are really worth spending some time reading and absorbing if you are in the security profession.  Second, this paper is a nice complement to the first in that it goes into great depth as to why a commonly accepted conceptual definition for security is needed.  Third, this paper actually does a good job of describing the conceptual underpinnings of security by explaining in detail the three required elements of a security context – namely, a Protector (the entity that desires security), a Threat (the entity that challenges the protector’s security), and an Asset (the object of conflict).  The general formula for security, S, is then S=f(P,T,A)Si, where the Si outside of the parenthesis is a variable that accounts for the situational factors underlying the relationship between P, T, and A.  If any one of P, T, or A are absent in a given situation, you do not have a security context, and as such it makes no sense to speak about managing risks.

At this point I finished discussing (as socratically as I could in the time I had available) the two articles.  Throughout I attempted to elicit from students answers to questions centered on Elder and Paul’s Eight Elements of Thought and Intellectual Standards to encourage critical analysis of who the people writing such articles are, their purpose for writing, points of view, concepts, assumptions, etc.  However, I tried not to stretch this discussion out too long given that I already had my students complete a written assignment that systematically addresses the eight elements and intellectual standards.

The next portion of this lecture centered on how risk management fits within the world of security.  Borrowing from Manunta’s Diogenes Paper No. 1 (ISBN: 0-9501575-4-6), I sought to leverage assumed prerequisite knowledge of Venn Diagrams and Set Theory to explain the concepts of Security and Not Security, where Not Security includes Total Insecurity and all degrees between.  The degrees in-between represents a fuzzy-boundary between security and not security, that is, if one accepts that the state of security is actually a fuzzy set.  The Venn diagram I used is shown below, though in class I actually drew it on a Tablet PC.

The point I stressed is as follows: in a security context, a Protector has finite resources to make progress toward an unbounded objective.  This is where risk management comes in – risk management is used to maximize the efficiency of these resources by applying them in such a way that maximizes our progress toward a state of security.  The balance of risk between what we want to achieve and what we can achieve is known as the residual risk.  Ultimately, given the options available to us to reduce risk in light of available resources, we want to minimize the residual risk.  But as Manunta points out in “What is Security?,” security involves risk management, but managing risk doesn’t necessarily guarantee security.  That is, risk management and security are not the same thing.

I ended the lecture with a light hearted game of “Risk Mad Libs.”  First, I offered a generic definition of risk intended to guide us through our thinking in the rest of the course.  The definition is as follows:

Risk: The uncertainty around future events

We discussed what was meant by the word “uncertainty” in this definition, and examined the different types of uncertainty that we often encounter in risk analysis.  This includes the variability associated with one or another event occurring among a set of mutually exclusive (distinct) and collectively exhaustive (complete) alternatives, the incertitude associated with whether elements in our set are relevant or whether our set of alternative events is complete, and the inherent vagueness in what any particular element of the set really means.  Unfortunately, my extemporaneous nature kept me from explaining the remaining two words – “future” and “events,” but if I could go back in time I would stress that risk has to do with the uncertainty in what will happen and not what has already happened, where the future “events” can be described as a situational description (“mom will get sick”) or in terms of some measures (“1 morbidity” and “$10,000 in medical fees”).

Now that we had a definition of risk to work with, I asked students to break into groups and fill in the blank:

____________________ Risk

where the blank can represent practically any word.  My specific instructions were to select one “serious” word and one “silly” word, fill in the blank with each in turn, and in doing so characterize the nature of what is meant by the resulting phrase (i.e., who would care, what are some causes of concern and what are outcomes of concern).  I started with the serious word “information” to form the phrase “information risk.”  Then I moved onto the word “political” followed by the silly word “dog.”  For each we identified someone who might be considered a stakeholder in such a field (e.g., “dog owner” for “dog”), and brainstormed what events could occur (“dog runs away”) and the spectrum of ensuing outcomes (“dog gets hit by car,” “dog bites pedestrian,” “dog comes home”).  In the remaining 2 minutes of class following the exercise, we had some cool responses, including “computer mouse risk,” “environmental risk,” “body odor risk,” etc.   The basic idea here was to enable students to reason out what is meant when you see a phrase such as “financial risk,” and after this lecture I am confident the students can do this.

Next Up

The next lecture stands to be a fun one – the topic is “The Role of the Risk Analyst and Decision Advantage.”  This lecture is the second of 3 “Philosophy of Risk” analysis lectures; after these, we will be way more applied in the classroom setting (something I am sure the students would appreciate).

Send article as PDF to

To follow up on my previous post regarding the work of Peter Sandman, I can’t help but advertise his short, yet important article entitled “Risk Words You Can’t Use” published in the August 2005 issue of The Synergist.  While this article is a quick read, I will distill it down further and caveat some with my personal experience:

• Conservative: To risk people, conservative means an overestimate of risk.  To laypeople, a “conservative” estimate is a low estimate.  So whereas a risk person would use conservative to overstate the risk, a layperson (or perhaps decision maker) may interpret the message to be an understatement of risk, and thereby think that the risk could be much worse.  Now, engineers and scientists understand what is meant by the word “conservative,” as in my “conservative analysis still shows the structure will not fail.”  And fortunately for me, when I described my idea of conservative discounting of expert opinions (to be explained in a later post that I will link to when it is available) I was speaking to an audience of security engineers.  I will keep Sandman’s advice to not use the word conservative when speaking to non-technical audiences, and instead opt for the word “overestimate.”
• Significant/Insignificant:To risk people and statisticians, a significant finding is one that is non-random.  To laypeople, whether an issue is significant depends on their emotions and value structure.  So, to tell people that the terrorism risk is insignificant might not communicate well.  It is true (right now based on our current understanding and situation) that a person’s individual risk to terrorism is very, VERY low, but the outrage is high, and thus the public’s emotional response might label terrorism as a significant threat.
• Positive/Negative: To risk people, a positive relationship means that when one variable goes up, so does the other.  To laypeople, a “positive” relationship is favorable from the point of view of risk.  The same can be said of negative relationships.
• Bias: Bias to a risk person means non random.  Bias to a layperson spells deceit.
• Anecdotal: Anecdotal evidence to a risk person means the evidence is just one sample from a much larger sample space.  Anecdotal to a layperson suggests the evidence is an amusing story.  This word might not bode well when talking about anecdotal evidence on poor public response following a catastrophic event.
• Risk [my personal favorite]: To risk people, the risk associated with a situation describes its probability and the corresponding consequences.  To laypeople, risk usually refers only to the probability component.  In fact, when lecturing on the use of “uncertainty phrases,” I often emphasize that the word “likely” is not an adverb tied to any particular notion, but one that can be used to qualify likeliness, confidence, and risk.  Of course, people probability consider how they feel about a hazard when judging whether the probability, or rather risk to them, is acceptable.  Others, particularly when speaking about finances, use risk to describe uncertainty – the higher the risk, the more uncertain the outcome.  The philosopher Frank Knight sides with these interpretations in his description of “risk proper,” or measurable uncertainty, described in Risk, Uncertainty, and Profit. Most people argue that the only measure of uncertainty, at least when it comes to gambling situations, is probability, so what Knight is suggesting is that assessing “risk proper” is equivalent to a probability assessment.  But Peter Sandman suggests that what people really mean by risk is how outraged they feel about the situation.
• Safe: To risk people, safety is the judgment of risk tolerance.  If we are safe, then the risk does not exceed some threshold value (whether implicit or explicit).  To laypeople, “safe” = “no risk,” that is they treat it as a binary concept – you are either safe or you are not.  Or rather, there is risk or there is not.  I suppose the same reasoning can be extended to the word secure: to risk people, if we are secure, then the residual adversary risk is low enough for us to accept; to laypeople, “secure” = “no harm will come to them” in the event of an attempt.  Relative statements about safety and security are unambiguous though – to say something is more or less safe or secure than another thing is perfectly acceptable.
• Prepared:To be prepared means that we possess the capabilities and vigilance necessary to deal with a hazardous situation when it arises.  To risk people, preparedness is tied to risk acceptability – if we are prepared, then we have the capabilities needed to keep risk overall at an acceptable level.  To laypeople, prepared, like safe and secure, is taken to mean no (or perhaps minimal) harm will come to them.
• Confident: To say to someone else that you are confident when you are merely hopeful is not okay.  In the eyes of laypeople, confident = surety, though perhaps not so much anymore if the word has lost its meaning in the eyes of risk communication consumers.

From my experience, I have five types of phrases to add:

• [Low/Moderate/High] Confidence:Philosophically speaking, to the analyst, anything said with a non-zero degree of confidence implies some degree greater than even odds of being correct.  This means that both “low confidence” and “high confidence” judgments are believed to be the right answer vice any alternative, but “low confidence” statements are afforded less commitment and as such are pegged to a representative probability value closer to 0.5 than a “high confidence” judgment.  To the decision maker, however, the scale may be expanded from a half probability scale to a full probability scale, where the words “low,” “moderate,” and “high” span the entire range.  So when the analyst says something with “moderate” confidence to indicate, say, a 75% chance of being correct, the decision maker might see it as a 50/50 judgment.  I would love to experiment with this to see whether or not what I just described is true.
• “In General”: When mathematicians use the phrase “in general,” they mean what they say applies to all cases.  When lay people use the phrase in general, they mean that what they say is believed to apply to a simple majority of cases.
• Likely, Probable [and other uncertainty phrases]:  To risk people, the word likely conveys some degree of likeliness that exceeds 50%.  To laypeople, likely may communicate likeliness or risk.  In the latter, one might find that something deemed “likely” to a layperson may have an objectively low probability of happening, yet a high enough impact if it does to warrant use of the term in their non-probabilistic minds.  But whoever said words like “likely” and “probable” can only be used in the context of probability theory?  After all, what came first – the word “probable” or the “theory of probability?”
• Likelihood versus Likeliness: To mathematicians, “likelihood” means something very specific.  The likelihood of something in the context of Bayes theorem is the functional expression Pr(B|A) (read as “the probability of B given A) whose input argument is “A.”  That is, the “likelihood” is the hypothetical probability distribution constructed over a space of events conditioned on the occurrence of “A.”  The “likelihood function” or simply “likelihood” L(A|B) is proportional to Pr(B|A).  To non-mathematicians, including most (if not all) dictionaries, “likelihood” describes the notion of chance, where probability is one such measure of likelihood for an event.  According to WordReference.com, the word “likeliness” is an equivalent word for “likelihood,” but doesn’t carry with it all the mathematical baggage that might confuse a mathematician.  This is why I always use the word “likeliness” to characterize the notion of chance instead of “likelihood.”
• Possible: To mathematicians and risk people, a “possible” event is one that carries with it a non-zero probability.  More specifically, a possible event is one that is admitted into the set of alternatives (sample space) for a given question.  To non-mathematicians and laypeople, the word “possible” may be used to describe degree of chance or even risk.  How often have you heard people use possible to convey the likeliness of an event?  I read a study published by Sarah Lichtenstein and J. Robert Newman in 1967 (Psychonomic Science, Vol. 9, No. 10, pp. 563-564) showed that a group of 177 people, when individually asked to place numbers on words that convey uncertainty, could not agree on a probability value for the word “possible.”  The results showed a range of responses spanning probabilities of 0.01 to 0.99, with a median at 0.49.  What does this say?  To me this study makes my point – possible means that the probability is greater than 0, but we don’t know where.  But it also says that, at a micro level, possible might actually assign a value to possible.  Fortunately, the word “impossible” does not suffer the same ambiguity.

I am curious to hear your thoughts on these and other words that we should be careful about using in the context of risk communication, or “analytic communication” for that matter.

Send article as PDF to

The Security Analysis and Risk Management Association (SARMA) just posted some presentations from their most recent conference (the 2nd National Conference on Security Analysis and Risk Management) on their website.  The titles of the available briefings as of 5/31/08 are:

• Charles Davis and Lisa Bendixen on ”Implementation of the NIPP and Sector Specific Plans.”  Ms. Bendixen presented this one, and I was the moderator.  Click for more information on the National Infrastructure Protection Plan, or NIPP.
• Robin Dillon, Catherine Tinsley and Matthew Cronin on ”Interpreting Past Events in Disaster Preparedness Decisions.”
• Geoff French on ”Intelligence Analysis for Strategic Risk Assessments.”  Mr. French also wrote a paper on this same subject as part of the GMU School of Law Criticial Infrastructure Protection Program Monograph entitled Elements of Risk (of which I have a paper on the definition of vulnerability coauthored with my doctoral advisor, Professor Bilal Ayyub)
• Benedict Conboy on “New York State’s Efforts to Link Threat, Consequence, and Vulnerability to CT Planning.”

Moreover, several Security Risk Analysis students from Penn State’s College of Information Sciences and Technology wrote in their personal blogs about different sessions of the SARMA conference.  These blog entries are as follows:

• Maiselog: Trip to 2008 Security Analysis and Risk Management Conference (SARMA)
• Maiselog: 2008 Security Analysis and Risk Management Conference: Overview of First Day
• Maiselog: 2008 Security Analysis and Risk Management Conference: Crisis Management and Defense Support of Civil Authorities & GAO Forum
• Maiselog: SARMApedia: A Common Knowledgbase for Security Analysis and Risk Management
• Hodgepodgeblog: SARMA Day 1 Opening Remarks
• Hodgepodgeblog: SARMA Day 1 – Christopher T. Geldart
• Hodgepodgeblog: SARMA Day 1 – Brandon Wales
• Hodgepodgeblog: SARMA Day 1, Luncheon with the Honorable Joel Bagnal
• Hodgepodgeblog: SARMA Day 1, Technical Session 1 – Steven Mabeus and Kristine Paptanich
• Hodgepodgeblog: SARMA Day 1, Technical Session 3 – William McGill
• Hodgepodgeblog: SARMA Day 2, Plenary Session Speaker – Dirk Maurer
• Hodgepodgeblog: SARMA Day 2, Plenary Session Speaker – Cathleen Berrick
• Counter-Terrorism and Security: 2nd Annual National Conference on Security Analysis and Risk Management
• Counter-Terrorism and Security: Daniel Polsby – Dean of George Mason’s School of Law – on the CIPP
• Counter-Terrorism and Security: Ed Jopek – President of SARMA – Keynote Speech
• Counter-Terrorism and Security: Christopher T. Geldart — Director, Office of National Capital Region Coordination, FEMA — Opening Plenary Session
• Counter-Terrorism and Security: Brandon Wales – Deputy Director, HITRAC, DHS – 2nd Plenary Session
• Counter-Terrorism and Security: Celina Realuyo – Safeguarding Supply Chains from Geopolitical Risks
• Counter-Terrorism and Security: Bob Liebe – Risk Analysis Frameworks for Counterterrorism
• Counter-Terrorism and Security: Dirk Maurer — Deputy Assistant Secretary for Crisis Management and Defense Support of Civil Authorities, DoD — 3rd Plenary Session
• Counter-Terrorism and Security:  Cathleen Berrick – Director, Homeland Security and Justice Team, US GAO – 4th Plenary Session
• Panos Koutsikos: The 2008 National Conference on Security and Risk Management
• IST Building Post: Thoughts from the SARMA Conference (Day 1)

I am sure more conference presentations will be posted in the coming weeks, so stay tuned.  And be sure to look out for the third National Conference on Security Analysis & Risk Management that, judging by my calculations, is most likely to take place in May 2009.

Send article as PDF to