News flash: Those so-called “risk” labels/ratings included in pentest and vuln scan reports are NOT actually “risk” representations.

I was in attendance at the OWASP Summit 2011 a couple weeks back, and the topic of “risk metrics” and labels came up during one session. As a result, I led a break-out session on what risk really looks like in the macro sense, in accordance with formal methods, and where these various scan/test results really fit in. The session had great conversation and highlighted for me a need to expand risk analysis training to a broader audience.

Below is a picture of the taxonomy of factors that make up a FAIR (Factor Analysis of Information Risk) risk analysis. Putting aside the discussion of how one generates the value ranges that go into each factor, let’s look at where pentest/scan results fall. Looking at the taxonomy below, note that there are two key halves: Loss Frequency and Loss Magnitude. As you peruse the factors that roll up to those halves, think about where your pentest/scan results might fit.

In order to properly estimate “risk” based on the results of a pentest or vuln scan, you need to understand the business impact in a number of structured scenarios. Simply understanding the Loss Frequency is not sufficient to estimate risk. However, that said, there is certainly valuable IF you process those findings accordingly. The main factor reflected by these test results will be “Resistance Strength,” which is to say that you will now have a better understanding of how much effort would be required to compromise your organization (given: compromise given enough time and resources is inevitable).

The next time you get a report from a vendor that talks about “risk,” please challenge their assertion. Ask them to explain how they estimated the financial impact of various weaknesses on your organization. Unless they interviewed your business management team to understand how a weakness could impact the business if exploited, then I submit that they’re not providing you with an actual risk rate. Instead, what you’re getting a snapshot reading of your Resistance Strength in a given context, as well as receiving some hints on what opposing Threat Capability might be. You’re still left needing to estimate the other factors under Loss Frequency, and you almost certainly need to fill in the blanks under Loss Magnitude (something that can be accomplished independently by developing loss tables for key systems, and then either providing those to 3rd parties performing the assessments, or combining them with the results on your own).

As an aside it’s worth noting that some 2nd generation “GRC” platforms are starting to integrate risk analysis capabilities like FAIR that can be leveraged to merge in scan/pentest results and to generate a reasonable risk analysis.

There has been much criticism of risk assessment and analysis over the past few years that amount to much ado about nothing. Why is it much ado about nothing? Well, because, quite simply, people oftentimes don’t understand what it is they’re criticizing, especially in the case of quantified risk analysis methods.

Before we get into risk measurement, let’s first make one thing clear: risk analysis is nothing more than a decision-analysis (or decision-support) tool. It helps provide reasonably accurate data points that decision-makers can use when make decisions. It is not a panacea for all things risk or infosec, nor is it some sort of special magic-sauce voodoo with no grounding in reality (at least not in terms of well-considered quant methods). Clear? Crystal, I’m sure.

When performing a risk analysis, we need to start with the basics. Here at Gemini we subscribe to the Factor Analysis of Information Risk (FAIR) methodology for performing quantified risk analysis. FAIR defines risk as “the probable frequency and probable magnitude of future loss.” What this means in real terms is that FAIR reduces “risk” into two main components: Loss Event Frequency and Loss Magnitude. Both are estimates that are created using Douglas Hubbard’s calibration technique, as advocated in his book How to Measure Anything.

Read the rest of this entry »

Risk assessment gets a bad rap these days, thanks in large part to a checkered past colored by qualitative analyses. Historically, risk assessments have been fuzzy, at best, and down-right inaccurate and misleading at worst. You know the ones I’m talking about: some hot shot consultant comes in, pokes around, maybe runs a couple scans, and then churns out a report with a bunch of High, Medium, and Low findings. However, as you dig into the results – particularly the so-called “High Risk” findings – you start finding extreme squishiness with no connection to reality, rational thought, or logic. And this is what we’re supposed to use to “better manage” security? Don’t think so…

Enter Factor Analysis of Information Risk (FAIR), a different sort of beast altogether, created by Jack Jones of Risk Management Insight (RMI). FAIR is a decision support tool that provides a means for performing a quantitative risk analysis around a given scenario. It allows you to conduct deep analysis into given asset+threat scenarios, digging into the business to arrive at accurate estimates (via ranges) for probabilities and expected losses. Loss events are divided between primary and secondary, wherein primary losses are often fairly well known (e.g. how much it costs to replace a server), whereas secondary losses can vary widely.

For an excellent introduction to FAIR, the RMI white paper “An Introduction to Factor Analysis of Information Risk (FAIR)” is highly recommended. In it, you’ll start to see the breakdown of FAIR into its component pieces. Overall, within the context of FAIR we define risk as a derived value measuring “the probable frequency and probable magnitude of future loss.” There is much that can be said about this definition and overall approach, but I’ll leave that for another day. In the meantime, I encourage anybody with an interest in risk analysis to take a deeper look at FAIR.