“I shall be telling this with a sigh
Somewhere ages and ages hence:
Two roads diverged in a wood, and I
I took the one less traveled by,
And that has made all the difference.”
(excerpt from “The Road Not Taken” by Robert Frost)

DHS and MITRE had a big announcement yesterday. MITRE has developed a new system for scoring weaknesses in applications, as well as for combining that score with “business value context” to produce a risk estimate. Overall, the work is interesting, though perhaps more from an academic perspective than anything else. What I find interesting is that we’re going back down this road again (“trust” evaluation), which seems like it will inevitably lead to another game-able system.

A brief history… in the 80s the federal government (DOD) published the “Rainbow Series” of security tomes, with a particular focus on the “Orange Book,” which set forth the “Trusted Computer System Evaluation Criteria” (aka TCSEC – I still have a set on my bookshelf, ironically placed next to Covey’s The 7 Habits of Highly Effective People and Scott Adams The Dilbert Principle :). The DOD’s Orange Book was used to establish criteria for evaluating the level of “trust” a given system might have, which could then be used to determine whether or not a given system could be used at a given security/clearance level.

The problem with this system is that it created a fracture in product development. Commercial products would be developed, and then someone would say “hey, maybe we should sell this to the federal government!” only to realize that their product didn’t meet the right trust level. So, rather than improve the overall product line, companies would split off a line of development specifically geared toward achieving certification, or they would come up with such ridiculous configurations that no enterprise would ever use it (e.g., Windows NT 4.0 and C2 certification).

This problem became progressively worse in the 90s (and beyond) with a shift away from TCSEC toward, first, SEI’s Capability Maturity Model (CMM), and then using Common Criteria (CC). In both of these cases we again saw gaming of the system. As with previous incarnations, we saw development projects intentionally fractured. In the case of CMM, they were split off into “pilot projects” designed to demonstrate a certain level of capability (usually CMM Level 3), but those changes were rarely integrated into mainstream development efforts. In the case of CC, we would see the same thing happen as happened with TCSEC; configurations would be so constrained that they could only meet a very specific point need with no applicability to the enterprise. Moreover, security was not improved, per se. More often than not, services/capabilities were simply removed or disabled in order to achieve the desired “trust” level.

Which brings us to 2011 and the release of the Common Weakness Scoring System (CWSS) and the Common Weakness Risk Analysis Framework (CWRAF). Given the history of these evaluation and scoring systems, you’ll forgive me if I’m a bit skeptical (or even cynical) about the prospects. I think CWSS itself is a very interesting advancement in terms of metrics and measurements, just as I’ve found the CVE and CWE frameworks to be of interest. It’s still too early to tell exactly how these tools will be used, though CWE (and OVAL – a related project) have been integrated into the “FY 2011 Chief Information Officer Federal Information Security Management Act Reporting Metrics.” CWSS would provide the means for scoring applications, and CWRAF may provide a bridge into the NIST Risk Management Framework (RMF). However, to what end? It’s unclear. Maybe they’ll do it the “right way” this time, but I’ll be surprised if we don’t see vendors attempting to game the system once again.

One thing I fully expect to see is rapid integration of CWSS (and possible CWRAF) into appsec tools and possibly GRC products (I could see CWSS going into appsec tools – both dynamic and static analysis – and CWRAF going into GRC products, with the ability to pull CWSS scores from the other tools). The question will be whether or not these are mainstream products or federal-only versions. We’ve already seen this fracturing with cloud services providers (e.g., Google has a “federal-only” cloud that is domestically based and secured to a different standard, but which is not available to non-federal customers).