GAO has delivered an updated version of the testimony from February 14th that I talked about here. I’m not going to rehash what I’ve already said, but I want to focus your attention on something I didn’t talk about then: incident statistics.

According to GAO, the number of incidents that were reported to US-CERT increased 259% (*cue shock and awe*, but I think that they forgot to add “average annual increase of 259%” because otherwise the math doesn’t even pass BOTE calculations) from 3634 in FY2005 to 13029 in FY2007. OK, so the number is increasing. But there are several failures in GAO’s logic here that need to be pointed out:

“The need for effective information security policies and practices is further illustrated by the number of security incidents experienced by federal agencies that put sensitive information at risk.”

In other words, they’re trying to indirectly draw a conclusion that the high number of incidents is directly proportional to their audit findings. While this may be true in some (most?) ways, it’s also bad to make this comparison in other ways because you would expect the number of incidents to go down over 2 years because the number of implemented, tested, and integrated security controls has gone up.

So really, what’s the dealio?

The first thing that I would like to point out is that security policies and practices have an indirect impact on security incidents. You don’t have a solid one-for-one comparison that you can use, so I think GAO is doing itself an injustice by trying to correlate these two things. However, you can use incident metrics as a holistic metric for measuring how well your information security program is doing, but overall it’s a very coarse method.

The second thing that I need to point out is the trend of the incident number itself. Anybody who starts tracking incident metrics has to ask themselves one question: because we’re now tracking the number of incidents, does it mean that we’ll now notice that there are more incidents simply due to the fact that we’re now measuring them? It’s the incident response equivalent to Schrödinger’s cat and the Measurement Problem. =)

There’s a couple of reasons that the incident count has increased 259% in just two years:

• First is the awareness of incidents. Government-wide, 2 things have happened in these 2 years that should have increased the number of reportable incidents: maturity of US-CERT to receive and categorize larger amounts of incident data; and the maturity of agencies to have their own incident response and reporting procedures. In short: the infrastructure to respond and report now exists where it really didn’t 3 years ago.
• A series of high-profile incidents around PII followed by OMB mandating that all incidents related to PII be reported to US-CERT within one hour. As a result, many more incidents are now being reported if there is a possibility that there is an incident and if there is a possibility that the incident involved PII because it’s the career-safe move: “When in doubt, report it up”. Whether they admit it or not, the people out in the agencies are now what we could call “gun shy” about PII incidents, and that increases the amount of reported incidents.
• The criteria for an incident is very broad and includes “improper usage”, “scans/probes attempted access”, and “investigations” which is classified as “Unconfirmed incidents that are potentially malicious or anomalous activity deemed by the reporting entity to warrant further review”.

If this were an SIEM or IDS, I would say that we’re flagging on too many things and need to tune our systems down a little bit. Keep in mind that it’s the nature of Government to underreport (when they’re not required to report) and overreport (when they are required to report).

You still need to track the aggregate number of incidents reported to US-CERT and in theory this number should trend downward as we get better at governance at the national level as sort of a “trickle-down infosec economy”. Keep in mind that this number should peak within 5-10 years and then slowly be reduced as we fine-tune our reporting criteria and as we get better at securing information. Of course, I won’t be surprised if it doesn’t due to the threat environment, but that’s a conversation for another day.

However, what I propose is the middle-ground on incident reporting: what we really need to pay attention to for the next couple of years is the number of “severe” incidents. Those are the incidents that have actually have an impact that we really care about. These are mentioned in the GAO report, and we should all be able to recall a handful of them without even seeing what GAO had to say.

Knowing this town, I propose we use “Rybolov’s Washington Post Metric”: How many security incidents were significant enough to be deemed “newsworthy” by the Washington Post and mentioned somewhere. For fine tuning, you could use, say, daily front page v/s the Sunday supplement technology section.

My parting shot for the FISMA-haters:  in the years of yore before FISMA (or GISRA if you want to go back that far), how many of these incidents would have been reported?  It seems like we’re failing if you take the numbers and the reports at face-value, but as GAO says in their title:  “Progress Reported, but Weaknesses at Federal Agencies Persist”.  What more do you need to know?

• Looking for the Charbo Testimony
• OMB Wants a Direct Report
• Cage Match: OMB Report V/S GAO Report, Only One Comes Out Alive
• GAO’s 5 Steps to “Fix” FISMA
• Towards Actionable Metrics