#1:  How does a company/organization convert from doing compliance management to doing true risk management?  I think it’s the difference between being good and being great.  There are a couple of non-IT models that we can look at:  Emergency Room care transitioning into long-term care being a good one.

#2:  Compare and contrast the metrics that are collected as part of the annual FISMA reports with the major initiatives that we have on the table.  They don’t add up.

OK, I think it’s time to go fish this weekend, I’m having dreams about LoB initiatives.  Mini-me says I need to do something non-IT/security/$foo for the 8 hours of the day that I’m NOT working.

• Core Belief #4 — Compliance is a Dead-End
• Now ISC2 Blogs have an Opinion on FISMA
• Core Belief #2 — Risk Management Uber Alles
• Comments on the Annual OMB Security Report to Congress
• It’s a Problem of Scale!

Heh, sensationalist title, but you get the point.  There are two worlds out there contained in two reports that came out last week.  And yet, they seem to contradict each other.

Let’s see our combatants, shall we:

In this corner we have GAO.  GAO issued THEIR report as a prepared testimony to Congress.  They’ve delivered it numerous times to various committees, and I dare say that Mr Wilshusen is getting some milage with this report.  Basic summary:  numbers are getting better, but 21 out of 24 agencies do not have a complete information security program.

And in this corner we have OMB.  OMB issued THEIR report as a formal report to Congress.  This is a one-shot annual deal, although afterwords there is bound to be some hearings on it.  Basic summary:  we’re doing pretty well and we’re working to police up the odds and ends even more efficiently.

Now keep in mind these two simple facts:  GAO works for Congress (Legislative Branch), OMB works for the President (Executive Branch).  This is critical to remember, so file it away.

The funniest thing for me as an outside observer to look at is that if you look at the numbers that they report, they’re identical.  A view behind the inner workings of the government:  both groups are working off exactly the same sets of data.

In preparing for this testimony, GAO analyzed agency, IG, Office of Management and Budget (OMB), and GAO reports on information security and reviewed OMB FISMA reporting instructions, information technology security guidance, and information on reported security incidents.   –GAO Report

In other words, GAO used exactly what was reported to OMB but came up with different conclusions.  Some of that is to be expected, it’s the same doers v/s the auditors conflict that’s been going on since the beginning of time, but wow, there is a huge disparity here that we need to account for.

I didn’t catch this with the GAO report, but I noticed it with the OMB report:  229 systems are not categorized, but 94% of these are certified and accredited.  Say what?  How can you tell if the security controls are implemented and the residual risk of the system is at an acceptable level when you have not determined what protection needs you have, much less your requirements?  This is akin to saying that a piece of software has passed through user acceptance testing when the user population doesn’t know what their needs or requirements are.  Now occasionally you don’t know how to classify a system because it breaks our model:  a low-criticality network that serves as the backbone for one highly-critical application, a legacy application that it’s just not worth it to classify because we’re in the process of decommissioning it, etc.

Now as much as I want to stand up and tell you that the agencies have been doing outstanding C&As, I just don’t believe the IGs whey they say that some of them have “satisfactory” C&A processes.  Maybe I’m just a little bit cynical, but that’s the way I call it.  I know some of these agencies, no way would I say “satisfactory” for some of them.

Now as far as the contradicting reports, let’s do a hasty analysis of the current political situation in DC, shall we?  The Executive Branch is controlled by the Republicans (and has been for 7 years), the Legislative Branch is controlled by the Democrats (for only a year), and oh yeah, it’s an election year.  You would expect the Congress-owned GAO to sing songs of woe and the President-controlled OMB to sing songs of praise.

And that, dear readers, is the difference between the two reports.

So in the end of all this, which report is the one true report because the other one is full of lies, damn lies, and statistics?  Well, they’re both just as accurate (they came from the same source data, remember), only from different angles.

The cynic/BSOFH in me says that you need to pull out the OMB report most of the time, especially when it’s time for your annual review, and pull out the GAO report when you need to justify your IT security budget.  But no, none of the CISOs or CIOs I know in the government would do that, would they?   =)

• FISMA: Better if PCI. WTF?
• Comments on the Annual OMB Security Report to Congress
• Reading Between the Letters G, A, and O
• GAO’s 5 Steps to “Fix” FISMA
• OMB Wants a Direct Report

Go check it out. In the mean time, I’ll see if there’s anything fun that I need to comment on. Those of you who know me know that there usually is, so prepare for the storm now.  =)

• William Jackson on FISMA: It Works, Maybe
• Old Saint NIST: Ho Ho Hold on, what’s this?
• NIST Security Automation Conference
• C&A Seminar in August, Instructor-to-Coolness Ratio Goes Up!
• GAO’s 5 Steps to “Fix” FISMA

GAO has delivered an updated version of the testimony from February 14th that I talked about here. I’m not going to rehash what I’ve already said, but I want to focus your attention on something I didn’t talk about then: incident statistics.

According to GAO, the number of incidents that were reported to US-CERT increased 259% (*cue shock and awe*, but I think that they forgot to add “average annual increase of 259%” because otherwise the math doesn’t even pass BOTE calculations) from 3634 in FY2005 to 13029 in FY2007. OK, so the number is increasing. But there are several failures in GAO’s logic here that need to be pointed out:

“The need for effective information security policies and practices is further illustrated by the number of security incidents experienced by federal agencies that put sensitive information at risk.”

In other words, they’re trying to indirectly draw a conclusion that the high number of incidents is directly proportional to their audit findings. While this may be true in some (most?) ways, it’s also bad to make this comparison in other ways because you would expect the number of incidents to go down over 2 years because the number of implemented, tested, and integrated security controls has gone up.

So really, what’s the dealio?

The first thing that I would like to point out is that security policies and practices have an indirect impact on security incidents. You don’t have a solid one-for-one comparison that you can use, so I think GAO is doing itself an injustice by trying to correlate these two things. However, you can use incident metrics as a holistic metric for measuring how well your information security program is doing, but overall it’s a very coarse method.

The second thing that I need to point out is the trend of the incident number itself. Anybody who starts tracking incident metrics has to ask themselves one question: because we’re now tracking the number of incidents, does it mean that we’ll now notice that there are more incidents simply due to the fact that we’re now measuring them? It’s the incident response equivalent to Schrödinger’s cat and the Measurement Problem. =)

There’s a couple of reasons that the incident count has increased 259% in just two years:

• First is the awareness of incidents. Government-wide, 2 things have happened in these 2 years that should have increased the number of reportable incidents: maturity of US-CERT to receive and categorize larger amounts of incident data; and the maturity of agencies to have their own incident response and reporting procedures. In short: the infrastructure to respond and report now exists where it really didn’t 3 years ago.
• A series of high-profile incidents around PII followed by OMB mandating that all incidents related to PII be reported to US-CERT within one hour. As a result, many more incidents are now being reported if there is a possibility that there is an incident and if there is a possibility that the incident involved PII because it’s the career-safe move: “When in doubt, report it up”. Whether they admit it or not, the people out in the agencies are now what we could call “gun shy” about PII incidents, and that increases the amount of reported incidents.
• The criteria for an incident is very broad and includes “improper usage”, “scans/probes attempted access”, and “investigations” which is classified as “Unconfirmed incidents that are potentially malicious or anomalous activity deemed by the reporting entity to warrant further review”.

If this were an SIEM or IDS, I would say that we’re flagging on too many things and need to tune our systems down a little bit. Keep in mind that it’s the nature of Government to underreport (when they’re not required to report) and overreport (when they are required to report).

You still need to track the aggregate number of incidents reported to US-CERT and in theory this number should trend downward as we get better at governance at the national level as sort of a “trickle-down infosec economy”. Keep in mind that this number should peak within 5-10 years and then slowly be reduced as we fine-tune our reporting criteria and as we get better at securing information. Of course, I won’t be surprised if it doesn’t due to the threat environment, but that’s a conversation for another day.

However, what I propose is the middle-ground on incident reporting: what we really need to pay attention to for the next couple of years is the number of “severe” incidents. Those are the incidents that have actually have an impact that we really care about. These are mentioned in the GAO report, and we should all be able to recall a handful of them without even seeing what GAO had to say.

Knowing this town, I propose we use “Rybolov’s Washington Post Metric”: How many security incidents were significant enough to be deemed “newsworthy” by the Washington Post and mentioned somewhere. For fine tuning, you could use, say, daily front page v/s the Sunday supplement technology section.

My parting shot for the FISMA-haters:  in the years of yore before FISMA (or GISRA if you want to go back that far), how many of these incidents would have been reported?  It seems like we’re failing if you take the numbers and the reports at face-value, but as GAO says in their title:  “Progress Reported, but Weaknesses at Federal Agencies Persist”.  What more do you need to know?

• Looking for the Charbo Testimony
• OMB Wants a Direct Report
• Cage Match: OMB Report V/S GAO Report, Only One Comes Out Alive
• GAO’s 5 Steps to “Fix” FISMA
• Towards Actionable Metrics

Geeks are cool now.  They even are members of Congress.

Gratz Representative Foster.

• What’s Happening at Wired?
• Blame it all on John Hughes….
• Making Press Releases
• Julie Amero Backlash Reaches Mass Media
• 8 Bit Peoples

How Zombies Work at “How Stuff Works”.  Wow, I can’t believe I didn’t post this before.

 Don’t be in ignorance of the law, know the facts on your legal rights when it comes to making your very own undead minions:

Zombies and Haitian Law

A law that seems to condemn zombie creation went into effect in Haiti in 1835. Article 246 of the Haitian Penal Code classifies the administration of a substance that creates a prolonged period of lethargy without causing death as attempted murder. If the substance causes the appearance of death and results in the burial of the victim, the act is classified as murder.

• Wednesday Zombie Post–Zombies on Uncyclopedia
• Wednesday Zombie Post–Ethical Zombies
• Wednesday Zombie Post–The Zen of Zombie
• Wednesday Zombie Post–Humans v/s Zombies
• Wednesday Zombie Post–The Federal Vampire and Zombie Agency