The Consensus Audit Guidelines (CAG) appear, at this point, to be a reasonable set of guidelines for mediating some human threats. I’m looking forward to seeing what CAG offers and have no doubt there will be worthwhile and actionable controls in the document. That said, there are significant reasons approach CAG with skepticism and assess it critically.

The motivation for CAG is described in a set of slides at the Gilligan Group site. It starts with a focus on what CIO’s fear most: attacks, reduced operational capability, public criticism, data loss, etc. Then it rightly questions whether FISMA is adequately addressing those problems. It doesn’t and this is the genesis of the CAG.

Consensus photo by Eirik Newth.

Unfortunately CAG subsequently develops by pairing this first valid premise with a set of false premises.  These propositions are drawn from slides at gilligangroupinc.com, attributed to John Gilligan or Alan Paller:

1. All that matters are attacks. The central tenet of Bush’s Comprehensive National Cyber Initiative (CNCI) is adopted as the CAG theme: “Defense Must Be Informed by the Offense”. CAG envisions security as defense against penetration attacks. As any seasoned security practitioner knows, attacks are a limited subset of the threats to confidentiality, integrity and availability that information and information systems face.
2. Security through obscurity. CAG seems to have taken the unspoken CNCI theme to heart too, “The most effective security is not exposed to public criticism.” Since its very public December 11th announcement no drafts have been made publicly available for comment.
3. False dichotomy. CAG has been promoted as an alternative to the OMB/NIST approach to FISMA. It isn’t. An alternative would target a fuller range of threats to information and information system security. CAG should be considered a complement to NIST guidance, an addendum of security controls focused on defense against penetration by hackers. NIST has even acted on this approach by including some CAG controls into the 800-53 Rev. 3 catalog of controls.
4. There is too much NIST guidance! This is the implication of one CAG slide that lists 1200 pages of guidance, 15 FIPS docs and the assorted Special Publications not related to FISMA as detriments to security. It’s like complaining that Wikipedia has too many articles to contribute to improved learning. Speaking as someone who scrambled to secure Federal systems before FISMA and NIST’s extensive guidance, having that documentation greatly improves my ability to efficiently and effectively secure systems.
5. NIST guidance doesn’t tell me how to secure my systems! NIST’s FISMA guidance doesn’t step you through securing your SQL Server. The Chairman of the Joint Chiefs also doesn’t deliver your milk. Why not? It’s not their job. NIST’s FISMA guidance helps you to assess the risks to the system, decide how to secure it, secure it accordingly, check that a minimum of controls are in place and then accept responsibility for operating the system. NIST also provides documents, checklists, repositories, standards, working groups and validation of automated tools that help with the actual security implementation.
6. Automated security controls negate human errors. With the premise of all threats being attacks this is nearly a plausible premise. But not all security is technical. Not all threats come from the Internet. DHS, NIST, Mitre, and their partners have pursued automated security controls to enforce and audit security controls for years but automated security controls can only go so far. Human errors, glitches, unexpected conflicts and operational requirements will always factor into the implementation of security.
7. Audit compatibility as a hallmark of good security. There is a conflict of focus at the heart of the CAG, it seeks to both improve its subset of security and improve audit compatibility. For technical controls this is somewhat achievable using automation, something NIST has pursued for years with government and industry partners. For operational and management controls it results in audit checklists. But audits are fundamentally concerned with testing the particular and repeatable, security needs focus on evaluating the whole to ensure the necessary security results. An audit sees if antivirus software is installed, an evaluation sees if the antivirus software is effective.
8. Metrics, but only these metrics over here. When selecting the current crop of CAG controls decisions on what to include were reportedly based on metrics of the highest threats. Great idea, a quantitative approach often discovers counter-intuitive facts. Only the metrics were cherry picked. Instead of looking at all realized threats or real threat impacts only a count of common penetration attacks were considered.
9. With a sample of 1. As a basis for determining what security should focus on the whole breadth of the security profession was queried, so long as they were penetration testers. Yes, penetration testers are some very smart and talented people but penetration testing is to security what HUMINT is to intelligence services. Important players, expert practitioners but limited in scope and best used in conjunction with other intelligence assets.
10. Assessments rely on paper artifacts. The NIST guidance does not require paper artifacts. The first line in the NIST SP 800-53A preface is, “Security control assessments are not about checklists, simple pass-fail results, or generating paperwork to pass inspections or audits-rather, security controls assessments are the principal vehicle used to verify that the implementers and operators of information systems are meeting their stated security goals and objectives.” NIST SP 800-37 specifically and repeatedly states, “Security accreditation packages can be submitted in either paper or electronic format.”

CAG is a missed opportunity. Of the myriad problems with our current FISMA regime a lot of good could be achieved. The problems with guidance have many causes but can be addressed through cooperative development of best practices outside of NIST. The Assessment Cases for SP 800-53A is an example of how cooperative development can achieve great results and provide clear guidance. Other problems exist and can be addressed with better training and community developments.

My hope is that the Consensus Audit Guidelines will move towards a more open, collaborative development environment. The first release is sure to deliver useful security controls against penetration attacks. As with all good security practices it will likely need to go through a few iterations and lots of critical assessment to mature. An open environment would help foster a more complete consensus.

Consensus photo by mugley.

• Clouds of CAG Confusion
• A Funny Thing Happened Last Week on Capital Hill
• How to Not Let FISMA Become a Paperwork Exercise
• When Standards Aren’t Good Enough
• “Machines Don’t Cause Risk, People Do!”

I was attending a conference at NIST (the National Institute of Standards) concerning the SCAP program (Security Content Automation Protocol; pronounced ESS-cap).  SCAP is focused on providing the Federal government with automated, common,  interoperable security solutions.  Specifically the SCAP program has developed a common set of standards for reporting security vulnerabilities for use in automated security scanners, security appliances and reporting systems.

Well, why do we need SCAP?  If we use the Godfather of all vulnerability management tools, the NESSUS vulnerability scanner as an example, we have seen that industry has produced a number of similar products.  Each has its own strengths and rich feature set.  However, none of them use the same “language” for detecting or describing or reporting a potential vulnerability.  This not only means that these various products can only be used to operate with each other with some measure of difficulty but, trying to aggregate and manage the result of reports from these systems can be tedious.

“Tim Bray at XML 2005” photo by Roland.

As a result of these efforts and vision of the dedicated employees at NIST, industry is already scrambling to get their related products SCAP certified.  And, Federal agencies are also specifying in contracts that products must be SCAP certified in order to be qualified for purchase.  This is real progress and great news for the tax payer who will get real better value for their tax dollar.  But, it is not a revolution — yet.  Where I see the revolution emerging is in six-month to a year time frame when industry takes note of the SCAP program and we begin to see SCAP certified and SCAP interoperable products being ordered.  It will not be long after that that we may see the SCAP protocol used in even consumer-level products like personal firewalls.  This ability to give us all a common language will significantly reduce the cost of building and supporting vulnerability scanners and vulnerability reporting tools.  This cost reduction will allow resources to be freed up to address prevention and mitigation concerns in a more meaningful manner.

For example, industry has tools that enable network and security support professionals to detect a mis-configuration in a desktop machine in their network and correct it.  But, only the largest and most well funded security IT security departments have such tools.  With the advent of SCAP, these kind of services will be much more affordable and supportable and thus more common.  In fact, because much of this can be automated, I can even envision the McAfee, Symantec, and others who are well placed in the vulnerability scanning market to offer support services over the wire to smaller businesses and to consumers.  Moreover, as this technology improves and becomes commoditized, I can see ISP’s offering security scanning and mediation as a service to their customers.

• NIST and SCAP; SCAP @ Large Part 2
• Security Automation Developers Conference Slides
• Comments on SCAP 2008
• Save a Kitten, Write SCAP Content
• Federated Vulnerability Management