Ok, this is my first guest appearance on his blog. I have worked with rybolov in the past and also spoke with him at Potomac Forum events. The topic today is Disaster Recovery/COOP/Contingency Planning depending on what language you speak.

About 3 weeks ago we had an “incident” at our office where our server room lost cooling sometime around midnight one night during the week. Now we thought we had processes and procedures in place that would notify building security, facilities and the proper IT staff in the event of an emergency. Oops, we were wrong!?? It was not until the following morning around 5 am when the first IT staff came in that anyone noticed that the server room was around 90 degrees. During the evening, one of the chillers went down due to a misconfiguration issue and the backup was not able to keep up with demand. Once someone noticed this problem, the AC vendor was notified and sent someone out ASAP to fix the problem. In the meantime, non-mission essential machines were brought down to reduce load, doors were open and makeshift fans were placed in the room to increase airflow.

The next day, one of the security guys (me) decided to investigate the “incident” further to find out what really went wrong. We had many different breakdowns at all levels. For starters, did anyone noticed the temperature alarms going off in the building? Well yes, but we get to that later. Once the guard desk was notified about an hour after the unit went down, a phone call was made. Problem is the person who is on call 24/7 to address facilities issues was unavailable. The guard left a voice mail. Was anyone else called and notified? Nope, there was not a call tree or contact list to follow up with. Issue #1: Create a call tree. It does not do any good to call just one person and give up, especially at 2 am when nobody really wants to answer the phone.

Next item addressed was who called the guard desk to tell them of the temperature issue? I went down and spoke to the guards and they did not take the name or number of who placed the phone call to them in the middle of the night. All they know is that it was some kind of monitoring service. But wait, since when do we have an alarm monitoring service? I asked numerous people in facilities, IT, Finance and Accounting if they knew about any monitoring contracts in place. Everyone was clueless. I called the vendor listed on the thermostat in the room. They had no record of an account with us or our parent company. So after about a week of fishing for this information what should I do? My “creativity” kicked in and I decided to set off the alarm again and this time ask the guard desk to write down the contact information when the monitoring service calls again. At this point I knew the temperature alarm was not tied into fire alarm so I was not worried about fire trucks showing up or sprinklers going off at this point. This is the funny part. I went over to a co-worker and asked her if said had an extra hair dryer I could borrow tomorrow. She looked at me and laughed. One of my nicknames is mini-me because I am bald and look like mini-me from Aston Powers. Next day she brings one in and I am walking around the building with a hair drier getting all sorts of dirty looks (also wearing an Hawaiian shirt for added effect). After heating the thermostat to around 85 degrees for about 30 minutes, BINGO!!!!! The monitoring service calls. It was the same vendor I had contacted a few days before but was a different office in another part of the country. After I spoke to them a few minutes later in the day we found out we were getting free monitoring since February but that is another long story. Issue #2: Document what you have so you can make educated decisions in the future.

The biggest issue out of all this was the airflow in the room itself. I asked our department if anyone had documentation on the BTU load in the room. Nope. OK, I then spent a few hours documenting everything in the room and came up with a rough number. We originally thought we had about 50% additional cooling available for a contingency situation. Wrong again! We are actually right at maximum capacity with very little room for growth. After addressing this issue, we later found out that smaller AC units were installed than originally planned. A basic air flow study of the room was conducted and it was determined the wrong floor tiles were placed in the floor which was causing local hot spots in the room and preventing the correct flow of air to cool everything. Issue #3: Have someone look at the physical layout of the room. Security overlaps many boundaries so make sure to tap many resources for different points of view.

What is the point of all of this? Spend some time on contingency planning to put yourself in a proactive mode instead of waiting until it is too late. Create reasonable contingency plans and actually test them on a periodic basis. Conduct table top exercises with management and incorporate various “what if” scenarios. My favorite was when the toilet on the second floor overflowed one weekend causing water to pour into the first floor electrical closet and completely bring down a large building for an entire week (this actually happened). Maybe think about actually pulling one of your backup tapes and attempting to restore it in real time. Check out the various resources on the web such as www.DRII.org to get ideas. Most of all, don’t wait until it is too late and something bad happens. A little planning can go a long ways.

• Build a Security Program
• Compensating Controls
• Can I at Least Get My Shoes?
• What the Government Looks for in a Product
• Even Better Spam

Those cheeky devils over at NIST have an interesting read out in draft form:  NISTIR 7328 (.pdf caveat).  It’s a draft Interagency Report, but in reality it’s a how-to on being assessed and being the assessor.

I’ve given it a glance and it’s all the things that successful Security Test and Evaluation teams have been doing all along.  I know there’s some kind of “take-away” (my MBA phrase for today) that works out in the private sector.

• Help the Government, Become Literate
• How I Do the “FISMA Thang”
• Micro Digital Signatures Howto
• On SP 800-39
• SP 800-53A Now Finally Final

Hello Everybody

I’ll be teaching again with Potomac Forum at the end of the month.  This will be a 2-day Certification and Accreditation seminar.

• Speaking Again
• C&A Seminar in August, Instructor-to-Coolness Ratio Goes Up!
• Certification and Accreditation Seminar, March 30th and 31st
• C&A Seminar, October 15th and 16th
• NIST Framework for FISMA Dates Announced

OK, this post will be big today. For starters, I use Fortinet products, they’re the heart of my key infrastructure and I’m pretty happy with them.

1. It’s GAO, OMB, and the House Committee on Government Oversight and Reform, not GSA.
2. This blog posting is very unprofessional of you, sir. I would expect more from a Chief Marketing Officer. Will your CEO read about how you treat your customers?
3. Obviously you do not understand your customer base and you are unable to understand their pain points. That is not being a good partner. The appropriate answer is “let’s grab a conference room and talk this over, I want to fix this for you.”
4. You just provided that individual with his migration plan from your gear onto somebody else’s.
5. You need to get out walking more and get some better shoes.
6. Yes, the CIO and his CISO bear most of the responsibility, but if they fail, you fail. Until you understand that, you have much to learn about the Government.

What neither Richard nor his CIO “friend” realize is that it takes a partnership between the Government and the vendors to make it work. Yes, the agencies receive a FISMA grade, but really that failing grade represents the efforts of both the Government and industry. You need to understand that before you go hating on the agencies for low grades.

We all get frustrated dealing with each other. It’s hard for contractors and vendors to understand the Government unless they’ve worked as a GS-scale or SES. I know the contractor side, I know some of the Government side, but I don’t claim to know it all.

But to go out in public and criticize your customers is unthinkable, especially in DC, and especially from a Chief Marketing Officer. You don’t make any permanent enemies here if you can help it, you never know who will end up in charge after the next reorganization.

On the other hand, the purpose of the FISMA grades is to give people a reason to have these conversations. The Government needs to be going to its vendors and saying that they cost too much and don’t fix their problems. That’s supposed to happen, only Richard didn’t handle it well. Don’t tell me this is the first time something like this has ever happened to him.

I just expect more from a vendor and their head of marketing. Thank you for level-setting my expectations for your company, Richard.

• The Vendors are Already Jumping on the 07-11 Bandwagon
• Once Again, I’m not a Bank!
• No, FISMA Doesn’t Require That, Silly Product Pushers
• On Why I Blog… FUD is the Reason for the Writin’
• You Didn’t Outsource Your Responsibilities

This blog post starts with the Data Security Lifecycle. I had a good IM conversation yesterday with Rich Mogull of Securosis fame about his recent Data Security Lifecycle theme. He’s been focusing on classifying data and from there determining what security controls he needs.

The detailed process according to Rich with how they translate to my world:

1. Design your basic classifications. I suggest no more than 3-4, and use plain English. For example, “Sensitive/Internal/Public”. If you deal with personally identifiable information (PII) that can be a separate classification, and call it PII, NPI, HIPAA, or whatever term your industry uses. (SP800-60, determine data types)
2. Pick one type of critical data that is easy to recognize. I highly recommend PII- credit card numbers, Social Security Numbers, or something similar. (FIPS-199)
3. Get executive approval/support- this has to come from as high as possible. If you can’t get it, and you care about security, update your resume. Beating your head against a wall is painful and only annoys the wall and anyone within earshot. (getting your FIPS-199 reviewed/approved by the DAA/CISO/whoever)
4. Issue a memo requiring everyone to identify any business process or IT system that contains this data within 30/60/90 days. (we don’t really do this, but it is a system boundary definition task)
5. Collect results. (write your boundary statement)
6. While collecting the results, finalize security standards for how this data is to be used, stored, and secured. This includes who is allowed to access it (based on business unit/role), approved business processes (billing only, or billing/CRM, etc.), approved applications/systems (be specific), where it can be stored (specific systems and paper repositories), and any security requirements. (grab a baseline of security controls and tailor the h*ll out of them)
7. Security requirements should be templates and standards with specific, approved configurations. Which software, which patch level, which configuration settings, how systems communicate, and so on. If you can’t do this yourself, just point to open standards like those at cisecurity.org. (hardening standards and catalog of controls–800-53 and the “new” SCAP stuff)
8. Issue the security standards. Require business units to bring systems into compliance within a specific time frame, or get an approved exception. (write your draft SSP which details all your security controls)
9. IT Security works with business units to bring systems/processes into compliance. They work with the business and do not play an enforcement role. If exceptions are requested, they must figure out how to secure the data for that business need, and the business will be required to adopt needed alternative security controls for that business process. (C&A staff work with the project team. Not a big deal, but more often than not, they don’t do it.)
10. After the time period to bring systems into compliance expires, the audit group begins random audits of business units to ensure reporting accuracy and that systems are in compliance with corporate standards. (do security test and evaluation)
11. Business units periodically report (rolling schedule) on any changes on use or storage of the now-classified data. (ongoing assessment and mitigation)
12. Security continuously evaluates security standards, issues changes where needed, and helps business units keep the data secure. (annual/periodic security review)
13. Audit plays the enforcement role of looking for exceptions. (annual/periodic security reviews)

Whoa, it’s a blast from the past! Looking back at his process, I realized that he had come full circle and reinvented certification and accreditation. Way back in June I did exactly the same thing and came to the conclusion that there is only one way to do things right, and that way is what Certification and Accreditation was meant to be.

Wait, you all need to see this picture again, this is the one that people should have tattooed on their arm for quick reference:

So the big question is, if C&A is so much nummie goodness, why is it that the conventional wisdom out there in the industry is that C&A doesn’t work? I think it’s 2 things really:

#1 C&A doesn’t work right now. And I’m going to get the locals knocking at my door with torches and pitchforks but one of the reasons we fail at this is because we put the wrong people in charge of C&A. The typical career path for a C&A person usually goes along the following route: English degree => policy and procedures writer => technical writer => security controls documenter=> C&A specialist => end of career. Nowhere in there is anything even remotely close to what a C&A specialist needs. The career path should be something like this: technology degree => IT operations (~2 years) => engineering (~2 years) => security engineering (~3 years) => Security Test and Evaluation Engineer (~2 years) => ISSO => ISSM => CISO. Somewhere around the SE/ISSO timeframe should be some business training.

Notice my second career path doesn’t have a dedicated C&A person. To be bluntfully honest, I don’t believe in a dedicated C&A specialist because all the C&A tasks are really security-in-the-SDLC tasks. So yes, I agree with the naysayers here on that point. To be brutally honest, I think that 3/4 of the dedicated C&A people need to be sleeping on a park bench off Constitution Avenue. But before you get to thinking I’m a complete hater, I also say the same thing about auditors and “those who would disagree with me.” =)

However, when you put the tech writers in charge of SDLC and risk management, they do what they know and what they know is grammar and styleguides, not threat-vulnerability-countermeasure pairings.

#2 SANS and Alan Paller. It’s one of those PR tricks: if you say something enough times, it becomes true. Paller and some of his instructors (not all, mind you) take every opportunity they can to use any news even to “prove” that C&A doesn’t work. Of course, Paller is a different blog post entirely, but truth is, he’s competing for training dollars with the policies and procedures guys and it seems like his job description involves getting as many butts in seats as he can. Hey, he even does a good job at it.  Given my reason #1 above, well, I think Paller is right.  Sometimes.  I’ll go hang my head in shame now. =)

So now I know you’re all thinking: If this C&A thing is so great, how do we fix it and turn it into something that it’s supposed to be instead of a bunch of overpaid ninnies arguing over whether a document should have 1 or 2 spaces after a period? Well, these are what I see as the keys to success:

• Recruitment of skilled engineers into security slots.  We need more clueful people, it’s that simple.
• Cross-training of senior and mid-level managers into some security knowledge.  If they keep thinking that the security people are voodoo practitioners, they can’t help us help them.
• Complimenting the technical side with the business side.  2 different worlds, and the good CISO sits in the middle of them.
• Reallocation of C&A specialist tasks to security engineer or ISSO tasks.  The only reason dedicated C&A specialists exist is because the people who should be doing the job do not understand what the job is–we’re back to peddling voodoo again.
• Understanding the mantra of “Above all, do risk management”–if what you’re doing doesn’t support reducing risk, why are you still doing it?

• Core Belief #4 — Compliance is a Dead-End
• Remembering Accreditation
• A Niche to a Niche is Still Hard to Staff
• An Open Letter to NIST About SP 800-30
• New SP 800-60 is Out, Categorize Yerselves Mo Better

SCAP is becoming one of my favorite government acronyms: Security Content Automation Protocol. OK, what does that mean in English? Well, it’s a glue to hold together a whole slew of xml nummie data goodnesses such as the National Vulnerability Database and a standard for asset inventory management.

I was pretty skeptical on SCAP (and the Federal Desktop Core Configuration–FDCC) when it was first announced–like wow, we have yet another obscure memo from Karen Evans that we have to address.

I had a change of heart after I heard the magical phrase “We know it’s going to break things, and we don’t care”. That made me take notice. I thought about it all weekend–I was getting really riled up over such an obvious irresponsible security hard line. But then I found the magic in what they were doing and learned to stop fearing SCAP and embrace the love that it brings. I’ll tell you why.

Imagine you’re Microsoft. You can’t harden down your OS because you have all the applications vendors (including the A-V/Malware guys) raising the big anti-trust flag. And they’re right to do so. Maybe at one point, you could make your software “secure by default” but that was 20 years ago, and if you would have done so, you would have been last to market.

But that doesn’t work to plug the holes in the OS. In my opinion, it’s the lesson of Vista: if you make it stronger, it breaks applications. We all know that, so a design choice is to either leave the holes or give you a nag-screen or a combination of the two. Speaking strictly from the security side of things, that–along with continuous OS patching–is just “polishing a turd”. Yeah, you can make it all shiny on the outside, but deep down inside it’s still nothing pretty.

But now put yourselves in the Government’s shoes: You buy an OS and spend how much time and effort into OS hardening. That’s money you could spend elsewhere. The people at the top of the Government understand this, that’s why they’re always looking at ways to simplify.

OMB and others have been pushing SCAP pretty hard. So far, most of the focus has been on the databases that exist (CVE, NVD) and the desktop configuration (FDCC).

Think about a pre-hardened Government OS. What it does is break applications–applications that are poorly designed. If your application is poorly designed and doesn’t work with the FDCC, then you’re squeezed out of the public sector. The true capitalists here would say something like “let the market decide who the winners are” or something like that. Realistically, if you want a slice of the federal IT budget, then you need to make your software compatible with their hardening standard. They make it easy to do, with tools to test your software and a certification program.

The part that I like about SCAP is that it’s the Government doing what the OS vendors can’t–put pressure on the applications guys. As usual, this should have a trickle-down effect for the private sector, with the beginning being free hardening guides and the vulnerability databases and the end being a comprehensive information security management toolset.

Check out the presentations from the SCAP conference last month. The Tim Grance presentation (.ppt) alone is worth the price of admission.

Right now SCAP is at the national/CISO level. Give it 6 months and it will be at the forefront of what people are doing.

• Government Can’t Turn on a Dime, News at 11
• Comments on SCAP 2008
• Pre-Hardened Government OS
• NIST and SCAP; Busting a cap on intruders Part 1
• Ed Bellis’s Little SCAP Project