Ok, this is my first guest appearance on his blog. I have worked with rybolov in the past and also spoke with him at Potomac Forum events. The topic today is Disaster Recovery/COOP/Contingency Planning depending on what language you speak.

About 3 weeks ago we had an “incident” at our office where our server room lost cooling sometime around midnight one night during the week. Now we thought we had processes and procedures in place that would notify building security, facilities and the proper IT staff in the event of an emergency. Oops, we were wrong!?? It was not until the following morning around 5 am when the first IT staff came in that anyone noticed that the server room was around 90 degrees. During the evening, one of the chillers went down due to a misconfiguration issue and the backup was not able to keep up with demand. Once someone noticed this problem, the AC vendor was notified and sent someone out ASAP to fix the problem. In the meantime, non-mission essential machines were brought down to reduce load, doors were open and makeshift fans were placed in the room to increase airflow.

The next day, one of the security guys (me) decided to investigate the “incident” further to find out what really went wrong. We had many different breakdowns at all levels. For starters, did anyone noticed the temperature alarms going off in the building? Well yes, but we get to that later. Once the guard desk was notified about an hour after the unit went down, a phone call was made. Problem is the person who is on call 24/7 to address facilities issues was unavailable. The guard left a voice mail. Was anyone else called and notified? Nope, there was not a call tree or contact list to follow up with. Issue #1: Create a call tree. It does not do any good to call just one person and give up, especially at 2 am when nobody really wants to answer the phone.

Next item addressed was who called the guard desk to tell them of the temperature issue? I went down and spoke to the guards and they did not take the name or number of who placed the phone call to them in the middle of the night. All they know is that it was some kind of monitoring service. But wait, since when do we have an alarm monitoring service? I asked numerous people in facilities, IT, Finance and Accounting if they knew about any monitoring contracts in place. Everyone was clueless. I called the vendor listed on the thermostat in the room. They had no record of an account with us or our parent company. So after about a week of fishing for this information what should I do? My “creativity” kicked in and I decided to set off the alarm again and this time ask the guard desk to write down the contact information when the monitoring service calls again. At this point I knew the temperature alarm was not tied into fire alarm so I was not worried about fire trucks showing up or sprinklers going off at this point. This is the funny part. I went over to a co-worker and asked her if said had an extra hair dryer I could borrow tomorrow. She looked at me and laughed. One of my nicknames is mini-me because I am bald and look like mini-me from Aston Powers. Next day she brings one in and I am walking around the building with a hair drier getting all sorts of dirty looks (also wearing an Hawaiian shirt for added effect). After heating the thermostat to around 85 degrees for about 30 minutes, BINGO!!!!! The monitoring service calls. It was the same vendor I had contacted a few days before but was a different office in another part of the country. After I spoke to them a few minutes later in the day we found out we were getting free monitoring since February but that is another long story. Issue #2: Document what you have so you can make educated decisions in the future.

The biggest issue out of all this was the airflow in the room itself. I asked our department if anyone had documentation on the BTU load in the room. Nope. OK, I then spent a few hours documenting everything in the room and came up with a rough number. We originally thought we had about 50% additional cooling available for a contingency situation. Wrong again! We are actually right at maximum capacity with very little room for growth. After addressing this issue, we later found out that smaller AC units were installed than originally planned. A basic air flow study of the room was conducted and it was determined the wrong floor tiles were placed in the floor which was causing local hot spots in the room and preventing the correct flow of air to cool everything. Issue #3: Have someone look at the physical layout of the room. Security overlaps many boundaries so make sure to tap many resources for different points of view.

What is the point of all of this? Spend some time on contingency planning to put yourself in a proactive mode instead of waiting until it is too late. Create reasonable contingency plans and actually test them on a periodic basis. Conduct table top exercises with management and incorporate various “what if” scenarios. My favorite was when the toilet on the second floor overflowed one weekend causing water to pour into the first floor electrical closet and completely bring down a large building for an entire week (this actually happened). Maybe think about actually pulling one of your backup tapes and attempting to restore it in real time. Check out the various resources on the web such as www.DRII.org to get ideas. Most of all, don’t wait until it is too late and something bad happens. A little planning can go a long ways.

• Build a Security Program
• Compensating Controls
• Can I at Least Get My Shoes?
• What the Government Looks for in a Product
• Even Better Spam

Seth Godin’s take on bacn, the spam you get from social networking sites to let you know that somebody has replied to your comment.

Living with 3 socially-aware people (read: girls) aged 10 to 37, I have a simple solution:  procmail rules to kill all the MySpace/Facebook/$FooSpace notifications on my server so the 200+ pieces of mail never make it to the /dev/null inbox. =)

Word to social networking sites:  put the entire content of the response in the email that way users can decide if it’s worth their time to respond or if it’s just somebody saying “OMFG me 2 gf LOL”.  Your users shouldn’t have to go to your website to read every one of these.

And hats off to the Word Press blog  software–it does put the text of a blog comment into the email notification along with the link to moderate.  That’s the way things are supposed to be.

• Targeted Spam
• A Day in the Life of the Feedmaster
• Omigod, I’m Part of a Botnet?!?!?!
• Server Upgrades
• Realistic NSTIC

I had dinner with Joe last night, and I thought I would add a little bit of fuel on his personal vendetta to rid the world of the concept of “SBU”–Sensitive But Unclassified. Let’s just say that I’m an anti-SBU sympathizer. =)

“SBU” is a pseudo-classification used by the government to say that a bit of information is unclassified but still needs to be protected.

The biggest question is, does the US Government have any data that is unsensitive in any way? Usually not. I’m trying to think of something, and I am drawing a complete blank, unless we want to talk about orders for new black Skilcraft ballpoints and Simple Green. But then again, there’s probably a purchase order involved which probably is sensitive in some way.  You could even extrapolate a traffic analysis attack using the quantity of pens ordered to determine how many people work at a specific place (not as effective as using the volume of pizza ordered by the Pentagon during planning for a troop surge as an indicator of pending missions), but when I start to go down that road I put on the tinfoil hat and the thoughts go away. =)

DODD 8500.1 defines SBU as “A term commonly and inappropriately used within the Department of Defense as a synonym for Sensitive Information, which is the preferred term.” Then there is a lengthy definition for Sensitive Information which you can go look up yourself.

Seriously, though, the last thing we need is for people to be making up their own classifications without official limits on what you can and can’t do with it. If you can’t mark it on a document and have people know what the marking means, then it’s not an effective classification. I think SBU meets this description, and that’s why it must die.

We have a classification, it’s called “For Official Use Only”. Use it, folks! =)

• The World Asks: is S.773 Censorship?
• Indicator Species
• Towards Actionable Metrics
• It’s All About Common Controls!
• Been Off Teaching

I promised myself I would stop with the vendor bashing at least long enough to catch my breath. Well, sometimes in your life something comes along that you just can’t help but comment on.

Press release on how a network emulator can help with FISMA reporting.

This class of products is great–simulated network lag so you can test your network devices, software, etc. Every lab should have this stuff.  I’m pretty sure that some of it is inside my building in the various replicas of customer networks that the engineers use.

But what does this have to do with information security management? Once again, it’s sprinkling the magic FISMA fairy dust and wishing that it makes your product a security device.  Makes me had the”make it secure” wand (complete with star on end and ribbons) that one CISO I know of carries about just for the purpose of being able to wave it around and say “*Poof* It’s secure now.”  I figure happy thoughts are in there somewhere, but I’m just not seeing the exact mechanism.

My friends have a theory that I should start selling SOX socks and FISMA underwear. I’m not so sure about that, but I figure if it works for all these other products, it might be a massive moneymaker for me.  =)

• Do You Know What FISMA Is?
• Blog Statistics and Search Strings
• Selling Water to People in the Desert
• On Why I Blog… FUD is the Reason for the Writin’
• Digital Forensics: Who should make the keys?

Interesting news article about some of Boeing’s problems.

This is an industry problem, one that we don’t talk about too much, and the heart of it is that it’s hard to manage security in huge organizations. Sure, there is the infosec frameworks like 7799/27001, FISMA, etc. If you look at the fairly undeveloped pieces of security, you’ll notice some trends:

• At the tactical level, we know vulnerability scanning, exploit writing, and hardening standards.
• At the operational level (Army sense of operational–we’re talking brigades and divisions here), we have risk management, certification, and my favorite whipping-boy, compliance.
• At the strategic level, we have enterprise architecture, inventory management, and capital planning.

My opinion, and it’s purely opinion, is that as you progress up the ladder to strategy, there is less and less of a knowledge base and a higher rate of opportunity for charlatans. But then again, it echoes IT management in general–everybody knows how to build a fairly secure server, not a whole lot of people know how to manage IT infrastructure for 75K users.

Purely as a sidenote, ISM-Community is working to be a player in the operational and strategic area of security, I’m just trying to figure out how to get more people involved.

• Core Belief #4 — Compliance is a Dead-End
• Lousy Security Advice
• In Which Our Protagonist Discovers We Need More Good Public Policy People Who Understand Security
• NIST’S FISMA Pase II–Who Certifies Those who Certify the Certifiers?
• Build a Security Program

It seems like every product or service that somebody is trying to sell me has the words “bank” or “financial institution” attached to it. The cynic in me would say that either the SOX cash cow is drying up and the vendors are trying to glom onto FISMA, or the only past performance that these small-fry vendors have is with a bank that bought their solution once.

Part of me also wants to know if banks will buy whatever junk I throw at them. =)

So is the secret to selling a product to the government a cleverly crafted Unix shell command like the following:

cat marketing.literature.sox.txt \

| sed ‘s/SOX/FISMA/’ \

| sed ‘s/bank/government agency/’ \

> marketing.literature.fisma.txt

You would think so based on the spam I get nowadays. It’s so obviously retreaded that I keep wondering “Do you guys even believe your own literature and hyperbole about what you’re trying to sell?” I don’t expect sales people to be the experts at my business, but how can you offer me a solution to my problems if you don’t understand the gist of what my problems are? If you don’t know that bank security is primarily modeled on integrity and that government security is primarily modeled on confidentiality, then we don’t really have a common language.

My vendor spam for today is below. “Compliance as a Service” makes my head explode. I think somehow I should be building a list of security spammers as a “Wall of Shame” to help out the people who would actually buy from these vendors. If anything, I’ll know who not to buy from–the list is getting large enough so that I need to write it down to keep track of.

 

Dear Rybolov,

The need for automated Security Review processes had already made developments in risk tracking one of the areas of greatest interest (and concern) to CIOs, CSOs, and Security Managers worldwide. Now, with the news of Google’s acquisition of Postini, many enterprise organizations are looking even more closely at risk management and compliance as a service.

Many companies lack a repeatable, automated security risk assessment process, and <redacted> would like to offer you a case study that provides an overview of how a leading global financial service provider was able to take advantage of compliance as a service to address risk management and compliance issues while improving business performance.

The specialists at <redacted> are pleased to offer you this case study in an effort to reduce the background noise surrounding this issue and help you focus on the aspects of the process that matter most.

To download this case study at no cost and with no obligation, simply visit: <redacted>

• On Why I Blog… FUD is the Reason for the Writin’
• Database Activity Monitoring for the Government
• More Vendor Craziness
• The Vendors are Already Jumping on the 07-11 Bandwagon
• Ack! With the Mandates!