Posts RSS
Subscribe via Email
Split-Horizon Assessments and the Oversight Effect
Posted July 7th, 2010 by rybolov
Going Off the Deep End
So I was thinking the other day (this is the part where people who know me in person usually go “oh cr*p”), partially spurred by a conversation I had with @csoandy and @secbarbie a couple of months ago. I’ll get the idea out there: as an industry we need to embrace the concept of split-horizon assessments.
Two Purposes for Assessments
Because this is an insane approach that I’m just feeling out, let me go on a solo riff and explain what I’m talking about. You see, I have two distinct purposes for getting a security assessment, both of which are in contention with each other:
- I want to fix my security by asking for money to fix the things that need attention. When I get an assessment for this purpose, enumeration of my badness/suckness is good. If I have a set of results that say that everything is great, then there’s no need for me to be given any more resources (time, money, people, gear). Short-term, I’m fine, but what about my infrastructure-type long-term projects? The net effect of a highly-scored annual assessment just might kill my program in 2 years as my funding and people are shifted elsewhere, especially in a .
- I want to keep my job and help my {company|agency|group} stay out of trouble by showing my zero-defects face and by demonstrating my due-diligence in protecting what has been given to me. While the assessor has helped me short-term by identifying my problems and being a total hardass, if I’m not around in 6 months to adopt the recommendations into my security program, has the assessor actually helped me?
And this is the dilemma for just about every security manager out there. One of the strategies is to alternate assessment types, but then your management wonder just what the heck it is you’re doing because you’re on top one year, then on the bottom the next.
Split Rock Lighthouse and Horizon photo by puliarf.
Assessor Window-Shopping
Now for the dirty little secret of the testing business: there are really good testers who are the ninjas of the InfoSec world and there are really bad testers who don’t even validate their unlicensed Nessus scan. I know, you’re shocked and it’s so blindingly obvious that Bruce Schneier will blog it 3 years from now. =)
But there’s the part that you didn’t know: security managers pick their assessor depending on the political mood inside their organization. This is nowhere near a science, from what I’ve seen it involves a lot of navel-gazing on the part of the security team to see which is the lesser evil: having everybody think you’re incompetent or never getting anything new ever again?
Building a Better Rat Race
In order to accomplish both of the goals that I’ve listed, what I really need is a split-horizon assessment. In other words, I need 2 reports from one assessment with different views for different audiences. I know this sounds highly cynical, but it’s something we’ve been doing for some time now but just informally. Might as well make it formal.
So are you sold on this concept yet? In true form, I have an idea on how to get to a world of split-horizon assessments. You can take any catalog of controls and divide it into “gotta have it” and “nice to have” (I almost divide these along the lines of “vulnerability mitigation” and “sustainable security program” or the “CISO” and “OMB and Congress”) buckets. Then in your compliance assessment standard, require 2 reports for each assessment. One is reported to the regulating authority and the other stays with the organization.
Indecision Strikes
I don’t know if I’ve solved the problemspace or not, but I’m looking for feedback “from the Peanut Gallery” so leave some comments.
Similar Posts:
Posted in Rants, What Doesn't Work, What Works | 
7 Comments »
Tags: 800-53 • 800-53A • accreditation • auditor • C&A • catalogofcontrols • certification • collusion • compliance • fisma • government • infosec • itsatrap • management • risk • scalability • security
July 7th, 2010 at 10:30 am
Maybe you’re solving the wrong problem? If a security assessment is placed within a proper risk landscape as part of an overall risk management strategy (say, legal defensibility), then your second bullet is a non-issues because the landscape is ever-changing, meaning you’ll always have work to do. As for your first bullet, it really then comes down to prioritizing/re-prioritizing your risk management activities to ensure that they’re in proper alignment with your risk management strategy, and with your organization’s true tolerance for risk burden. fwiw.
July 7th, 2010 at 5:20 pm
G@ds honest truth is that system owners (SOs) can barely understand one type of risk assessment. Putting on two hats at the same time of good cop and bad cop just doesn’t work. It’s not a bad idea but the reality is that most SOs and business owners hardly understand the technology they are responsible for so how can we expect them to comprehend a nuanced tight rope walk style risk assessment approach? A cold truth of our industry is that we have always been and will always be a cost center. Call me a hardened cynic here but I believe that John and/or Jane business owner don’t know, don’t want to know, and/or just don’t care about information security. John and Jane are happy to go through their day blissfully ignorant of the risks they take.
“Our job is keeping 99% of the population safe from the other 1%. Problem is we have to spend half our lives with that 1%, and the better we do that job, the less the other 99 think they need us. Their clueless, the only ones paying attention on the streets are the cops and the criminals. Everyone is else is just going somewhere, or shopping.” – Robert De Nero as Tom ‘Turk’ Cowan in “Righteous Kill”.
Despite all of the required user training and education we push out. Folks just want to get to the end of the day, punch out, fight traffic, eat dinner, watch tv, sleep (and wash, rinse, repeat). If you give John or Jane business owner the choice of good cop or bad cop they will always choose the more pleasant route of the good friendly cop. We all know that for the most part most IT systems out there will not hold up against a concerted logical attack and that most business owners either know this or live in denial about the threats they face. We (INFOSEC Dweebs) have the unpleasant job of not only being the deliverers of bad news but we also have to continually justify our own existence for a landscape that cannot be seen, touched, or often fully quantified. But when attacked and disrupted sends palatable shockwaves throughout our society.
A possible middle ground approach, rather than a split approach, is engaging the path of continuous improvement. The path acknowledges the monsters in the closet but says that they may/or may not come out and eat you tonight while you’re sleeping. We must acknowledge that the monsters are real and that they can come out of the closet anytime but we also must accept our own limitations (money, people, gear, etc). In the end it’s up to the business owner to decide which monsters are real to them and which are just a squeaky mice. In the end all we can do is point out that there are monsters in the closet, request funding for the right monster killer, document that we did all these things and hope that Jon and/or Jane business owner see the light and go with our recommendations.
There will never be a truly righteous risk assessment as long as our fate is tied to people whose chief concern at the end of the day (and sometimes during the business day) is shopping. We will always be asked to compromise, no sacrifice, what we know is right and good in exchange for happy customers who see us as good cops who aren’t constantly pushing the monster in the closet panic button. We live, eat, breath in this world of good and bad actors everyday so it’s easy to forget that the other 99% just doesn’t give a dam until the day comes when the system has been breached.
July 7th, 2010 at 7:01 pm
[…] This post was mentioned on Twitter by novainfosec, alex knorr. alex knorr said: Split-Horizon Assessments and the Oversight Effect: Going Off the Deep End So I was thinking the other day (this i… http://bit.ly/9RXywE […]
July 8th, 2010 at 1:21 pm
I have to agree this makes things a little too complicated for the average system owner/customer.
I’ve read your stuff for a while and never commented, but I was having this same conversation during an assessment 2 weeks ago, hence this reply. I was discussing with a client how transparency into known risks was a “good thing”, because you can’t fix problems without some investment and you can’t justify investment without being able to show the problem. They wanted us to walk out giving them a squeaky clean report. (we won’t sacrifice integrity for clean results, if our business suffers, so be it…i just hope as an industry we aren’t racing to the bottom in this regard)
I think good assessment results should already include a split horizon concept. Some reports mandate a formal structure, but for those that don’t there should always an executive overview providing nuance and a big picture risk assertion. We need to explain and guide our customers as to what the risks mean to them. THIS is our value as assessors now that tools and methods have become somewhat of a commodity.
If results aren’t presented in a fashion to suit multiple audiences already, you are doing a disservice to your clients.
July 8th, 2010 at 2:59 pm
I’ll admit, I am not sure I understand what the 2 assessments would be (or 2 reports off 1 assessment). I think I understand where you’re going, but not sure. 🙂
I would hope that any real assessment that is going to score a department and impact budget would have some wording about how maintaining that score requires a certain baseline of budget/effort/people and how ongoing changes in the environment also change risk and need to be addressed. Few things are truly static.
I may be operating a level lower than you, so look forward to reading more about your ideas!
August 16th, 2010 at 4:28 pm
Sad that this question is so interesting to me. So what is the solution. I have to say that alot of the customers I talk to fall on the bottom-up side of the fence. IOW, they are deploying tools everywhere without first doing a risk assessment or understanding the business they are securing. As a rsult they have 10s of 1000s of events to sift through each day and end up sending them off to some other company to analyze. Hard to find that one grain of sand on a beach. Yet no customers seem to be begging for the top down tools that would document their systems, supply/chains, risks, etc. in a way that would allow them to more reasonably define and implement controls.
And for compliance, does a top/down, risk based, and business aware approach to security suffer because it is largely associated with large and brain-dead regulatory compliance frameworks? Guilt by association?
But it sure would be nice for every CISO to know that I have spent X $$$ securing this machine that is responsible for Y $$$ in revenue to the business. As a result, I will take people off of security project A to help fund security project B which will mitigate more risk to the business. Etc.
August 16th, 2010 at 4:29 pm
Just to sharpen a point above. I hear more customers asking for feature X or Y in their SIEM product, or VA product, or whatever, more than an analysis of what controls should be placed where and for what reason and for how much.