Worried about this question?  Well, you’re not the only one.  As more and more IT functions are outsourced, we’ll keep wondering and trying to figure it out through trial and error.
Consider all the responsibilities that a security program in the public sector has:

• Policy and procedures creation and maintenance
• Risk Management
• Security Monitoring
• SSP Support
• Incident Response
• Continuity of Operations
• Project team support
• Audit and Compliance

Just about all of these can be fulfilled by contractors.  The interesting thing is that, most of the time, the government doesn’t know how the outsourced system is built because they are only providing oversight.  They depend on the contractor to tell them what the system looks like.
So what parts of security can we never outsource?  Well, the role of system owner has to be
a government employee because it’s a conflict of interest for the contractor to be the ultimate authority on the scope of their project–they can chose the security posture that is more advantageous to their bottom line instead of the needs of the government.

The  rest of security can be outsourced, either as a stand-alone service or bundled with general IT outsourcing.  The trick is, as I have mentioned earlier this week, that you need to do your homework up front before you decide to outsource.

The best approach is to come to the outsourcing RFP with an understanding of what gaps you have in your security program and getting the outsourcing provider to fill those gaps.

So, since we here at ISM-Community believing in bridging from the theoretical to the practical, and since I’ve just given you the theoretical part of this post, how about I give you a couple samples of working models that I have seen in use out around town?

The first example:

• Large contract
• Outsourcing most IT services
• 25+ inventory systems (systems in agency portfolio with an Exhibit 300)
• Existing security organization

While the government already had a fairly well-staffed security organization, they realized that a large contract would require personnel to augment their existing staff.  The security solution provided by the vendor was to staff at one person on the contract as a counterpart to each section in the government security structure (Risk Managment, C&A, Architecture, Policy and Procedures, Incident Response).  The contractors were matrixed and dotted-lined directly into the agency’s security organization.

Another example:

• Medium-sized contract
• Complex data-management system
• One inventory system
• One security person on government side

The security team for the program management office was small–just one person who served as the ISSO for the system and who worked directly for the program management office.  The security solution from the contractor included 5-10 people who provided an end-to-end security solution with the ISSO’s only role to manage the government-side politics and to be the ultimate authority for security decisions on the contract.

Final example:

• Medium-sized contract
• Outsourcing of large web application
• One inventory system
• Well-established, well-staffed security organization

You might think that the contract really doesn’t need a security person, but I have a belief that both the government and the contractor need at least one security point-of-contact.  So this contract was staffed with one person to manage the inter-contractor politics and to provide support to the existing security organization.

• You Didn’t Outsource Your Responsibilities
• Keys to the Outsourcing Kingdom
• Build a Security Program
• Compensating Controls
• Effective Inventory Management

OK, everybody take a deep breath and have a seat.  I’m going to tell you something that is either so obvious to you that you’ll pass out laughing or is so unethical to you that you are going to have an aneurysm and the cops will come beating down my door asking why you died looking at my blog.

So here it is:

When it comes to outsourcing, you cannot own all of the system, even if it’s in your contract.

Let me say that another way:  although you require the contractor to be compliant with FISMA and NIST standards and guidelines and your information security policy and anything else you feel necessary to bind them to, there will be a part of the contract that you are paying money for as a service that will be out of your Certification and Accreditation boundary and that you will not be able to control.

Calmed down yet?  Let me explain what I mean.

For example, consider the circuits that the telcos run to your facilities.  You don’t want to manage them both from a capabilities standpoint and from a risk management standpoint.  If you go and try to manage “your” circuits, and they break, then you have no recourse with the vendor–you have accepted the risk in their eyes.
Quite a few agencies nowadays use MPLS.  You don’t even need to know what the acronym means.  You just need to know that it’s a virtual private Internet that is owned and operated by the telcos.  You can take all your facilities, give them circuits to the MPLS cloud, and you have all the good redundancy of paths that the Internet provides, but in a more-controlled environment.  Deep down inside, you don’t really care how the cloud gets managed, you care that data that enters one end eventually makes it out the other end.
Now you’re contracting for server collocation with a vendor who provides you rackspace in their data center.  They have 20 different agencies with equipment in the data center.  The vendor maintains a common controls package that you now have to rely on for assurance.

Or you have contracted a vendor for end-user support, ie, on-site IT technicians.  These people need to be able to share trouble-ticket information with both people inside their company and the government agency that they support and to interface the trouble-ticket system with their billing, staffing, and cost systems.  Looking at all the requirements for a trouble-ticket system like this, about the only viable way to do it right is to have it on the vendor’s network and accessible from the government LAN/WAN.

I’ve talked to numerous people on how to manage these shared environments.  One Big 4 auditing firm told me that what I needed to do was to determine which agency had the majority of their data on the shared system.  They then would “own” the system with management of the system being a collaborative effort between all the agencies.

I had to wonder, what kind of crack was this guy smoking?  Seriously, most of the time I can’t get people in the same agency to agree with each other, much less people from different agencies with different missions and different risk cultures.  There is bound to be disagreement over how the system is run, with the end result is that the system will not be provided adequate security because it is bogged down in bureaucracy.

Yes, this is what the Lines of Business initiative will provide once it’s working, but even then, you have the same considerations.  Who really owns the various components of the system?

The thing to remember is that the vendors use shared systems because they work–this is the model that commercial providers use whether they tell you or not.  They also provide a significant cost savings to the government through economy of scale.  Inside our NOC, if we have 5 clients, wouldn’t it be nice (not to mention cheaper) to watch one screen instead of 5?  Let’s see, $14000 times 4 equals a nice chunk of currency that I just saved and can carry through to the client.  Beyond 5, and my business doesn’t scale–I can’t have 20 screens to watch, and I can’t afford to hire enough people to watch them all.  This goes the same for data centers, trouble ticket queues, security incident and event monitoring infrastructure, network monitoring systems–the list goes on and on.

I have basic architectural principles I use everyday at work as a managed services provider.  One of the most key principles is segregation of impact: that one client cannot be allowed to affect the service that I provide to another client.  I don’t let all 20 of them make decisions that affect you, and in exchange, I don’t let you affect them.  It’s really how people have coexisted with each other for bazillions of years, only now we’re talking IT security, which gets everybody’s heart pumping.

Part of this principle of segregation of impact means that I have some parts of my service offerings that I can’t give to the government from a boundary perspective.  Because if I do that, then I will have clients managing each other’s space.  It’s not only wrong from a service provider’s standpoint, but somewhere in there, I’ve double-charged the government for services.

The second principle that I use is that of transparency.  I tell you how I do it.  I manage the shared systems just like they are a government system, only I own them.  Our security policy executive statement states that we will meet or exceed government security requirements, keeping in mind that we are not a government agency, so some of the requirements don’t apply to us (reporting to OMB and the government style of CPIC in particular).  However, I do security planning, risk management, contingency planning, all the key points that the government does.

On the government side, what they get is a fairly simple risk management decision:  do we do it ourselfves with whatever costs it entails, do we keep the cost savings of the shared systems and give up some control, or do we want to pay more for something that we now “own”?  That really is the best of both worlds, because the government can determine which option they want based on cost/benefit/risk comparisons.

By providing transparency, I make that decision much easier.

The third principle is transference.  By providing a shared service attached to a service-level agreement (this works very well for the telco examples), the government has effectively transfered the risk to me as a vendor.  The problem is that SLAs for a very complicated outsourcing engagement are harder to define due to the complexity of the solution.  The other caveat is that the government still bears ultimate responsibility when things go wrong, so you still have a certain amount of residual risk.

Notice what I didn’t explicitly say here, but I did describe it in detail… we use compensating controls to manage the risk of the shared infrastructure from the perspective of the government.  That’s the only viable way to manage the risk of outsourcing.

The lastest revision of Special Publication 800-53 talks about this in section 2.4.  Sadly, though, most people don’t read that part.  They usually get to the catalog of controls and start making a checklist out of it.  However, if you are doing any kind of outsourcing in the US government, you need to read and re-read that section.

• Lines of Business, Relationships, and Trustability
• How Much Security Should the Outsourcer Do?
• Always Behind
• Effective Inventory Management
• Feds to Embrace SaaS, End is Nigh for Security!

Do I really need to say this?  As a government agency outsourcing some of its IT functionality, you did not outsource your responsibilities to provide adequate security to the outsourced systems.  OMB, the GAO, and your IG don’t care if you outsourced the system or not–they expect a certain level of security performance.

What you did do when you made a decision to outsource, however, was enter into a partnership with a vendor.  Yes, sometimes it feels like an abusive relationship. =)  But the key here is that it is a partnership.

The key to this partnership is open communication and understanding of the nature of this relationship.  The crux of the problem with outsourcing in the government is that it amplifies the operations-cost-paranoia triangle that you manage on a daily basis.  Where you used to be able to handle security problems internally, now you have several layers of abstraction (contracting officers, funding, 2 political structures) to work through.

So, for example, the life-cycle of a vulnerability looks something like this (everybody does it differently, so don’t cut-n-paste this into a process document unless you’re happy with it):

• Vulnerability discovered by vendor’s security team during monthly scanning
• Vulnerability reported to government ISSO and System Owner
• ISSO requests vendor to provide a rough order of magnitude (ROM) as a cost estimate
• Vendor provides ROM to ISSO through contracting
• System Owner and ISSO make a risk-based decision to mitigate and approve a budget for mitigation
• Mitigation plan transfered through contracts officer to require the vendor to mitigate
• Vendor mitigates the vulnerability and reports completion through contracts officer
• ISSO contacts vendor security team to validate mitigation through a “regression scan”

That’s pretty complex if you think about it, and there are numerous handoffs that could be missed if we don’t have a healthy, communicating relationship.

The conclusion you should have come to by now is that in order to succeed in outsourcing engagements, both sides of the contracting “fence” need to figure out tactics, techniques, and procedures to effectively manage this new layer of complications.

One thing that you’ll notice with outsourcing is that you have to have at least one knowledgeable security person on both sides.

One thing that works is to add a contract clause that lets the contractor’s security people talk to your security people.  The first ground rule is that you decide on what is the security-correct thing to do first, then figure out how to manage the business and contract end of things.

Another thing that works is to decide on a firm delineation of responsibilities before you put the RFP out on the street.  Look at your strengths and weaknesses as an agency.  It could be that you need a full turnkey solution from the vendor along with a large security team to support it because you only have one security practitioner on your project staff (for example, a large LAN/WAN rollout) or you just need a small team from the vendor to dovetail into your already-existing staff.  More on division of labor later this week….

You might want to consider a second, smaller, contract, for security support if you have a small staff on both sides of the contracting relationship.  This adds complexity to our relationship, but it also fixes some conflict-of-interest issues.  An even better way to use a second contractor is for a short engagement to perform a “gap analysis” to tell you what support you need the outsourcing vendor to provide.

Ultimately, though, the thing to remember is that as the government, it is your system, and you have a responsibility to provide adequate security .  Outsourcing just changes the nature of the business, not the endstate.

• How Much Security Should the Outsourcer Do?
• Keys to the Outsourcing Kingdom
• What the Government Looks for in a Product
• Compensating Controls
• Always Behind

Over the next week, I will be doing a blog theme on outsourcing in the US Government and my thoughts on how to effectively manage the security concerns associated with it.  This, of course, dashed with the usual dry wit that people have come to appreciate, or not, as the case may be. =)

For non-government outsourcing, I recommend keeping an eye on the ISM-Community outsourcing forum.  You can even read about myself and kimonos there.

• You Didn’t Outsource Your Responsibilities
• Keys to the Outsourcing Kingdom
• How Much Security Should the Outsourcer Do?
• Compensating Controls
• C&A Seminar, October 15th and 16th

I have a unique vantage point on how security, outsourcing, consulting, and the public sector work.  While I am a contractor who works for a company that provides outsourcing services, most of the time I end up arguing the government’s standpoint to my company, and I end up arguing my company’s standpoint to the government.

In short, I argue a lot.  =)  So much, in fact, that most of the time people have to ask who I work for and they’re shocked to find out the truth.

One issue that I hold close to my heart is how to manage security in outsourcing.  I’ve seen what works, and I’ve seen what doesn’t work, but usually more of the latter.

The truth is that while the government is getting smarter about outsourcing, we still have miles to go.  For example, more often than not a government contract nowadays contains a clause that “the contractor must be compliant with FISMA and NIST standards and guidelines”.  On one hand, I like this, it shows that “The Word” (TM–thanks, Joe) is slowly sinking in.  However, it’s not enough, and I’ll explain why.

The problem is that if you are wearing a contractor hat, what you are looking at is a simple problem: “How do I estimate the cost to me for providing security support to the government as part of this contract?”  This is a problem because “compliant with FISMA and NIST standards and guidelines” runs an entire range of situations from horrible Bataan-death-march malicious compliance contracts to well-managed common controls shops, and unless I have prior experience with this particular agency, I don’t know which side of the spectrum they are on.

As a contractor, when it comes to security, we are almost always responding reactively to security requirements.  This might mean that we have to significantly change the solution mid-stream in order to satisfy what the government wants.  Contractors don’t like this because it means we have to fight with the government over costs, which is just no fun for either of us.  On the government side, we’re wondering why the contractors don’t just Do What I Mean, since I obviously said “be compliant or else”.

Just thinking back, I know what to expect from a handful of agencies, but if they’ve experienced key personnel changes or were suddenly born-again security-wise because of a bad report card, then suddenly my crystal ball has gone black.  Wouldn’t it be nice if we could ask them in advance on which side of the teeter-totter they are sitting on?

Well, I have a partial answer.  You cynics were waiting for this, weren’t you?

How about the government starts their System Security Engineering process and their Certification and Accreditation before they send out the Request for Proposal or Request for Information?  That way, when we send out something for the industry to respond to, they have something to base their security estimates on.  According to the fancy manuals, this is how we should be running the Capital Planning and Investment Control Process all along.  You’re not that mature as an agency?  Don’t worry, you’re not alone.
We have the tools already built to do this.

We have FIPS-199 (ratings for Confidentiality, Integrity, and Availability) which can provide a contractor with a very rough order of magnitude on how much security to build into their response.  As soon as we know what we want to use the system for, we can determine data types and, by extension, criticality.

And we have FIPS-200/SP 800-53 which, although we don’t want to build the entire set of controls, we can select controls out of the catalog which absolutely need to be implemented.  We base this on, take a seat for this one… risk assessment.  We can then put these controls (a dozen or so of the high-priority controls at the most, otherwise you’re defining the solution space and limiting the flexibility of the vendor to be cost-effective) as security requirements or even functional requirements.

Now isn’t that better than a blanket FISMA/NIST clause?

• Keys to the Outsourcing Kingdom
• Imagine that, System Integrators Doing Security Jointly with DoD
• Your Security “Requirements” are Teh Suxxorz
• How Much Security Should the Outsourcer Do?
• Compensating Controls

In the circles I frequent, we have a saying that “Either you do it or you get it”.

The people who do it are fairly smart.  They have a stack of regulations that they are well-versed in.  They talk about matching 800-53 controls to implementation details.  They worry about SSP content.  They’re fairly competent.  They can accomplish most of the information assurance tasks out there.

But these people are only 75% of the solution.  We need more of the second type of people if we are going to succeed as a government with this security game.

There is a small subset of security people who get it.  You know who these people are within 3 minutes of talking to them.  They understand what the “rules” are, but they also understand where you have to break the rules because the rules contradict each other (have cost-effective security but implement this entire catalog of controls).

The difference between these 2 groups of people is that the people who get it understand one additional thing.  They know risk management.  They practice risk management on a minute-by-minute basis.  They are able to make cost/benefit/risk comparisons, which is something that you can’t really learn out of a book.

Doctors have the Hipocratic Oath: “First, do no harm.”  Why don’t security practitioners have the Smith Oath: “Above all, do risk management”?

• Core Belief #1 — Security is not Different
• CSIS and Recommendations
• Mike’s Guide to Information Security Policy
• Blow-By-Blow Commentary
• An Open Letter to NIST About SP 800-30