Trouble Tickets

Posted March 21st, 2007 by

In the operations world, if something dies and doesn’t make a ticket, did it really die?

The answer is, of course, “yes”, but there is a caveat:  if it doesn’t make a ticket, it doesn’t get looked at.

This is a simple fact of life in the operations world.  Yes, we have the large screens with network monitoring system dashboards available 24/7, but a red light on a dashboard is not as instantaneous as a trouble ticket.  It’s because people can only do a handful of things at the same time.  They can’t field user calls and at the same time investigate potential problems shown on the big screen because they need undivided attention on the task at hand.

What does this have to do with security?  That’s an interesting question, and an explanation follows.

I’m half toying with the idea of making trouble tickets for vulnerabilities and audit findings, and here’s why:

  • Tickets get assigned to the tier-3/4 administrators to fix
  • Tickets are unavoidable for the most part
  • The ticketing system provides metrics on what is fixed and not fixed
  • The operations guys are already accustomed to having tickets introduced as a stimulus
  • Our operational staff is rated on the number of tickets that they close
  • Ticketing systems support the operational work flow

Notice what I didn’t explicitly say here?  I’m adapting my security mindset to the operational mindset because that is my environment.  It’s strange because I’ve always been more in the engineering world, so I have to wrap my brain around the operations way of doing things.



Similar Posts:

Posted in Outsourcing, Technical, What Works | No Comments »

Clearances and Economy of Scale

Posted March 20th, 2007 by

Our government hasn’t had a ton of time to get their act together.  It’s only been 225 years, give or take, and in that time, we might have learned a thing or two.

So why is it that it took us this long before we could get one agency to recognize the clearances from another agency?  Even though OMB keeps trying to unify the clearance management systems, as a managed service provider I still have to convince every new client that I’m trustworthy because I have a DoD Top Secret.
So what’s the benefit?  Well, from my angle, one unified clearance system means economy of scale simply by avoiding a “Not Invented Here” attitude.  If I have 5 clients, 100 employees, and $5000 per person for a clearance, the savings add up really quickly if I just have to clear people once.

Now the strange thing is that by agencies having their own individual clearance system, we are creating an artificial scarcity of cleared people.  For somebody with a TS/SCI, starting salary in the DC area is around $80K/year, simply because of lack of supply.  That’s an indirect cost that we could avoid.



Similar Posts:

Posted in Odds-n-Sods, Outsourcing, What Doesn't Work, What Works | No Comments »

Keys to the Outsourcing Kingdom

Posted March 9th, 2007 by

While I’m still on an outsourcing kick, let me go through what I call the keys to the wonderful kingdom of outsourcing.

In true ISM-Community form, let me give you a “Top 10 Keys to Outsourcing” list.

  1. Know Thyself:  Like the Oracle at Delphi, the most important thing you can do before outsourcing is to know where your weaknesses are and to sculpt the contract to your needs.  More on this subject.
  2. Bottom Line up Front:  It works for newspaper editors, it can also for you.  Being as explicit with security categorization and must-have requirements as you can without defining the solution will allow the contractor to determine a solid security model and staffing to support the outsourcing effort.  More on this subject.
  3. Use Compensating Controls:  You can’t control the vendor’s infrastructure, so use compensating controls where you have to sacrifice control for economy of scale.  More on this subject.
  4. Retain Authority to Make Decisions:  This is why the contractor shouldn’t be the system owner.  If you lose the authority to make decisions, you will be “worked over” by the vendor.  More on this subject.
  5. Understand the Added Complexity of Outsourcing:  Things sometimes get slower when you outsource them because there is the added layer of abstraction in the contracting officers.  More on this subject.
  6. Have Requirements-Driven Roles and Responsibilities:  Start with what your security requirements, roles, and responsibilities and then divide them up between government and contractor.  More on this subject.
  7. Tie into the Existing Security Organization:  Create a dotted line on the organization chart between your organization and the contractor’s.  More on this subject.
  8. Get a Gap Analysis:  Bring in a third party to assess you for areas that need fixing, both pre-outsourcing and during outsourcing.  More on this subject.
  9. Hedge Your Bets:  By having a minimum of one security representative on either side of the contracting fence, you have a parallel path other than through contracting officers.  More on this subject.
  10. 2 Is Sometimes better than 1:  If you have IT services outsourced through one contractor, you can have a second contractor provide security support.  It sometimes adds more complexity at reducing conflict-of-interest.  More on this subject.
  11. (The Freebie) It’s all About Transparency:  Be wary of vendors and contractors that withhold information.  In a partnership, you share information.  More on this subject.


Similar Posts:

Posted in FISMA, NIST, Outsourcing, What Works | No Comments »

How Much Security Should the Outsourcer Do?

Posted March 8th, 2007 by

Worried about this question?  Well, you’re not the only one.  As more and more IT functions are outsourced, we’ll keep wondering and trying to figure it out through trial and error.
Consider all the responsibilities that a security program in the public sector has:

  • Policy and procedures creation and maintenance
  • Risk Management
  • Security Monitoring
  • SSP Support
  • Incident Response
  • Continuity of Operations
  • Project team support
  • Audit and Compliance

Just about all of these can be fulfilled by contractors.  The interesting thing is that, most of the time, the government doesn’t know how the outsourced system is built because they are only providing oversight.  They depend on the contractor to tell them what the system looks like.
So what parts of security can we never outsource?  Well, the role of system owner has to be
a government employee because it’s a conflict of interest for the contractor to be the ultimate authority on the scope of their project–they can chose the security posture that is more advantageous to their bottom line instead of the needs of the government.

The  rest of security can be outsourced, either as a stand-alone service or bundled with general IT outsourcing.  The trick is, as I have mentioned earlier this week, that you need to do your homework up front before you decide to outsource.

The best approach is to come to the outsourcing RFP with an understanding of what gaps you have in your security program and getting the outsourcing provider to fill those gaps.

So, since we here at ISM-Community believing in bridging from the theoretical to the practical, and since I’ve just given you the theoretical part of this post, how about I give you a couple samples of working models that I have seen in use out around town?

The first example:

  • Large contract
  • Outsourcing most IT services
  • 25+ inventory systems (systems in agency portfolio with an Exhibit 300)
  • Existing security organization

While the government already had a fairly well-staffed security organization, they realized that a large contract would require personnel to augment their existing staff.  The security solution provided by the vendor was to staff at one person on the contract as a counterpart to each section in the government security structure (Risk Managment, C&A, Architecture, Policy and Procedures, Incident Response).  The contractors were matrixed and dotted-lined directly into the agency’s security organization.

Another example:

  • Medium-sized contract
  • Complex data-management system
  • One inventory system
  • One security person on government side

The security team for the program management office was small–just one person who served as the ISSO for the system and who worked directly for the program management office.  The security solution from the contractor included 5-10 people who provided an end-to-end security solution with the ISSO’s only role to manage the government-side politics and to be the ultimate authority for security decisions on the contract.

Final example:

  • Medium-sized contract
  • Outsourcing of large web application
  • One inventory system
  • Well-established, well-staffed security organization

You might think that the contract really doesn’t need a security person, but I have a belief that both the government and the contractor need at least one security point-of-contact.  So this contract was staffed with one person to manage the inter-contractor politics and to provide support to the existing security organization.



Similar Posts:

Posted in FISMA, NIST, Outsourcing, What Works | No Comments »

Compensating Controls

Posted March 7th, 2007 by

OK, everybody take a deep breath and have a seat.  I’m going to tell you something that is either so obvious to you that you’ll pass out laughing or is so unethical to you that you are going to have an aneurysm and the cops will come beating down my door asking why you died looking at my blog.

So here it is:

When it comes to outsourcing, you cannot own all of the system, even if it’s in your contract.

Let me say that another way:  although you require the contractor to be compliant with FISMA and NIST standards and guidelines and your information security policy and anything else you feel necessary to bind them to, there will be a part of the contract that you are paying money for as a service that will be out of your Certification and Accreditation boundary and that you will not be able to control.

Calmed down yet?  Let me explain what I mean.

For example, consider the circuits that the telcos run to your facilities.  You don’t want to manage them both from a capabilities standpoint and from a risk management standpoint.  If you go and try to manage “your” circuits, and they break, then you have no recourse with the vendor–you have accepted the risk in their eyes.
Quite a few agencies nowadays use MPLS.  You don’t even need to know what the acronym means.  You just need to know that it’s a virtual private Internet that is owned and operated by the telcos.  You can take all your facilities, give them circuits to the MPLS cloud, and you have all the good redundancy of paths that the Internet provides, but in a more-controlled environment.  Deep down inside, you don’t really care how the cloud gets managed, you care that data that enters one end eventually makes it out the other end.
Now you’re contracting for server collocation with a vendor who provides you rackspace in their data center.  They have 20 different agencies with equipment in the data center.  The vendor maintains a common controls package that you now have to rely on for assurance.

Or you have contracted a vendor for end-user support, ie, on-site IT technicians.  These people need to be able to share trouble-ticket information with both people inside their company and the government agency that they support and to interface the trouble-ticket system with their billing, staffing, and cost systems.  Looking at all the requirements for a trouble-ticket system like this, about the only viable way to do it right is to have it on the vendor’s network and accessible from the government LAN/WAN.

I’ve talked to numerous people on how to manage these shared environments.  One Big 4 auditing firm told me that what I needed to do was to determine which agency had the majority of their data on the shared system.  They then would “own” the system with management of the system being a collaborative effort between all the agencies.

I had to wonder, what kind of crack was this guy smoking?  Seriously, most of the time I can’t get people in the same agency to agree with each other, much less people from different agencies with different missions and different risk cultures.  There is bound to be disagreement over how the system is run, with the end result is that the system will not be provided adequate security because it is bogged down in bureaucracy.

Yes, this is what the Lines of Business initiative will provide once it’s working, but even then, you have the same considerations.  Who really owns the various components of the system?

The thing to remember is that the vendors use shared systems because they work–this is the model that commercial providers use whether they tell you or not.  They also provide a significant cost savings to the government through economy of scale.  Inside our NOC, if we have 5 clients, wouldn’t it be nice (not to mention cheaper) to watch one screen instead of 5?  Let’s see, $14000 times 4 equals a nice chunk of currency that I just saved and can carry through to the client.  Beyond 5, and my business doesn’t scale–I can’t have 20 screens to watch, and I can’t afford to hire enough people to watch them all.  This goes the same for data centers, trouble ticket queues, security incident and event monitoring infrastructure, network monitoring systems–the list goes on and on.

I have basic architectural principles I use everyday at work as a managed services provider.  One of the most key principles is segregation of impact: that one client cannot be allowed to affect the service that I provide to another client.  I don’t let all 20 of them make decisions that affect you, and in exchange, I don’t let you affect them.  It’s really how people have coexisted with each other for bazillions of years, only now we’re talking IT security, which gets everybody’s heart pumping.

Part of this principle of segregation of impact means that I have some parts of my service offerings that I can’t give to the government from a boundary perspective.  Because if I do that, then I will have clients managing each other’s space.  It’s not only wrong from a service provider’s standpoint, but somewhere in there, I’ve double-charged the government for services.

The second principle that I use is that of transparency.  I tell you how I do it.  I manage the shared systems just like they are a government system, only I own them.  Our security policy executive statement states that we will meet or exceed government security requirements, keeping in mind that we are not a government agency, so some of the requirements don’t apply to us (reporting to OMB and the government style of CPIC in particular).  However, I do security planning, risk management, contingency planning, all the key points that the government does.

On the government side, what they get is a fairly simple risk management decision:  do we do it ourselfves with whatever costs it entails, do we keep the cost savings of the shared systems and give up some control, or do we want to pay more for something that we now “own”?  That really is the best of both worlds, because the government can determine which option they want based on cost/benefit/risk comparisons.

By providing transparency, I make that decision much easier.

The third principle is transference.  By providing a shared service attached to a service-level agreement (this works very well for the telco examples), the government has effectively transfered the risk to me as a vendor.  The problem is that SLAs for a very complicated outsourcing engagement are harder to define due to the complexity of the solution.  The other caveat is that the government still bears ultimate responsibility when things go wrong, so you still have a certain amount of residual risk.

Notice what I didn’t explicitly say here, but I did describe it in detail… we use compensating controls to manage the risk of the shared infrastructure from the perspective of the government.  That’s the only viable way to manage the risk of outsourcing.

The lastest revision of Special Publication 800-53 talks about this in section 2.4.  Sadly, though, most people don’t read that part.  They usually get to the catalog of controls and start making a checklist out of it.  However, if you are doing any kind of outsourcing in the US government, you need to read and re-read that section.



Similar Posts:

Posted in FISMA, NIST, Outsourcing, What Doesn't Work, What Works | 1 Comment »

You Didn’t Outsource Your Responsibilities

Posted March 6th, 2007 by

Do I really need to say this?  As a government agency outsourcing some of its IT functionality, you did not outsource your responsibilities to provide adequate security to the outsourced systems.  OMB, the GAO, and your IG don’t care if you outsourced the system or not–they expect a certain level of security performance.

What you did do when you made a decision to outsource, however, was enter into a partnership with a vendor.  Yes, sometimes it feels like an abusive relationship. =)  But the key here is that it is a partnership.

The key to this partnership is open communication and understanding of the nature of this relationship.  The crux of the problem with outsourcing in the government is that it amplifies the operations-cost-paranoia triangle that you manage on a daily basis.  Where you used to be able to handle security problems internally, now you have several layers of abstraction (contracting officers, funding, 2 political structures) to work through.

So, for example, the life-cycle of a vulnerability looks something like this (everybody does it differently, so don’t cut-n-paste this into a process document unless you’re happy with it):

  • Vulnerability discovered by vendor’s security team during monthly scanning
  • Vulnerability reported to government ISSO and System Owner
  • ISSO requests vendor to provide a rough order of magnitude (ROM) as a cost estimate
  • Vendor provides ROM to ISSO through contracting
  • System Owner and ISSO make a risk-based decision to mitigate and approve a budget for mitigation
  • Mitigation plan transfered through contracts officer to require the vendor to mitigate
  • Vendor mitigates the vulnerability and reports completion through contracts officer
  • ISSO contacts vendor security team to validate mitigation through a “regression scan”

That’s pretty complex if you think about it, and there are numerous handoffs that could be missed if we don’t have a healthy, communicating relationship.

The conclusion you should have come to by now is that in order to succeed in outsourcing engagements, both sides of the contracting “fence” need to figure out tactics, techniques, and procedures to effectively manage this new layer of complications.

One thing that you’ll notice with outsourcing is that you have to have at least one knowledgeable security person on both sides.

One thing that works is to add a contract clause that lets the contractor’s security people talk to your security people.  The first ground rule is that you decide on what is the security-correct thing to do first, then figure out how to manage the business and contract end of things.

Another thing that works is to decide on a firm delineation of responsibilities before you put the RFP out on the street.  Look at your strengths and weaknesses as an agency.  It could be that you need a full turnkey solution from the vendor along with a large security team to support it because you only have one security practitioner on your project staff (for example, a large LAN/WAN rollout) or you just need a small team from the vendor to dovetail into your already-existing staff.  More on division of labor later this week….

You might want to consider a second, smaller, contract, for security support if you have a small staff on both sides of the contracting relationship.  This adds complexity to our relationship, but it also fixes some conflict-of-interest issues.  An even better way to use a second contractor is for a short engagement to perform a “gap analysis” to tell you what support you need the outsourcing vendor to provide.

Ultimately, though, the thing to remember is that as the government, it is your system, and you have a responsibility to provide adequate security .  Outsourcing just changes the nature of the business, not the endstate.



Similar Posts:

Posted in FISMA, NIST, Outsourcing, Risk Management, What Works | No Comments »

« Previous Entries Next Entries »


Visitor Geolocationing Widget: