Posts RSS
Subscribe via Email
Power Outages Do Happen
Posted July 10th, 2007 by rybolov
Finally had one today. It was great. The generators kicked on. The building-wide UPS worked. The NOC and SOC still had power. The other working areas ran out of AC, power, and people before I got there. No problem, the engineers can go elsewhere to work as long as the operations people still have what they need.
Before you ask, no I didn’t create the problem as a BOFH DR test, and the outage did not occur immediately after a line like “so, what does this big, shiny, red button labeled ‘EPO’ really do?”. =)
Similar Posts:
Posted in Risk Management, The Guerilla CISO | 
5 Comments »
July 10th, 2007 at 11:12 pm
You say:
“The other working areas ran out of AC, power, and people before I got there.”
People? Did you dump the bodies outside and threw extra random pieces of furniture at the doors?[1]
Big shiny red buttons are the only way to test DR. What is the use of all the DR preparations if people don’t actually pull the plug unannounced? Last I checked, disasters didn’t quite announce themselves way in advance and asked when would be a good time to strike. 😉
[1] It’s Wednesday here; obligatory zombie reference.
July 11th, 2007 at 10:10 am
I didn’t see you come in and kick the generator dead, did I?
July 11th, 2007 at 11:46 pm
Me? Never. Who do you think I am? Dolphin?
Soooo… are you going to press that big red button in a couple of weeks time, just to see if generator got refuelled after this little outage?
July 12th, 2007 at 7:44 am
You don’t understand, when you push the shiny red button, even the generator can’t help you anymore.
It’s designed for those hopefully rare days when the data center is on fire or knee-deep in water.
July 12th, 2007 at 8:05 pm
Ah, *that* red button. Yeah, I missed the fact you said *EPO* somewhere there.
Talking about data centres knee-deep in water. Have I ever mentioned a company, a global investment bank, that had world’s first underwater data centre? It was all by accident, of course, but rumour had it that facilities people couldn’t believe their luck when they scored that several-football-fields-sized data centre for next to nothing.
Except that besides the cost being next to nothing, the facility itself was next to a river. 😉