This article is more than 1 year old
BOFH: The company survived the disaster recovery test. Just. The Director's car, however...
Well, it was an angry mob simulation
Episode 9 "So what happened yesterday?" the CEO asks, not looking too pleased.
"It was an unauthorised action by our contractors," our Director says, nodding at the PFY and myself.
"Or a miscommunication," the Boss adds, pouring oil on the troubled water.
"We were just following orders," the PFY says, setting that oil on fire.
The Nuremberg defence is something typically saved until things are really bad, but in this case the PFY has decided to use it as our opening gambit.
"True enough," I add, interrupting before the Director or Boss can get involved. "We DID say that we thought the whole thing was unwise but they" – I nod back at the Boss and the Director – "INSISTED we had to have full and comprehensive disaster recovery test results to show the auditors."
"You never told us what a disaster would be," The Boss chips back.
"I'd have thought that was implied in the title," I counter.
"So you turned the power to the building off?" the CEO asks.
"It was the only way to simulate the power going off to all the computing devices in the building," the PFY says.
"During the day?" the CEO asks.
"We said it would cost money in overtime and they said they weren't prepared to pay for that," the PFY chips back.
"So you turned the power off, locked the switch room then broke your key off in the lock so that it couldn't be unlocked?!"
"Yes," the PFY says, "to simulate a typical two-hour outage. I did call the locksmith before I did it, though."
"Was breaking the key off really necessary?" the CEO asks.
"If we wanted a comprehensive report on the company's DR capabilities without the interference of others, yes."
"And then you left the building?"
"I did," I respond, "while my assistant here ran things from the office."
"With the door locked and desks piled up against it," the Director says.
"Terrorist action could well be one of the disasters we were planning for," the PFY burbles.
"And so we had a DR system no one could get to because none of their machines had power?"
"The auditor's brief for DR and Business Continuity was to focus on core systems, which were all intact. In some ways it was the perfect outage," I say. "The company website was completely unaffected and phones failed over to the off-site call centre service."
"And you didn't think to run your plan by anyone?"
"We considered it, but it's not much of a disaster if everyone knows when it's happening," the PFY responds, nodding slowly.
"He's right," I say. "You can see that every time there's fire drill – everyone heads straight to the pub 10 minutes before the alarm."
"And that would be why the fire alarms went off at the same time?" the CEO asks.
"Terrorism has no respect for safety..." I say.
"BY SETTING FIRE TO MY CAR!" the Director snaps.
"... or property," I add.
"I see," the Director says. "And what was the point behind beating the car with golf clubs prior to setting it on fire?"
"I believe it was some form of angry mob simulation," The PFY says. "You know Extinction Rebellion protests, that sort of thing. I understand that's why the culprits – whoever they were – were wearing masks. It may even have been some real terrorists – we may never know."
"Did we not have camera footage?"
"Sadly the camera footage went out with the power," I say, "one of the learnings of this whole incident. It seems that the power point in Security labelled 'UPS' was not in fact a UPS power point, which is why all the secure doors released."
"Which explains how someone managed to empty the boardroom drinks cabinet," the CEO says drily. "Did we get any other concrete learnings from this exercise?"
"We have compiled a list of valuable learnings based on our observations at the time, which we fed back to our management," I say, nodding at the Boss and Director. "There's the mislabelling of the UPS outlets in Security that we've already mentioned, the locks failing for the same reason and the building being susceptible to fire alarm triggering from a burning vehicle outside the fresh air intake fan."
"We also noted," the PFY continues, "and have passed on, that we need to lock down the company's Facebook page so that someone can't repeatedly post FIRE SALE pictures to it..."
"Bloody terrorists," I chip in.
"... but I think the main lesson in all of this is that a DR test really needs to be done when people least expect it if you want to get realistic results that aren't skewed by foreknowledge."
"And so you'd claim that the whole exercise was a success?"
"Not that one, no, but the next one is bound to show the efficiency of the management chain of command."
"And that would be happening when?" the CEO asks, just as the power goes out and the fire alarms start.
"Right about now, I guess," the PFY says, turning to the Director. "Bring the wife's car in today, did you?"
Biting the hand that feeds IT
