BOFH: The company survived the disaster recovery test. Just. The Director's car, however...

Well, it was an angry mob simulation


BOFH logo telephone with devil's hornsEpisode 9 "So what happened yesterday?" the CEO asks, not looking too pleased.

"It was an unauthorised action by our contractors," our Director says, nodding at the PFY and myself.

"Or a miscommunication," the Boss adds, pouring oil on the troubled water.

"We were just following orders," the PFY says, setting that oil on fire.

The Nuremberg defence is something typically saved until things are really bad, but in this case the PFY has decided to use it as our opening gambit.

"True enough," I add, interrupting before the Director or Boss can get involved. "We DID say that we thought the whole thing was unwise but they" – I nod back at the Boss and the Director – "INSISTED we had to have full and comprehensive disaster recovery test results to show the auditors."

"You never told us what a disaster would be," The Boss chips back.

"I'd have thought that was implied in the title," I counter.

"So you turned the power to the building off?" the CEO asks.

"It was the only way to simulate the power going off to all the computing devices in the building," the PFY says.

"During the day?" the CEO asks.

"We said it would cost money in overtime and they said they weren't prepared to pay for that," the PFY chips back.

"So you turned the power off, locked the switch room then broke your key off in the lock so that it couldn't be unlocked?!"

"Yes," the PFY says, "to simulate a typical two-hour outage. I did call the locksmith before I did it, though."

"Was breaking the key off really necessary?" the CEO asks.

"If we wanted a comprehensive report on the company's DR capabilities without the interference of others, yes."

"And then you left the building?"

"I did," I respond, "while my assistant here ran things from the office."

"With the door locked and desks piled up against it," the Director says.

"Terrorist action could well be one of the disasters we were planning for," the PFY burbles.

"And so we had a DR system no one could get to because none of their machines had power?"

"The auditor's brief for DR and Business Continuity was to focus on core systems, which were all intact. In some ways it was the perfect outage," I say. "The company website was completely unaffected and phones failed over to the off-site call centre service."

"And you didn't think to run your plan by anyone?"

"We considered it, but it's not much of a disaster if everyone knows when it's happening," the PFY responds, nodding slowly.

"He's right," I say. "You can see that every time there's fire drill – everyone heads straight to the pub 10 minutes before the alarm."

"And that would be why the fire alarms went off at the same time?" the CEO asks.

"Terrorism has no respect for safety..." I say.

"BY SETTING FIRE TO MY CAR!" the Director snaps.

"... or property," I add.

"I see," the Director says. "And what was the point behind beating the car with golf clubs prior to setting it on fire?"

"I believe it was some form of angry mob simulation," The PFY says. "You know Extinction Rebellion protests, that sort of thing. I understand that's why the culprits – whoever they were – were wearing masks. It may even have been some real terrorists – we may never know."

"Did we not have camera footage?"

"Sadly the camera footage went out with the power," I say, "one of the learnings of this whole incident. It seems that the power point in Security labelled 'UPS' was not in fact a UPS power point, which is why all the secure doors released."

"Which explains how someone managed to empty the boardroom drinks cabinet," the CEO says drily. "Did we get any other concrete learnings from this exercise?"

"We have compiled a list of valuable learnings based on our observations at the time, which we fed back to our management," I say, nodding at the Boss and Director. "There's the mislabelling of the UPS outlets in Security that we've already mentioned, the locks failing for the same reason and the building being susceptible to fire alarm triggering from a burning vehicle outside the fresh air intake fan."

"We also noted," the PFY continues, "and have passed on, that we need to lock down the company's Facebook page so that someone can't repeatedly post FIRE SALE pictures to it..."

"Bloody terrorists," I chip in.

"... but I think the main lesson in all of this is that a DR test really needs to be done when people least expect it if you want to get realistic results that aren't skewed by foreknowledge."

"And so you'd claim that the whole exercise was a success?"

"Not that one, no, but the next one is bound to show the efficiency of the management chain of command."

"And that would be happening when?" the CEO asks, just as the power goes out and the fire alarms start.

"Right about now, I guess," the PFY says, turning to the Director. "Bring the wife's car in today, did you?"

Similar topics

Broader topics


Other stories you might like

  • Cheers ransomware hits VMware ESXi systems
    Now we can say extortionware has jumped the shark

    Another ransomware strain is targeting VMware ESXi servers, which have been the focus of extortionists and other miscreants in recent months.

    ESXi, a bare-metal hypervisor used by a broad range of organizations throughout the world, has become the target of such ransomware families as LockBit, Hive, and RansomEXX. The ubiquitous use of the technology, and the size of some companies that use it has made it an efficient way for crooks to infect large numbers of virtualized systems and connected devices and equipment, according to researchers with Trend Micro.

    "ESXi is widely used in enterprise settings for server virtualization," Trend Micro noted in a write-up this week. "It is therefore a popular target for ransomware attacks … Compromising ESXi servers has been a scheme used by some notorious cybercriminal groups because it is a means to swiftly spread the ransomware to many devices."

    Continue reading
  • Twitter founder Dorsey beats hasty retweet from the board
    As shareholders sue the social network amid Elon Musk's takeover scramble

    Twitter has officially entered the post-Dorsey age: its founder and two-time CEO's board term expired Wednesday, marking the first time the social media company hasn't had him around in some capacity.

    Jack Dorsey announced his resignation as Twitter chief exec in November 2021, and passed the baton to Parag Agrawal while remaining on the board. Now that board term has ended, and Dorsey has stepped down as expected. Agrawal has taken Dorsey's board seat; Salesforce co-CEO Bret Taylor has assumed the role of Twitter's board chair. 

    In his resignation announcement, Dorsey – who co-founded and is CEO of Block (formerly Square) – said having founders leading the companies they created can be severely limiting for an organization and can serve as a single point of failure. "I believe it's critical a company can stand on its own, free of its founder's influence or direction," Dorsey said. He didn't respond to a request for further comment today. 

    Continue reading
  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading
  • Amazon investors nuke proposed ethics overhaul and say yes to $212m CEO pay
    Workplace safety, labor organizing, sustainability and, um, wage 'fairness' all struck down in vote

    Amazon CEO Andy Jassy's first shareholder meeting was a rousing success for Amazon leadership and Jassy's bank account. But for activist investors intent on making Amazon more open and transparent, it was nothing short of a disaster.

    While actual voting results haven't been released yet, Amazon general counsel David Zapolsky told Reuters that stock owners voted down fifteen shareholder resolutions addressing topics including workplace safety, labor organizing, sustainability, and pay fairness. Amazon's board recommended voting no on all of the proposals.

    Jassy and the board scored additional victories in the form of shareholder approval for board appointments, executive compensation and a 20-for-1 stock split. Jassy's executive compensation package, which is tied to Amazon stock price and mostly delivered as stock awards over a multi-year period, was $212 million in 2021. 

    Continue reading
  • Confirmed: Broadcom, VMware agree to $61b merger
    Unless anyone out there can make a better offer. Oh, Elon?

    Broadcom has confirmed it intends to acquire VMware in a deal that looks set to be worth $61 billion, if it goes ahead: the agreement provides for a “go-shop” provision under which the virtualization giant may solicit alternative offers.

    Rumors of the proposed merger emerged earlier this week, amid much speculation, but neither of the companies was prepared to comment on the deal before today, when it was disclosed that the boards of directors of both organizations have unanimously approved the agreement.

    Michael Dell and Silver Lake investors, which own just over half of the outstanding shares in VMware between both, have apparently signed support agreements to vote in favor of the transaction, so long as the VMware board continues to recommend the proposed transaction with chip designer Broadcom.

    Continue reading

Biting the hand that feeds IT © 1998–2022