BOFH: The Great Backup BACKDOWN

'We have to be sure the system works!'


Episode 7

"So I see that backup check went well?" the Boss asks, trying to drag me into a conversation that will have nothing to do with me, but sounds technical enough that I would be an idiot and say...

"What backup check?" the PFY pre-empts me.

NGAAAAAAARGH!

"The backup check," says the Boss. "The Financial Director wanted to be sure that we would be able to recover our data in the event of a system failure, so he deleted some files and then asked for them to be recovered."

"Was that what I was doing yesterday," the PFY asks, "running around recovering files which were deleted on purpose just to test the system?"

"Well we have to be sure the system works," The Boss says.

"So what if it hadn't worked?" the PFY asks.

"In that case we would have recovered them from the USB stick where the Financial Director had saved them."

"So you had him running around recovering files which were deleted on purpose and were in fact just on a USB stick and could have been recovered in a couple of minutes?" I ask.

"WE HAVE TO BE SURE THE SYSTEM WORKS!!!" the Boss says. "And it does!"

"You realise that a far safer way of performing that test would have been to have CREATED SOME NEW FILES then deleted them a day or two later and seen if they were recoverable?" the PFY asks.

"I don't see how tha..."

"Because if you're REALLY testing the system you'd be making sure that files are continuously being backed up. I mean I could have probably recovered those Policy Document files and the Budget projection files from six weeks ago and you would never have known because they haven't changed for months..."

"Yes, but we needed to see if the system works. Which it did."

"It APPEARED to work," I say. "If you'd talked to us about it we could have told you about some files that would have actually PROVED that it worked. Files that had changed in the last 24 hours and that should have been recoverable."

"But we couldn't talk to you about it. If there'd been some problem with your backups then you might have realised and fixed them before the test."

"Ah, so the point of the exercise was to catch us out and not check the recoverability of files?"

"Uh. No?"

"But surely if you wanted to prove that files could be recovered you could have just come here and asked us to get back a file from yesterday?"

"Uuuuh, no, we needed to test the system works."

"Let me make a wild guess. One of the beancounters was on some course or something in the past few weeks where they heard some horror story about a company that hadn't been taking proper backups and when something went wrong they couldn't recover ANYTHING!?"

"I... don't know the specifics."

"And part of what they recommended was that you run regular backup checks to make sure that your data is safe by deleting files and seeing if your IT people could recover them?"

"I guess so. But it's really important to know that these systems are working! Why else would we have them? The company RELIES on this data being recoverable. It's a huge business risk."

"Yes, I take your point," I say. "It is a good thing to know and I suppose in retrospect it's good that we know that the systems we have in place are working."

"I couldn't agree more," nods the Boss.

>>10 minutes later<<

"You went a little easy on him didn't you?" the PFY asks.

"Yeah well, you know they're going to delete a bunch of new files shortly just to make sure that we're not recovering stuff from 6 weeks ago..."

"Ah yes, of course.." the PFY says

>>Two days later<<

"I was wondering if you can recover some files for us?" the Boss asks, entering Mission Control with the Financial Director following closely behind.

"Which files?" the PFY asks.

"These ones," the Boss says, putting down a list of folders and files.

"Sure thing."

>>10 minutes later<<

"Yeah, I can't recover those. Probably the nearest I can get is an archive from about six months ago." the PFY says.

"WHAT?"

"Yeah, well after our chat the other day, I realised that if we have all this data from six weeks ago, there's no way to tell if the backup system is working, so I erased all the old backup data. Only problem was the erase took two days to complete and in the process the backup index files were locked so the backup system wouldn't write files."

"WHAT!?"

"Well we had to be sure the system worked," the PFY says. "So we started with a blank slate."

"Did you take copies before you deleted all the backups?"

"Well no. I mean the indexes were locked."

"Why the hell didn't you tell anyone you were doing it - so we'd be careful?"

"But how would we know if the Beancounter system of being careful about deleting files was working?" the PFY asks.

"We'll talk about this some more," the Boss says, "mean time we'll be recovering the data from USB drive - which I hope you'll be backing up tonight?"

"By USB drive you mean the ones that were in the Financial Director's filing cabinet?" I ask

"Ah... yes. Why?"

"Oh, only I broke into his office last night and chucked all those usb sticks into the rubbish bin. I mean the office is alarmed so I just assumed that security would come in and discover them before the cleaner tipped them out?"

"We don't set the alarm at night," the Financial Director blurts. "It keeps tripping out because of the air-conditioning."

"Surely it's important to know that they alarm system is working? Why else would we have one? Surely the company RELIES on the intruder alarms? It's a huge business risk!"

For a while there things are looking grim, but then a ray of hope...

"Actually I took a backup as well," the Boss says smugly, "onto a portable hard drive."

"A portable hard drive in the top drawer of your office desk?" the PFY asks.

"Ah... yes. Why?"

"Only I slipped into your office this morning and replaced your top desk drawer with a metal one with a small incendiary device. It won't set the office on fire but it will burn everything in the drawer."

>JANGLE!< >JANGLE!< "WHY THE HELL WOULD YOU DO THAT!?!" the Boss shouts.

"To test the fire system!" the PFY gasps. "Surely the company relies on the fire systems working? Surely it's a huge business risk??!"

"Why the F*CK didn't you run this past someone?!"

"I did. I had a chat to security about it and they thought it'd be a good test of our evacuation procedures."

"BUT THERE WAS CRUCIAL INFORMATION IN THAT DRAWER!"

"Surely we've got a policy about storing crucial information in desk drawers?" I ask. "It could be a huge business risk..."

Similar topics

Broader topics


Other stories you might like

  • Stolen university credentials up for sale by Russian crooks, FBI warns
    Forget dark-web souks, thousands of these are already being traded on public bazaars

    Russian crooks are selling network credentials and virtual private network access for a "multitude" of US universities and colleges on criminal marketplaces, according to the FBI.

    According to a warning issued on Thursday, these stolen credentials sell for thousands of dollars on both dark web and public internet forums, and could lead to subsequent cyberattacks against individual employees or the schools themselves.

    "The exposure of usernames and passwords can lead to brute force credential stuffing computer network attacks, whereby attackers attempt logins across various internet sites or exploit them for subsequent cyber attacks as criminal actors take advantage of users recycling the same credentials across multiple accounts, internet sites, and services," the Feds' alert [PDF] said.

    Continue reading
  • Big Tech loves talking up privacy – while trying to kill privacy legislation
    Study claims Amazon, Apple, Google, Meta, Microsoft work to derail data rules

    Amazon, Apple, Google, Meta, and Microsoft often support privacy in public statements, but behind the scenes they've been working through some common organizations to weaken or kill privacy legislation in US states.

    That's according to a report this week from news non-profit The Markup, which said the corporations hire lobbyists from the same few groups and law firms to defang or drown state privacy bills.

    The report examined 31 states when state legislatures were considering privacy legislation and identified 445 lobbyists and lobbying firms working on behalf of Amazon, Apple, Google, Meta, and Microsoft, along with industry groups like TechNet and the State Privacy and Security Coalition.

    Continue reading
  • SEC probes Musk for not properly disclosing Twitter stake
    Meanwhile, social network's board rejects resignation of one its directors

    America's financial watchdog is investigating whether Elon Musk adequately disclosed his purchase of Twitter shares last month, just as his bid to take over the social media company hangs in the balance. 

    A letter [PDF] from the SEC addressed to the tech billionaire said he "[did] not appear" to have filed the proper form detailing his 9.2 percent stake in Twitter "required 10 days from the date of acquisition," and asked him to provide more information. Musk's shares made him one of Twitter's largest shareholders. The letter is dated April 4, and was shared this week by the regulator.

    Musk quickly moved to try and buy the whole company outright in a deal initially worth over $44 billion. Musk sold a chunk of his shares in Tesla worth $8.4 billion and bagged another $7.14 billion from investors to help finance the $21 billion he promised to put forward for the deal. The remaining $25.5 billion bill was secured via debt financing by Morgan Stanley, Bank of America, Barclays, and others. But the takeover is not going smoothly.

    Continue reading

Biting the hand that feeds IT © 1998–2022