BOFH: There's a fatal error in the blinkenlights

Who needs actual server hardware when you can fire up a Linux laptop with the Matrix screensaver?

Episode 7 The Boss wants a Big favor. He wants us to go to another company and help their geek out with a server issue.

Digging deeper we find that it's a personal favor, not work-related in any way.

"Five Pints," the PFY says.

"Each," I add.

The Boss makes a show of pretending to think this over carefully, then reluctantly agrees. The company concerned is an investment outfit that the Boss has his retirement money with. Whilst the returns are good, their technical problems have rapidly eroded the Boss' confidence to the point that he wants us to get them running long enough for him to get his cash out. Today.

"Oh, right!" I say. "Totally understand. Ten Pints."

"We agreed Five!" the Boss quibbles.

"You know, I think we have that six-week long I.T. inventory to do," the PFY suggests. "It's quite urgent too. Perhaps we should get onto that instead?"

...

A deal is struck, and the PFY and I grab a cab to the company. Their resident geek, told to expect us, is waiting at the door.

"It's the main server we're having problems with," he says, as we ride up in the lift with several power-dressed investment types.

"Just through here," he says, leading us down a maze of twisting corridors, all alike.

The server room is small but amazing. Four racks of gleaming equipment humming away, connected in cable via streamlined cable management.

...

"So this is all a front?" I ask, after a minute or so of looking around.

"What?!" he asks.

"The racks, the servers - it's all a big fake?"

"I don't know what you're talking about," he says, confused.

"Please. I mean look at that cabling. It looks like it was put in by an OCD architect."

"And?"

"And that never happens. I mean it might look like that on Day One, but after a couple of years it looks like an explosion at a spaghetti factory. And look at your hardware."

"What about our hardware?" he gasps.

"None of them have their covers half off. Besides, you've got four G4 DL320s in there. Sure, the front bezels have been polished, but they're over 15 years old."

"There's a proliant 3000 up there," the PFY chips in, pointing at another old piece of kit with a red LED glowing angrily.

"Oooh look!" I say. "An aftermarket LED 'upgrade'."

...

A minute of silence passes, then finally the geek cracks. There's no server hardware. Nothing. Over the last five years the entire company operation has moved into online services - theoretically leaving our geek with no job.

"So what do you... do all day?" the PFY asks.

"SOME days, I'll take a complete snapshot of our cloud infrastructure," he says.

"Once a month you mean?" the PFY surmises. "So what do you do with the rest of your time?"

"I, um, manufacture outages," he admits.

"Manufacture outages?"

"Yeah, I'll light up the RED lamp on a server and, uh, take a cloud service offline."

"Why?"

"Because then they'll call me and get me to fix it. I'll bring them in here, fire up a linux laptop with the Matrix screensaver, edit a JPEG with a Hex editor, pretend to find a virus signature or an internal consistency error, then 'fix' it and bring the service back online again."

It seems so simple now that he says it.

"And no one's rumbled you?"

"They're too busy trading. None of them knows where the data lives, what services they're subscribed to, or what the underlying infrastructure is. Two weeks ago I told them the enterprise license for the English Language keyboard was about to expire and they'd either have to pay the £2,000 licence fee or they'd automtically be switched to the open source Turkish keyboard."

"And they okayed two grand?"

"They okayed FIVE grand. For a three-year license."

"So... this is all you do all day is it?"

"Most days. Other days Carl or Peter does it."

"Carl or Peter?"

"Yeah, we work shifts - because the market never sleeps."

"So let me get this straight. You don't have any servers, you don't have any real work - AND THERE ARE THREE OF YOU - so you just make problems to keep yourself in a job?"

"Yep, That's pretty much it." 

...

"I feel like I've wasted my life," I admit sadly, on the way back to the office after the "server" is back online.

"Me too," the PFY says.

Half an hour later I'm back in Mission Control when the Boss enters nervously.

"How did it go?" he asks. 

"Well it looks like they're up and running, but to be honest their hardware has some real issues, so if you were going to be getting your money out, I'd be doing it today."

"Really?" the Boss gasps.

"Really," I say. "The system management is appalling and I don't think their IT guy has a clue what's going on."

The Boss almost knocks the PFY down on his way out the door, but he's back again inside a minute.

"There's something wrong with my computer!" he says, waving his laptop around madly. "I need you to look at it now."

"What seems to be the problem?" I ask, taking the laptop from him. "Let's see, all looks fine to me... oh..."

The PFY looks over, and shakes his head.

"Has someone forgotten to pay their enterprise English language keyboard licence?" he asks. ®

• BOFH: Previous episodes on The Register
• The Compleat BOFH Archives 95-99