BOFH: Everyone deserves a little DOWNTIME

Titsup tech holiday special


Episode 14

Isn't it always the way that when you're hours from the office sporting only a smartphone with a dodgy data connection that something critical claps out back at work.

And then something's wrong with the VPN so you have to gain access through a convoluted chain of remote desktop, ssh and telnet links just to end up with a PowerShell interface which should NOT be accessible to you - or anyone. All because you need to bypass the security you not-so-carefully crafted – to perform a reboot.

And just when you're about to hit the final <enter>, some *GIT* rings you on the phone, which sends the command session you just had going into limbo.

"Yes?" I seethe.

"Just wondering if you know that the financials server is down?" the Boss says.

"It's not down, it just looks like it's down because the DNS component of the domain controllers has some syncing issue which is causing it to return deprecated DNS data." "I... what now?"

"Something's wrong in the main brain of the servers.” I explain carefully. "I need to look at it to see why it's wrong, but I can't do it while I'm being phoned."

I hang up and continue.

>tapitty< >prod< >RRRIING!<

"I told you - I can't do it while I'm being rung!!!"

"Yes I know," the Boss blurts quickly. "I was just wondering if you could tell me when things would be back as there's people wanting to work."

"I was going to say I don't know," I respond, "but now it's I don't know PLUS the time of this phone call, PLUS the time to navigate back to the remote desktop app."

"So you don't really have any idea?"

"I don't know exactly what's wrong yet, so no."

"Should I leave you my number?"

"The one that's the last two entries on my recent calls list?"

"Yes. I see. Well I'll just wait for your call. Any idea when tha.. >click<"

>tappity< >prod< >prod< >swipe< >RIIIINNNNG<

"Yes?"

"Hi, I know you've talked to Alan but I was just wondering if you could give us a ROUGH idea of when things will be working again. ROUGH - doesn't have to be too accurate," the IT Director says.

"Oh well, in that case, this week some time."

"Ha ha, yes. Well played. Maybe a little more accurate than that?"

"I could narrow it down to the days in this week which have not already passed..."

"Well, will it be up today?" he asks.

"Before people started calling me, I would have said possibly - but now I'm not so sure."

"Well can you give me likelihoods? Like a percentage guess on how likely it would be to be up and running this afternoon."

"Well off the top of my head I'd say it was something like a 78 per cent chance."

"So 78 per cent?"

"No, LIKE 78 per cent. So it could be 7 or 8 or 78 per cent. It's hard to tell with all these calls."

"I see, well I'll just let you get on with it. If you could just give me a ca.."

>click<

>tappity< >tappity< >tweak< >tweak< >click< >prod< >prod<

Moments later I give the PFY a call.

"Where are you?" I ask.

"At the pub," the PFY says. "It's lunchtime."

"OK could you pop over the road and kick the domain controller in the guts, AFTER you pull the plugs on the backup DC? I think the backup is poisoning the primary."

"Not really," the PFY says.

"Why not?"

"I'm in Paris."

"YOU SAID YOU WERE IN THE PUB!"

"I am - I'm picking up some USB sticks."

"WHY FROM BLOODY PARIS?!!"

"Well you know how the Boss said he HAD TO HAVE a particular brand and a particular size of stick, AND he didn't care where I got them from so long as I got a good price? Well, I worked out that with the Eurostar special fare it was actually CHEAPER if I was buying more than 47 of them. So I'm sitting at a bar at Gare du Nord with a carton of 50. AND I've saved the company somewhere in the vicinity of 18 quid."

>click<

>tappity< >tappity< >click< >click<

Now I'm calling on the powers of memory that I don't have to remember to try and guess the address of the lights out management card. Worse still - I never really configured any of them outside of an address so they pretty much all look the same once you login.

>tappity< >clack!<

A quick ping tells me that the BDC is still there, so I've turned off something else in that server rack. I'm sure the monitoring system will page me in due course.

Unless I just turned off the monitoring system.

>tappity< >tappity< >click< >click<

>tappity< >clack<

Nope, another dud.

>tappity< >tappity< >click< >click<

>tappity< >click< SUCCESS! The BDC on port 9 has stopped responding to pings! I send a reboot to the PDC - which I know for a fact has an IP address one less than the BDC and pop into a pub to get myself a lunchtime lager.

>RING<

"Did you fix it?" the Director asks. "Only it looks like some people are getting back in."

"Yes. It should be right in half an hour or so. Now that the backup domain controller is out of the picture people will start getting information from the right one."

"So this... domain controller.. Should we replace it?"

"Undoubtedly."

"How soon can you do it so this doesn't happen again?"

"Well if I go through our normal suppliers, maybe a week or two."

"We have to do it sooner than that. We can't have this happen again! Are there no other options?"

"Well I could go and get one direct, which would take a day or two..." I say, directing Gina's gaze to the Amsterdam specials in a travel agents window...

I really should have written those IP addresses down.

Similar topics

Broader topics


Other stories you might like

  • Running Windows 10? Microsoft is preparing to fire up the update engines

    Winter Windows Is Coming

    It's coming. Microsoft is preparing to start shoveling the latest version of Windows 10 down the throats of refuseniks still clinging to older incarnations.

    The Windows Update team gave the heads-up through its Twitter orifice last week. Windows 10 2004 was already on its last gasp, have had support terminated in December. 20H2, on the other hand, should be good to go until May this year.

    Continue reading
  • Throw away your Ethernet cables* because MediaTek says Wi-Fi 7 will replace them

    *Don't do this

    MediaTek claims to have given the world's first live demo of Wi-Fi 7, and said that the upcoming wireless technology will be able to challenge wired Ethernet for high-bandwidth applications, once available.

    The fabless Taiwanese chip firm said it is currently showcasing two Wi-Fi 7 demos to key customers and industry collaborators, in order to demonstrate the technology's super-fast speeds and low latency transmission.

    Based on the IEEE 802.11be standard, the draft version of which was published last year, Wi-Fi 7 is expected to provide speeds several times faster than Wi-Fi 6 kit, offering connections of at least 30Gbps and possibly up to 40Gbps.

    Continue reading
  • Windows box won't boot? SystemRescue 9 may help

    An ISO image you can burn or drop onto a USB key

    The latest version of an old friend of the jobbing support bod has delivered a new kernel to help with fixing Microsoft's finest.

    It used to be called the System Rescue CD, but who uses CDs any more? Enter SystemRescue, an ISO image that you can burn, or just drop onto your Ventoy USB key, and which may help you to fix a borked Windows box. Or a borked Linux box, come to that.

    SystemRescue 9 includes Linux kernel 5.15 and a minimal Xfce 4.16 desktop (which isn't loaded by default). There is a modest selection of GUI tools: Firefox, VNC and RDP clients and servers, and various connectivity tools – SSH, FTP, IRC. There's also some security-related stuff such as Yubikey setup, KeePass, token management, and so on. The main course is a bunch of the usual Linux tools for partitioning, formatting, copying, and imaging disks. You can check SMART status, mount LVM volumes, rsync files, and other handy stuff.

    Continue reading

Biting the hand that feeds IT © 1998–2022