On-Call Welcome again to On-Call, our regular tale of things that happen when readers are called in to fix big messes on weekends and evenings.
Before we get to this week's tale, a quick reminder we've some prizes for new submissions as part of our Sysadmin Day celebrations. Write to me if you've a story of being called out to do something interesting or weird.
A reader named Joe did so a couple of months back, but we'll toss his name in the prize draw hat anyway because he brings us a tale of a hot, sunny Sunday afternoon. Or seeing as Joe was in England at the time, perhaps the hot, sunny Sunday afternoon of the summer.
Joe's workplace featured what he describes as "a lab on the top floor in direct sunlight, and the heat exchangers in direct sunlight".
So when on that sunny afternoon his phone rang and his manager was on the other end, he wasn't surprised to learn that that the lab “was warming up and the sensor alarms have triggered.”
Joe's boss had taken a call from facilities and learned that the air conditioners' heat exchanger vents were also in direct sunlight. In a colossal surprise, one "had a hissy fit and shut down to protect itself".
Joe was asked to remote in "and start shutting down machines before we lose the second AC unit".
"I'm fairly new to the team and running the lab wasn't really a part of my role," Joe recalls, "but no problem, it will be a bit slow, but lots of the machines are fairly hefty multi U types, so each one shutdown will be significant."
Of course this didn't work: a newly-installed anti-virus product wouldn't let anyone do remote access unless their machine ran the same AV software. And Joe's machine was locked down by IT, so he couldn't just install the software himself and get on with things.
“Meanwhile,” he recounts, “someone in facilities has armed themselves with a fire extinguisher and was hosing down the vanes of the air con exchangers to try and cool them down, but to no avail. The remaining air con unit decided shutting down would be the best option."
Joe kept trying, and finally managed to log on to one of his employer's machine in Australia but was thwarted once again.
“Sadly it has no routing through to the UK lab, which is on a VLAN with limited access. The latency is pretty horrible and I concede defeat.”
As did the facilities team: the ambient air temp in the lab hit 50C, so the team simply switched everything off.
“Over the next few weeks hard disk failures are a daily experience and some servers go to the great bit bucket in the sky,” Joe recalls. “An enterprising chap in facilities bought a gazebo from a local DIY store to shield the air con heat exchangers from direct sunlight, hundreds of thousands of pounds of hardware being protected by a £50 plastic tent.”
“It does help a little, while some of the more obvious bloopers like moving the forced air floor vents so they are at the front of all of the racks are fixed. Eventually the windows are tinted and the layout reorganised.”
“I know British Summers aren't renowned for being summery,” Joe says, “but the building in question was a new one, commissioned by the company, and nobody involved in the design seemed to have thought about the remote possibility of 'yeah, but what if the sun does shine? For more than a couple of hours?'”
What have you been asked to do when called out at night or on weekends? Share your stories by writing to me. And don't forget: we've got prizes! ®