Ask a builder to fix a server and out come the vastly inappropriate power tools
Sure, go ahead, make more dust while I deal with a client who doesn't care about redundancy
On Call With Friday upon us, the green-thumbed among you may have some gardening planned for the weekend. Before you dig into that chore, The Register presents another instalment of On Call, our weekly reader contributed tale of being asked to get into the weeds of malfunctioning tech.
This week, meet a reader who asked to be Regomized as "Bob" and told us he used to work for a company that sold kit to radio stations – some in remote locations.
"I wrote the software but it was a small company so I also sometimes had to double as Linux support, as I was the only one who could setup the servers the company sold," he told On Call.
Most of this involved staying in his nice warm office and fixing things remotely. But some of his clients were in Africa, where a studio was being expanded – without the budget for a new server.
Thankfully, Bob's employer usually installed a backup server. So even though it wasn't ideal for the expanded studio to run without a backup, that arrangement was reluctantly chosen.
And then regretted, because the backup server was discovered to have failed – and nobody knew for how long!
Bob was duly despatched, along with a bag of spare parts.
"I got to the customer, and it was literally a building site with new bits being added and the roof being redone," Bob told On Call.
The first server he saw was running but with "a horrible noise from the fan and a lot of clicks from the discs." The backup was utterly dormant, so Bob removed it from the rack, opened it and found "the worst case of dust I've ever seen – two or three inches deep and really clogged."
At this moment, the station engineer who chaperoned Bob "barks an order at an underling who grabbed the server and said 'We'll sort out the dust'."
Bob watched as the station employees spoke to the construction crew through the nearest vacant window hole, and then as a leaf blower was passed through that aperture.
"Before I could shout 'Noooooo!' the station engineer blasted the dust out," Bob told On Call, before pausing the remind us that this story took place on an active building site – so a leaf blower was likely to push more dust or other material into a server!
- Making the problem go away is not the same thing as fixing it
- Workload written by student made millions, ran on unsupported hardware, with zero maintenance
- Police ignored the laws of datacenter climate control
- Techie labelled 'disgusting filth merchant' by disgusting hypocrite
Then came danger from above.
"Someone on the roof had finished with a large bucket filled with murky liquid and asked themselves 'Why carry it down to the ground, when you can just empty it over the side?'"
The resulting deluge just missed the server, sparking much shouting from the station engineer about the need to take more care of delicate electronics.
Bob was eventually able to bring the server into a clean and safe space, peered inside, and was "amazed that all the components were still on the board" – such was the force of the leaf blower's output.
He was even more surprised when he installed a new power supply and the server booted up as if nothing had happened.
Next came sorting out the other server, before installing software, racking and stacking, and then heading home.
"I left a delighted station owner who was oblivious to how close he came to a big bill and delayed opening," Bob recalled.
What's the least appropriate tool you've seen used to maintain hardware? Click here to send On Call your story and we may harvest your story here on a future Friday. ®