This article is more than 1 year old
Data centre dangers: Killing a tree and exploding a UPS
Five funny adventures from the small hours
On-call It's the weekend, so it must be time for another instalment of “On-call”, in which we share readers' tales of the odd goings-on that take place when they're required to tackle a job outside of normal hours.
This time around, readers Graeme shares five incidents in just one data centre.
The first started when someone on the other end of the phone let it be known that “THERE'S WATER COMING THROUGH THE DATA CENTRE ROOF!"
Graeme says the problem was in an East Midlands data centre housing several thousand machines, lots of which were co-located. The culprit was a radiator pipe in the offices above, where a central heating system had decided to pop a joint.
The water pooled in a light fitting in the data centre, where it did no damage but caused a lot of panic. Graeme says the main casualty was “a night shift support bloke with a very cold thumb having located the leak and plugged it before we got there!”
Incident two again started in the wee small hours and again featured an anguished caller, this time complaining that "I'M IN DC1 AND IT'S DARK AND IT SMELLS FUNNY".
This time the responsible entity was a UPS. Which had exploded. “Unfortunately it did so in a way that meant we couldn't bypass it easily, so we had to wait for the company sparky to turn up to do the honours.”
On another occasion, Graeme tells us he was “sitting in a pub after hours with my boss who was visiting for the day when he got a call about another data centre (up North) which had a power outage".
“No biggie, huge UPS and generator in situ ... except the landlord of the building had in some way decided to partially block the inlet and outlet vents of the generator, so it was feeding on its own exhaust and getting a bit upset."
"And very hot. And very smoky, so much so that the fire brigade demanded it be shutdown.”
In yet another data centre, which Graeme says experienced the occasional outage, the UPS was tested and worked properly.
“But on one occasion just at the end of the working day there was an outage and the (singular) static switch designed to throw load from mains to generator went bang. Cue a lot of running around shutting as much down as gracefully as possible before the UPS emptied, which wasn't very long!”
That data centre's intermittent problems were eventually traced to a mains incomer that was rather close to its design limit. “Some time later a new feed was pulled in (and new generator installed) but not before the nice pretty trees and grass outside went rather wilty, then brown, then died,” Graeme tells us.
Why the die-off? Well, "because the water round their roots was being boiled by the overstressed three phase mains".
“And then there was the lovely round lug cut out of a section of mains trunking and left behind in a data centre in Docklands, which subsequently decided to fall out of said trunk and cross two phases of a three phase distribution board.”
Graeme would tell us more, but this one happened during normal working hours.
Have you had interesting nocturnal adventures? Were said adventures in the course of your employment? If you can answer “Yes” to both questions, let us know and you could become a future On-Call story. ®