On-Call Here is the latest festive instalment of On-Call, the place to be for techies with tales to tell. It might be cold outside but this may just make the blood boil as it goes to show what happens when bean-counters take over the asylum.
Meet P, who told El Reg about troubles at a brand spanking new data centre that he managed in New Mexico, where no expense was spared "except common sense". Ouch.
The data alarm went off during the middle of a blizzard and the "trusty team" then "hot footed it" to get a handle on what was at the root of the glitch.
"Temperatures were soaring above 100˚F (C. 38˚C). All the Sun servers [were] going tits up. Windows in the adjacent offices got opened (smashed open in one case) to let in freezing air and the data room doors were jammed open and a buddy at Home Depot got up in the middle of the night and delivered some industrial fans."
The morning after, the proverbial hit the spinning blades when the resident company veep wanted to understand why Sun and Solaris servers were powered down and one of the windows broken.
"Remember those cost saving initiatives that were implemented with all the bright bods (consultants) external input? Wellllll..." was the response from Pete.
Some bright spark had decided to position all the humidifiers on the same duct system, so when the temperatures outside plunged, a "wall of ice formed on the air inlets like a fucking cork".
This left the heating, ventilation and air conditioning (HVAC) unit sucking in air, "which in this case was being nicely provided through exhaust air flowing back from the buildings main system which naturally was linked to the heaters to keep the other zone temperatures at 75+[˚F]" (23˚C).
The secondary HVAC units that should have been on standby were "deemed not essential as the [data centre] supplier worked out the tonnage required very carefully, but entered sea level as the centre location and we are at 5,000 feet which means significantly higher tonnage. WTF."
Instead, a "single HVAC system was provided and all cries from the peons about 'how the fuck do we take these offline for maintenance in a 365/7/24 operation?' were deemed rubbish".
The last twist in this sorry tale came when Sun's Gold service team came on site to assess the damage and concluded it was "all self-inflicted therefore warranty on our beautiful $15m system was void".
Lovely stuff. Have you got stories that deserve a public outing but in which you or your colleagues want to remain anon? Get in touch with our very own On-Call desk. ®