Security? Working servers? Who needs those when you can have a shiny floor?
The root cause was a buffer error – but not the kind of buffer you're thinking of
Who, Me? Ah, gentle reader, once again it is Monday and all that entails. But fear not, for The Reg is here with Who, Me? and another tale of things going not quite so well as might have been hoped. Perhaps this will lift your day.
This week's raconteur, who we'll Regomize as "Murph," brings us a tale tying together two of our recent themes: security tech that doesn't behave as it should, and power points being used in ways they were not meant to be.
Murph works in an older building that's been renovated, with "marble entry and tasteful carpeting in the work areas." The server room inhabits a repurposed gymnasium (no servers under the break room bench here) and there are tiled floors everywhere. In fact, "with lots of tile across seven floors, we have a decent size maintenance staff to keep it looking good."
Now as you might imagine in a facility like that, different staff have access to different areas. Security fobs ensure that people go only where they are meant to. All well and good.
Until one day, when a new member joined the maintenance staff. Decent bloke. Conscientious. Hard worker. Well liked.
For some reason – we will likely never know the reason – when this new affable cleaner signed on, he was not given the standard security fob that maintenance staff were issued. He was given an old fob that had formerly belonged to a retired member of the IT team. Obviously this is a problem for a bunch of reasons. But that's not where our story ends.
The new guy was asked to clean the floors in the IT department. All well and good – and with his super-powered fob he could clean even more of it than anyone expected.
Take, for example, that former gymnasium, with its expanse of dingy tile that hadn't had a good polish in who knows how long.
- Data cleanser did its job, but – oopsie! – also doubled customers' bills
- A toast to being in the right place at the right time
- Fed up with slammed servers, IT replaced iTunes backups with a cow of a file
- Seriously, boss? You want that stupid password? OK, you get that stupid password
That grime was just fine, of course, because it's well known that mops and datacenters don't mix.
Conscientiously – diligently, even – he plugged his industrial floor polisher in to a power board labelled "A". As it turned out, this connected to the "A" rack of uninterruptible power supplies.
"Nine minutes and twenty-three seconds later (confirmed by the logs and surveillance system)," Murph tells us, "alarm emails and cell phone vibrating sensor alerts were sent out" as the UPS batteries were depleted by the unexpected load.
Murph got his alert and dashed to the server room, arriving just in time to stop the highly motivated cleaner from diligently – conscientiously, even – plugging in the polisher (which had lost power when the UPS died) to the rack labelled "B".
It's a story in which not one thing but several had to go wrong – but it has a happy ending. You see, that "B" rack of UPSes kept everything going. In spite of the alarms and alerts and whooping sirens (OK, there weren't whooping sirens) the system stayed up and worked as it should.
Between a well-liked cleaner and an anonymous screw-up in HR, the system was given one heck of a stress test, and passed. Murph just hopes it doesn't happen again.
Sadly we're unsure if the server room floor ever got that much-needed polish.
Have you ever given a system a necessary if unintentional stress test? Tell us all about your exploits in an email to Who, Me? and we'll tell the world.