Watch a RAID rebuild or go to a Christmas party? Tough choice

That time Exchange said 'it's a fair cop, guv' before the disks tumbled down the stairs

Who, Me? The long arm of the law is unexpectedly severed by the antics of Microsoft Exchange this week as another reader explains why some of Her Majesty's finest were once bereft of festive email. Welcome to Who, Me?

Today's story, from a reader Regomized as "Sam", takes us back to the mid 2000s and happier days before Brexit, Trump and the pandemic showed up.

During these halcyon times, Sam was working as a civilian employee for a local UK Police force as a member of the desktop support team. He'd been there a few months when the festive season rolled around.

Christmas parties were a thing for Sam and the team, and each group had its own festivities during the day. Other groups would cover for them while crackers were pulled and silly hats worn. Each party normally ran from lunchtime until well into the evening.

On the day in question it was the server team's turn to head off while Sam and the desktop crew ensured everything was covered in their absence. Naturally, there was a bit of mistrust between administrators and mortals, and so Sam and co naturally took to sellotaping festive wishes over the lasers on the undersides of the admins' mice as well as other japes.

It was all fun and games until 3pm, when two of the server team (who'd left behind their mobile numbers "just in case") burst back into the room.

Something had gone terribly wrong.

"A little background..." explained Sam, "We had 2 Exchange Servers. Your placement was dependent upon your staff number; odd on one, even on the other."

One of the Exchange Servers had abruptly died. "Consequently, half of the constabulary had lost email!"

"Being a public service, we had top-notch Microsoft support at a better price than many so made copious use, long into the wee small hours..."

Eventually it transpired that a disk in one of the servers had failed the day previously. Sabres in the form of support contracts were rattled and a replacement had arrived that morning. The server was ok - after all, it could handle a failed a disk. The replacement was popped in and a rebuild was started.

And then the desktop team set off for their festive frivolities. The server would sort itself out, as designed.

What could possibly go wrong?

As it turned out, quite a bit.

"While in the process of rebuilding the array, a second disk decided it was going to join the party and while not quite failing, caused enough disturbance in the force, that Windows and Exchange violently soiled themselves and gave up, crying in the corner," said Sam.

To be fair to the Server team, they had warned that the Exchange setup was living on borrowed time, was way overloaded (compared to Microsoft's recommendation) and, according to Sam, the phrase "It's a disaster waiting to happen!" had been thrown at management on more than one occasion.

And then, in time-honored IT fashion, it did.

"Nothing the many iterations of increasingly technical Microsoft guys threw at it helped one bit," said Sam. "It wouldn't repair and it wouldn't restore from backup.

"It had to be rebuilt and restored by-datastore in order to get it all back and working."

The long and tedious job (which required user data being restored in batches) took a little under two weeks to complete, making for a miserable Christmas and New Year for all concerned.

And the person at the end of the queue? The biggest of Police cheeses: the Chief Constable. His calendar had been used as audit trail of his activities, meaning that most of a datastore was all about him, and getting it back presented the most problems.

The curse of being a Very Important Person.

While putting Bobbies on the Beat might garner favourable headlines, neglecting basic IT principles can result in all manner of catastrophe. Have you tried out your disaster recovery plan recently? Or hasn't your company considered the consequences of when that innocuous beige box inevitably turns brown? Share your tale with an email to Who, Me? ®