No, you cannot safely run a network operations center from a corridor

Rack full of kit predictably became a network disruption center

On Call With a whole month of 2023 already consigned to history, The Register brings you another instalment of On-Call, our weekly column in which readers share their stories of past deeds performed in the service of keeping computers copacetic.

This week, meet "Nick" who once ran the Japanese branch of a storage-as-a-service provider back in the early 2000s.

"We installed and managed disk arrays and storage area networks at customer sites and in datacenters," Nick explained. In these pre-cloud days, the as-a-service part involved 24x7 monitoring from a network operations center (NOC) in the US.

"We also had a Japanese partner who wanted to run a NOC in Japan, so we helped them get one set up," Nick recalled.

As you'd expect, that facility needed a rack full of kit – which Nick's firm provided. And as Nick was the chap in charge, he was asked to inspect it.

"When I got to their NOC to see how things were progressing, I found this rack sitting in a hallway, with a network, hard-wired frame relay line, and power connection on the wall."

Nick pointed out that the rack really needed to be in a secure area – but that advice was dismissed. He was assured that the Japanese partner was confident that if they told their reliable and honest staff to leave the rack alone, no harm would befall it. Because of course.

Early in the morning a couple of months later, Nick was called into action – the NOC had disconnected from the network.

"After lots of running around and troubleshooting, we discovered that the frame relay line coming from the wall had been ripped out, then just stuffed back into the junction box."

None of the reliable and honest staff who had been told to be careful near the rack and leave it alone admitted to the deed. Because of course.

"They did move the rack to a secure location, though," Nick recalled.

So at least he eventually got that message across.

Has your advice been ignored, leading to utterly predictable problems? If so, click here to send us an email and we'll try to share your story in the future. ®