Beware the big bang in the network room

Trust me, we're going to need a bigger maintenance window


Who, Me? Cables can be unkind, especially when one has the confidence of youth but not the dark cynicism of experience. Welcome to an edition of Who, Me? to tug at, if not the heart strings, then certainly the RJ45s.

Today's hero is a Register reader Regomised as "Colin." Colin had accepted his first proper role as a network engineer for a small UK consultancy. Up to that point, his networking experience had consisted of swapping out the hub of a previous employer for a switch and getting the company's self-hosted web site and email server up and running.

He was confident. Gosh, he was confident.

Our story takes place in the early part of the last decade. The client had a London data centre and he was tasked with swapping out some of the switches within for modern units, replete with 10 gigabit uplinks.

"Regarding myself as 'not a total cowboy'," he told us, "I had surveyed the racks in advance and recoiled at the sprawling mass of Cat 5 cables that had 'grown organically' around this 48-port switch and confirmed that yes, of course there wasn't an empty slot in the rack..."

"I was quite new to the whole networking lark," he went on, "and had the bullishness to believe that with no assistance, and in spite of the Gordian-knot-like cable nest, since this was a Layer 2 Switch with identical VLAN config to its replacement, I should be able to replace it along with a sister switch in another rack within the same maintenance window."

"Mostly because I didn't want to run the hassle of agreeing multiple windows with the client (and their clients)."

Colin was careful. He labelled all the cables. Undid all the bolts. And ever so gently, he began sliding the old switch out of the rack.

"I felt some resistance and pulled gently since the RJ45 jacks on Cat 5 cables can take some strain without damage, and all the cables were out anyway..."

Out of the switch, perhaps. Alas, by the time Colin realised what was causing the resistance, the second power lead had popped out of the server that happened to be running BGP on an ancient version of Linux for the customer's entire network. And, of course, there was no backup server waiting to helpfully step in.

Maintenance windows can be used to cover many sins, and Colin was burning through his at a terrifying rate. Yes, the Penguin Gods smiled and the Linux box restarted without issue, but there was work to do and time was marching on.

Tired, Colin came to the final switch. It was at the top of a full height rack. Of course it was. The Data Centre helpdesk team had not shared their step ladder due to "health and safety", necessitating some precarious balancing but at least the cables were a bit more accessible. They needed to be - Colin hadn't been given the key to get into the rear of the rack.

"I swapped the switch and attached the power, fibre uplink, and customer Cat 5 cables one-by-one," said Colin, "Since I had been meticulously checking for traffic on each customer port as I connected them, I noticed that one cable was missing."

He hunted and hunted. But cable there was none. "I think it was somewhere around the side but falling towards the back under its own weight," he told us.

Midnight came and went and Colin was exhausted. There was a real chance that he might make things oh-so-much worse with tired hands. The maintenance window was also closing, "so I figured that the customer would understand."

He messaged a senior colleague to flag up the port missing its cable. Someone in the morning would have to deal with it. It was, after all, just one cable. How bad could it be?

"My colleague impressed upon me the following day that the only missing cable belonged to the client's biggest customer, and he did not see my message until after a while had gone by since the customer's business hours began and they were fully affected by the outage."

Colin was allowed to forget his mistake. It was used, he said, "as a prime example of why you should never go 'big bang' with a swap-out like this, not to work onsite alone, and to always declare a much longer window than you think you need."

Lessons learned after event tend to be the toughest ones. Ever found yourself fighting through a forest of cables and finding a customer less than understanding and forgiving of an unforced error? Confess all with an email to Who, Me? ®

Similar topics

Broader topics


Other stories you might like

  • An international incident or just some finger trouble at the console?
    All routers are equal, but some are more equal than others

    Who, Me? Welcome to an edition of Who, Me? where some configuration confusion left an entire nation cast adrift.

    Today's story is set in the early 2000s and comes from a reader Regomized as "Mikael" who was gainfully employed at a European ISP. The company had customers in multiple countries and Mikael's team was responsible for the international backbone.

    "Us senior network engineers were widely regarded as consummate professionals," he told us, before adding, "at least amongst ourselves."

    Continue reading
  • A discounting disaster averted at the expense of one's own employment
    I know what this process needs: Microsoft Access!

    Who, Me? A tale of discounts and process improvement via the magic of Excel, Access and a fair bit of electronic duct tape we imagine. Welcome to Who, Me?

    "James" is the Regomized reader of record today, and continues the theme of running the risk of doing a job just that little bit too well with an ancedote from the end of the last century involving his first job out of university, at a certain telecommunications giant.

    The job involved a process of calculating the discount received by big customers (the ones with multiple branches). "For the life of me I can't remember what the main DB was called," he told us, "but it was the old style green writing on a black screen that took forever to download the necessary data."

    Continue reading
  • In IT, no good deed ever goes unpunished
    When being helpful can mean being shown the door

    Who, Me? Going above and beyond in IT can sometimes lead to also going directly out of the door, as one Register reader found when discovering that sometimes efficiencies can be less than rewarding.

    A reader Regomised as "Will" told of us his days working at a now-defunct company that produced large telephone switches. In those days whenever a major software revision occurred, customers were expected to send in their configurations and Will's group would merge them into the latest and greatest. A new load would then be returned to the customers.

    It was not a fun process, not least because of constant hardware and software failures during the merge process. "When I first started, there was a constant grumble about how unreliable the machine used for the merging was," Will told us.

    Continue reading
  • An early crack at network management with an unfortunate logfile
    It's a backronym, right?

    Who, Me? Come with us on a journey back to the glory days of Visual Basic 6, misplaced enthusiasm and an unfortunate naming incident. Welcome to Who, Me?

    Today's tale comes from a reader Regomised as "Stephen", who was working in the IT department of a Royal Air Force base. "My duties were many," he told us, "from running daily backups of an ancient engineering system using (I kid you not) reel-to-reel tapes to swapping out misbehaving printers."

    This being the early 2000s, his boss loaded up our hero with more tasks. He could change printers and tapes, so Visual Basic (and its bedfellow, Access) should present no problem.

    Continue reading

Biting the hand that feeds IT © 1998–2022