After three weeks of night shifts, very tired techie broke the UK’s phone network

And got away with it when someone else broke it even more comprehensively

Who, Me? Welcome to another working week, and therefore to another instalment of Who, Me? It’s The Register’s reader-contributed Monday column that shares stories of your worst moments at work, and how you kept your career alive once the extent of the damage was discerned.

This week, meet a reader we’ll Regomize as “Wayne” who in the 1980s maintained the specialized “System X” hardware that ran telephone exchanges.

Bad things happen when phone exchanges go down, so the System X machines Wayne maintained were installed in pairs named Processor 1 and Processor 0. Processor 1 was the active device and Processor 0 was the hot standby.

One evening, Wayne and his colleagues were asked to upgrade a major phone exchange that handled all the traffic for the North West of England and connected that region to the south of the country.

Upgrading the machines required them to be powered down so new hardware could be installed. System X’s designers deliberately made that hard to do, to prevent accidental shutdowns.

To perform an upgrade, Wayne therefore needed to type commands into a terminal to put components into a state of “soft shutdown” which was indicated by a green light turning red.

Once the lights on all components in a Processor turned red, it was safe to turn it off and replace hardware.

At the time of this job, Wayne had been doing similar upgrades for about three weeks and they all required pulling an all-nighter. Wayne’s routine saw him roll into work at 21:00 to start preparation, before the serious part of the upgrade started at around 02:00 the next morning.

After three weeks of overnight upgrades, Wayne was a mess.

He was also disconcerted, because for reasons he never understood Processor 1 was always installed to the left of Processor 0. Wayne felt that Processor 0 should be on the left, and Processor 1 on the right to match the convention of left-to-right Western numbering and writing.

You can guess the rest: After performing a soft shutdown of Processor 1, Wayne’s discomfort with left-to-right numbering collided with his extreme fatigue, and his brain decided the green lights on Processor 0 were an invitation to remove its components.

Processor 1 was now in a state of soft shutdown and Processor 0 was missing the hardware that made it work.

“All trunk telephony traffic between the North and the South of the country went down,” Wayne admitted to Who, Me?

Wayne still isn’t sure why alarms didn’t start going off.

And because they didn’t, he kept doing the tests he always did at this point of the upgrade. All his previous jobs had gone fine. This time his terminal showed all sorts of anomalies.

Eventually, his tired brain recognized the errors he was seeing and remembered the last time he’d seen them: At a training course during which he’d practiced shutting down an entire telephone exchange.

Panic lanced Wayne’s brain.

“I rushed back to the processor and started slamming cards back in,” he told Who, Me? “The lights stayed red a frustratingly long time but then slowly started to turn green.”

He raced back to the terminal. “My sweaty fingers slipped over the keyboard as I typed the recovery commands as fast as I could, desperately willing the status to improve.”

Over the next few minutes, more lights turned from red to green and he was confident phone calls were again making it across the UK.

As if to prove it, Wayne’s phone rang. He picked it up heard colleagues at the network monitoring center ask what had just happened.

“I could have lied and said I didn't know but I couldn't think of anything clever, so I admitted my mistake,” Wayne told Who, Me?

“We'll have to report this,” he was told.

And then, in the wee small hours, with weeks of fatigue in his bones, brain and blood, Wayne drove home and flopped into bed.

With his head full of existential dread about what this incident meant for his career, sleep would not come. “I just lay in bed, staring at the ceiling, waiting for morning and the inevitable phone call from my boss,” he told Who, Me?

Later that day …

Wayne eventually roused himself and the phone rang.

The call went like this:

Boss: "How are you? I believe you had a long night."

Wayne: “I’m sorry, I was tired. I know it's no excuse."

Boss: "What are you talking about? I thought you were upgrading a trunk exchange last night?”

Wayne: "Yes, but ..."

Boss: "Some contractors dug through a load of fibre by accident next and completely killed transmission."

“And that was that” Wayne told Who, Me? The network monitoring center intended to report his mistake, but the cable cut caused a much bigger outage that had the same effect as the one Wayne had created. His mistake was never mentioned again and had zero consequences.

Has somebody else’s mistake hidden your mistake? Don’t make the mistake of not sharing your story: Click here to send us an email so we can tell your story in a future edition of Who, Me? ®