To err is human. To really tmux things up requires an engineer

The unbearable persistence of data

Who, Me? A reminder to check and check again in today's Who, Me? as a Register reader learns the true meaning of persistence.

The latest confession comes from an exotic South American city and one of the country's many ISPs. Our reader, helpfully Regomised as "Paolo", was an engineer working for the company and, as is the case for so many of us in the IT world, a wearer of multiple hats.

Not to worry though, thanks to tmux (a terminal multiplexer) he had everything at his fingertips. Consoles for different servers, BGP and ERP settings, FTP and so on. Over his 10 or so tmux sessions he had pretty much every acronym nailed down and was god of all he surveyed.

The event in question took place on a Friday evening, which is when everyone does all their best work. The office was closing up – engineers were heading off into the night to do what engineers do with their time off.

In between his tmux juggling, Paolo kept an eye on the clock as the big hand neared the 12 and the little hand settled on the 5.

Finally. Home time.

Switching to the tmux pane of his local machine, Paolo issued the shutdown command. The notebook screen was slammed shut. Stuff was gathered. The weekend was on the way.

Except... except... there had been a LOT of tmux panes open. Had he gone to right one? The answer came all too soon, and all too loudly as the boss's despairing shriek echoed through the building: "WHO SHUT DOWN THE BGP SERVER?"

"It was not my own machine," recalled Paolo, "but the firewall and BGP server."

One facepalm later, and an admission of his mistake, and Paolo was in the network room, powering up the server he'd accidentally turned off.

"Except this wasn't a normal firewall and BGP server," he told us, "it was a custom one, integrated with the ERP. Upon booting, it would parse the ERP database, fetch and create the VLANs, firewall rules, and so on."

Seems reasonable enough. However, Paolo went on: "But a few engineers, to speed up folks working on the street in cabling and new clients, would issue commands direct into the firewall, later they would log into ERP and register the new client... thus, no persistence."

We'll pull the covers over these shoddy practices, and we're sure no Register would ever stoop to such bodgery but, for Paolo, the damage was done.

Rather than a carousing through the local public houses and sampling adult beverages, Paolo's (and his team's) Friday night was instead spent painstakingly working out which client hadn't been saved in the ERP. The task took hours.

"I never issued a poweroff in the wrong tmux pane anymore."

What lesson did you learn the hard way? Or were you at the sharp end of another's educational moment? Tell us with an email to Who, Me? ®