To err is human. To really tmux things up requires an engineer

The unbearable persistence of data


Who, Me? A reminder to check and check again in today's Who, Me? as a Register reader learns the true meaning of persistence.

The latest confession comes from an exotic South American city and one of the country's many ISPs. Our reader, helpfully Regomised as "Paolo", was an engineer working for the company and, as is the case for so many of us in the IT world, a wearer of multiple hats.

Not to worry though, thanks to tmux (a terminal multiplexer) he had everything at his fingertips. Consoles for different servers, BGP and ERP settings, FTP and so on. Over his 10 or so tmux sessions he had pretty much every acronym nailed down and was god of all he surveyed.

The event in question took place on a Friday evening, which is when everyone does all their best work. The office was closing up – engineers were heading off into the night to do what engineers do with their time off.

In between his tmux juggling, Paolo kept an eye on the clock as the big hand neared the 12 and the little hand settled on the 5.

Finally. Home time.

Switching to the tmux pane of his local machine, Paolo issued the shutdown command. The notebook screen was slammed shut. Stuff was gathered. The weekend was on the way.

Except... except... there had been a LOT of tmux panes open. Had he gone to right one? The answer came all too soon, and all too loudly as the boss's despairing shriek echoed through the building: "WHO SHUT DOWN THE BGP SERVER?"

"It was not my own machine," recalled Paolo, "but the firewall and BGP server."

One facepalm later, and an admission of his mistake, and Paolo was in the network room, powering up the server he'd accidentally turned off.

"Except this wasn't a normal firewall and BGP server," he told us, "it was a custom one, integrated with the ERP. Upon booting, it would parse the ERP database, fetch and create the VLANs, firewall rules, and so on."

Seems reasonable enough. However, Paolo went on: "But a few engineers, to speed up folks working on the street in cabling and new clients, would issue commands direct into the firewall, later they would log into ERP and register the new client... thus, no persistence."

We'll pull the covers over these shoddy practices, and we're sure no Register would ever stoop to such bodgery but, for Paolo, the damage was done.

Rather than a carousing through the local public houses and sampling adult beverages, Paolo's (and his team's) Friday night was instead spent painstakingly working out which client hadn't been saved in the ERP. The task took hours.

"I never issued a poweroff in the wrong tmux pane anymore."

What lesson did you learn the hard way? Or were you at the sharp end of another's educational moment? Tell us with an email to Who, Me? ®

Similar topics

Broader topics


Other stories you might like

  • An international incident or just some finger trouble at the console?
    All routers are equal, but some are more equal than others

    Who, Me? Welcome to an edition of Who, Me? where some configuration confusion left an entire nation cast adrift.

    Today's story is set in the early 2000s and comes from a reader Regomized as "Mikael" who was gainfully employed at a European ISP. The company had customers in multiple countries and Mikael's team was responsible for the international backbone.

    "Us senior network engineers were widely regarded as consummate professionals," he told us, before adding, "at least amongst ourselves."

    Continue reading
  • A discounting disaster averted at the expense of one's own employment
    I know what this process needs: Microsoft Access!

    Who, Me? A tale of discounts and process improvement via the magic of Excel, Access and a fair bit of electronic duct tape we imagine. Welcome to Who, Me?

    "James" is the Regomized reader of record today, and continues the theme of running the risk of doing a job just that little bit too well with an ancedote from the end of the last century involving his first job out of university, at a certain telecommunications giant.

    The job involved a process of calculating the discount received by big customers (the ones with multiple branches). "For the life of me I can't remember what the main DB was called," he told us, "but it was the old style green writing on a black screen that took forever to download the necessary data."

    Continue reading
  • In IT, no good deed ever goes unpunished
    When being helpful can mean being shown the door

    Who, Me? Going above and beyond in IT can sometimes lead to also going directly out of the door, as one Register reader found when discovering that sometimes efficiencies can be less than rewarding.

    A reader Regomised as "Will" told of us his days working at a now-defunct company that produced large telephone switches. In those days whenever a major software revision occurred, customers were expected to send in their configurations and Will's group would merge them into the latest and greatest. A new load would then be returned to the customers.

    It was not a fun process, not least because of constant hardware and software failures during the merge process. "When I first started, there was a constant grumble about how unreliable the machine used for the merging was," Will told us.

    Continue reading
  • An early crack at network management with an unfortunate logfile
    It's a backronym, right?

    Who, Me? Come with us on a journey back to the glory days of Visual Basic 6, misplaced enthusiasm and an unfortunate naming incident. Welcome to Who, Me?

    Today's tale comes from a reader Regomised as "Stephen", who was working in the IT department of a Royal Air Force base. "My duties were many," he told us, "from running daily backups of an ancient engineering system using (I kid you not) reel-to-reel tapes to swapping out misbehaving printers."

    This being the early 2000s, his boss loaded up our hero with more tasks. He could change printers and tapes, so Visual Basic (and its bedfellow, Access) should present no problem.

    Continue reading

Biting the hand that feeds IT © 1998–2022