Hyphens of mass destruction: When a clumsy finger meant the end for hundreds of jobs

From a time before: 'This will do something awful. Are you sure? (Y/N)'


Who, Me? Welcome back to Who, Me?, The Register's weekly dip into the suspiciously bulging mailbag of reader confessions.

Today's tale of mainframe madness comes roaring in from the 1970s, courtesy of a reader whom we shall call "Jed".

Jed was a computer operator at what was then a large automotive and aerospace component manufacturer. As befits a high-tech industry, the company had a state-of-the-art data centre, in which lurked a mighty IBM System/370 Model 158.

The specs were eye-popping for the time, as Jed explained: "We had about 3.2GB disk storage and about eight tape drives, all the size of a fridge." 1MB storage supported "about 25 IMS terminals and about 50 TSO terminals dotted about the Midlands, robotic machine tools, plus constant batch and application development work."

Pirate

If you're going to exploit work's infrastructure to torrent, you better damn well know how to hide it

READ MORE

To support the same workload today, you'd "need at least an 8-core 64GB server", Jed observed.

"Things worked more efficiently back then," he added.

Other hardware enjoyed by users included screens, keyboards and light pens. "Apart from more colours, higher screen resolution and a mouse, not much different to today."

Since few could afford such power to be at their personal disposal, users would often borrow a bit of time on the mainframe out of hours to do their own work. One morning, by around 6am, a friend ("even operators had friends") of Jed's who was using the computer decided he didn't need any printed output.

"And then the fun started."

Jobs sat in a work queue. Deleting a job was a simple case of typing something along the lines of "£PJ12345", according to Jed, where 12345 was the desired victim. A neat feature of the software, HASP, was that a range could be specified to save laboriously entering numbers.

As each command was processed, it would be displayed on the screen and spat out on the high-speed printer.

"We had all done something similar," Jed recalled.

Things took a dramatic turn for the worse as soon as the purge command was run.

"The screens," said Jed, "were flooded with messages and the printer noise was deafening. Something had gone wrong, the printer noise told us that."

The command was hurriedly checked. Jed's friend had typed "£PJ12345--12349", which looked OK on the screen.

Pat yourself on the back if you spotted the not-so-deliberate mistake. A rogue hyphen had crept in.

"The extra hyphen," said Jed, "meant HASP had been told to purge jobs 12345 to -12349, and was determined to do so."

The Stop button on the computer console was hit, but not before a few hundred jobs had met an untimely end.

"For some reason most of the night shift were laughing, apart from my friend who kept saying 'It's not 'kin funny, it's not 'kin funny', which just prompted more laughter."

And it wasn't funny. The issue was rapidly escalated; the computer was stopped and users were starting to complain. IBM was contacted – how could this evil command, which was set to wipe the night's work, be stopped?

The answer was not good: "Even a reboot (IPL in those days) would not fix the problem."

The command had been recorded and would resume as soon the computer was restarted. There was no going back.

"So the Stop button was released and the noise restarted. To more cries of 'It's not 'kin funny', a mix of managers and operators watched all the overnight work disappear."

Hearteningly, Jed's friend did not cop all the flack for the incident from the data centre manager. In fact, most of the rage was directed at Big Blue, "who had allowed a simple mis-type… to have such big consequences".

Although until a fix was implemented to treat a double hyphen as invalid syntax, the team were warned to be very, very careful about which commands were typed.

Jed's friend went on to become a System Programmer, while Jed himself moved into application and database design and then architecture.

"But I still smile when I remember when a hyphen became a minus."

Ever accidentally set off a command of mass destruction and found yourself unable to stop the havoc? Typed a thing in jest, expecting an "Are You Sure?" but instead found yourself measuring the blast radius? You know you have, and it is time to confess all in an email to Who, Me?

Where were you 20 years ago? Were you frantically cutting COBOL or adding a crucial extra byte or two to a date field? Or a bodge that might last to, oh, 2050 before it explodes? Who, Me? and On Call would also like to hear your sordid Y2K tales for a festive feast of near-failures and dodged bullets. ®

Similar topics


Other stories you might like

  • Everything you wanted to know about modern network congestion control but were perhaps too afraid to ask

    In which a little unfairness can be quite beneficial

    Systems Approach It’s hard not to be amazed by the amount of active research on congestion control over the past 30-plus years. From theory to practice, and with more than its fair share of flame wars, the question of how to manage congestion in the network is a technical challenge that resists an optimal solution while offering countless options for incremental improvement.

    This seems like a good time to take stock of where we are, and ask ourselves what might happen next.

    Congestion control is fundamentally an issue of resource allocation — trying to meet the competing demands that applications have for resources (in a network, these are primarily link bandwidth and router buffers), which ultimately reduces to deciding when to say no and to whom. The best framing of the problem I know traces back to a paper [PDF] by Frank Kelly in 1997, when he characterized congestion control as “a distributed algorithm to share network resources among competing sources, where the goal is to choose source rate so as to maximize aggregate source utility subject to capacity constraints.”

    Continue reading
  • How business makes streaming faster and cheaper with CDN and HESP support

    Ensure a high video streaming transmission rate

    Paid Post Here is everything about how the HESP integration helps CDN and the streaming platform by G-Core Labs ensure a high video streaming transmission rate for e-sports and gaming, efficient scalability for e-learning and telemedicine and high quality and minimum latencies for online streams, media and TV broadcasters.

    HESP (High Efficiency Stream Protocol) is a brand new adaptive video streaming protocol. It allows delivery of content with latencies of up to 2 seconds without compromising video quality and broadcasting stability. Unlike comparable solutions, this protocol requires less bandwidth for streaming, which allows businesses to save a lot of money on delivery of content to a large audience.

    Since HESP is based on HTTP, it is suitable for video transmission over CDNs. G-Core Labs was among the world’s first companies to have embedded this protocol in its CDN. With 120 points of presence across 5 continents and over 6,000 peer-to-peer partners, this allows a service provider to deliver videos to millions of viewers, to any devices, anywhere in the world without compromising even 8K video quality. And all this comes at a minimum streaming cost.

    Continue reading
  • Cisco deprecates Microsoft management integrations for UCS servers

    Working on Azure integration – but not there yet

    Cisco has deprecated support for some third-party management integrations for its UCS servers, and emerged unable to play nice with Microsoft's most recent offerings.

    Late last week the server contender slipped out an end-of-life notice [PDF] for integrations with Microsoft System Center's Configuration Manager, Operations Manager, and Virtual Machine Manager. Support for plugins to VMware vCenter Orchestrator and vRealize Orchestrator have also been taken out behind an empty rack with a shotgun.

    The Register inquired about the deprecations, and has good news and bad news.

    Continue reading

Biting the hand that feeds IT © 1998–2021