Dumpster diving to revive a crashing NetWare server? It was acceptable in the '90s
Broken pencil + ball of stolen Blu Tack = IT joy
Who, Me? Today is Monday the 58th of March 2020. As a service to stop the days blurring into one, take a moment to enjoy a tale of NetWare, Blu Tack and a broken pencil in today's Who, Me? reader confession.
This time we return to the 1990s, and to a region of Australia that our protagonist, a person we'll call "Howard", described as "a three-horse town where everyone knew who's dating everyone else's sister".
Top marks for the use of the word "dating" there, Howard.
Howard was plugging through the end of his education and working part time at a local computer store. His job mainly entailed popping out to deal with customers bigger than those that might be merely toiling along with word processing on a Windows 3.11 box.
His confession starts at what he told us was "a decent-sized financial institution" having trouble with NetWare 3.11. The system in question had to log (for legal reasons) card transactions that had been sent to an AS/400 for batch processing. While IBM's metal kept chugging along, at some point between 2 and 6am the NetWare server would lock solid.
While not privy to what this logging app was doing, "given NetWare 3.11 was co-operative multitasking and the custom logging app was a NLM [NetWare Loadable Module] and ran in Ring0, it was most likely the culprit crashing the whole show... but it was out of scope."
NLMs turned up in NetWare 3.x and were the bane of many an administrator's life. The handy binaries could do all manner of things, such as implementing hardware drivers, but bad behaviour could take down a server or make it unresponsive. Things improved in version 4 and 5, but the smell of instability lingered.
Getting a pizza the action, AS/400 styleREAD MORE
The NLM in question here was off the table so Howard had to come up with another solution.
"The usual procedure," he recalled, "was that one of the locals would arrive for work the next morning and they'd eyeball the server."
No flickering of the hard disk LED after a few minutes meant the reset button needed a prod and the night's transactions manually reconciled. Fortunately, the fact the crashes always occurred in the small hours meant that there weren't too many but...
"Management," he remembered,"would have an early morning meeting containing a minor blame-storm."
This gradually escalated, as the frequency of the incidents increased, into a full blown "rage-fest," according to Howard, which was "shoved through the phone line at us."
An immediate workaround was demanded for the reboots with a fix for the cause needed at some undefined point in the future.
"It already felt like we were circling the drain," sighed Howard.
However, there was this newfangled thing called "Linux" doing the rounds with some flaky NetWare Link Services Protocol (NLSP) support. It wasn't complete, but would allow a type of "Who's on the LAN" broadcast. Parse the response, and it would do for an "Are you alive?"...
"The penny dropped with a resounding clang."
At this point, Howard transformed from being merely a computer shop part-timer with a penguin fetish into a hero for all time.
A battered old 486 box was pulled from storage and had Linux installed. A bit of Perl was used to check if the NetWare server was still alive; if there was no response within a set period then a reset was needed.
And the reset?
"I found a broken CD-ROM mechanism along with its Mitsumi 8bit ISA card in a dustbin, and with a generous amount of stolen Blu Tack sourced from a few dozen or so cubicles, a broken pencil and some tape, this was all added to the front of the non-reading CD-ROM drive as a 'finger'."
You can guess what the next step was.
"When the server would not respond, the Perl script (running as root) would eject /dev/cdrom, send an email to double-check for dropped transaction log entries, and go to sleep for 20 minutes till the server came back up."
The final flourish was the positioning of the old 486 in front of the NetWare tower server. With a pair of Yellow Pages, and some copies of a long-defunct print publication for fine-tuning of height, the CD-ROM tray and its "finger" were perfectly positioned in front of the reset button.
Better still, Howard ensured there was not enough space for the CD-ROM tray to fully open, so having jabbed the reset button it would retract: "Ready to strike again."
"GENIUS!" he told us, modestly.
Despite a slap on the back and the promise of a free beer on Friday, Howard soon moved on to other things and thought no more about his hack until 1999 when a friend told him about "the most dodgy thing [he'd] *EVER* seen in this accounting and finance firm that had a crappy old 486 rebooting a server tower via a pencil on the front of the CD-ROM tray."
Howard's hack had spent half a decade toiling away.
"I laughed," he said, "and drove on quietly."
Can you top Blu Tack, a pencil and broken CD-ROM? Share your tales of blatant bodgery from days long gone (or all too recent). Discretion (and a dodgy pseudonym) assured via an email to Who, Me? ®