Funny, that: Handy script for wiping directories is capable of wreaking havoc beyond a miscreant's wildest dreams

Beware the three options of the apocalypse


Who, Me? Remember when September seemed so far away? Those of you still working from your bedroom since March should probably have changed your pyjamas by now. We'll wait. When you're ready, enjoy a tale from the Who, Me? vault courtesy of a reader who knows all about unplanned undergarment changes.

"O" spent the early part of this century in the capable hands of Windows Server 2003, and was always keen to find new ways of keeping his systems ticking over: "Being the only experienced professional running the systems with a few 'independent consultants' who looked down at any really hard server work," he said, modestly, "there was always an area to improve."

The consultants managed only one of the company's five or so sites and while complexity was swerved where possible, they were "very vocal about any issue that made them look good".

The ad-hoc spaghetti of the gang's various solutions had been beaten into submission through "Quality Control" and "Change Management" although disk space woes remained a constant. The issue, recalled O, was "due to a surprisingly high resistance to RAID5 (and the crazy cost of those SCSI disks!)"

Upgrading the primary server (lurking in the same location as the consultants) was to be a simple job. O would fly in, plan, start and finish the task. Everyone was happy.

Well, not quite.

panic

This PDP-11/70 was due to predict an election outcome – but no one could predict it falling over

READ MORE

"A new cluster with plenty of disk space seemed to draw out the complaints on disk space across the other regions," O explained. So he put together a script that could read a list of user-approved sacrificial directories when disk space was getting low, and wipe the things.

A cautious fellow, he also had the script check which server it was running on ("to prevent misfires!") and added a "reasonable amount of intelligence and automation".

As with so many things in the IT world, O's handy script sat in the background and was soon forgotten about as new nodes, BDCs and the like were added with the expansion of the company.

It wasn't until a planned weekend outage that the wheels first showed signs of coming off. O noted that some servers were blocking updates due to disk issues.

Not a problem. He remoted into the consoles of the servers, double-checked where he was and ran his handy clean-up script.

"I watched it start," he said, "then went off to do the other patching reboots."

The calls began eight hours later. A newly added BDC in the south of the country had died, showing winnt errors on reboot. Odd, but after fettling a boot disk (and thanks given to how the disk system, NTFS, managed Access Control Lists) things were soon back up and running.

"Initial root cause was bad disk," said O, "because of the lack of investment into RAID5."

Time passed, and even the BDC incident began to fade into memory. And then another call came in, this time from the site with the cluster and that team of consultants.

"A failover had happened, and things looked bad," explained O. "A PDC going down was bad news, but adding to the complexity was the main site and ... the consultants"

They looked at the problem and helpfully determined that, no, this wasn't RAID-related. Instead the passive node's C: drive looked like somebody had run del *.* /s on it.

How on earth could such a destructive command (which would have a crack at deleting all files in the current directory and all subdirectories) possibly have been run?

"Looking back at the 'bulletproof' script in hindsight as I was cleaning my desk," sighed O, "the script was not designed to run where it should not run.

"It was designed to check for the server name and execute based on where it was meant to run."

However, if it didn't find the name, it did not simply exit. Oh no.

"The script added 'del specific drive\path\', but when running on a new node for which it hadn't been told about it became only 'del \*.* ..."

As well as the /s parameter, O added /f (to force the deletion of read-only files) and /q (to stop Windows asking if the user was sure about that wildcard). /s, /f and /q – the three options of the apocalypse.

It was, he said ruefully, "the cherry on top".

Ever created what you thought was the neatest utility ever, only to realise that you have unleashed a data-destroying monster? Or conducted an impromptu test of your company's backup and restore strategy? You have? Then an email to Who, Me? will clear your conscience. ®

Similar topics

Broader topics


Other stories you might like

  • September 16, 1992, was not a good day to be overly enthusiastic about your job
    If I get in early and work hard, everyone will notice, right?

    Who, Me? "The early bird trashes the business" is a saying that we've just made up, but could easily apply to the Register reader behind a currency calamity in today's episode of Who, Me?

    Our hero, Regomized as "Mike", was working as a "data entry operative" for a tourism company in 1992. The company ran bus tours to the then brand-new EuroDisney, parent company of Disneyland Paris (now the most visited theme park in Europe), which had opened earlier that year.

    Mike was an eager beaver, his youthful naivete having convinced him that if he worked extra hard, came in extra early, and kept the in-tray clear, then his efforts would be both noticed and rewarded with promotion and a bump in pay.

    Continue reading
  • An international incident or just some finger trouble at the console?
    All routers are equal, but some are more equal than others

    Who, Me? Welcome to an edition of Who, Me? where some configuration confusion left an entire nation cast adrift.

    Today's story is set in the early 2000s and comes from a reader Regomized as "Mikael" who was gainfully employed at a European ISP. The company had customers in multiple countries and Mikael's team was responsible for the international backbone.

    "Us senior network engineers were widely regarded as consummate professionals," he told us, before adding, "at least amongst ourselves."

    Continue reading
  • A discounting disaster averted at the expense of one's own employment
    I know what this process needs: Microsoft Access!

    Who, Me? A tale of discounts and process improvement via the magic of Excel, Access and a fair bit of electronic duct tape we imagine. Welcome to Who, Me?

    "James" is the Regomized reader of record today, and continues the theme of running the risk of doing a job just that little bit too well with an ancedote from the end of the last century involving his first job out of university, at a certain telecommunications giant.

    The job involved a process of calculating the discount received by big customers (the ones with multiple branches). "For the life of me I can't remember what the main DB was called," he told us, "but it was the old style green writing on a black screen that took forever to download the necessary data."

    Continue reading
  • In IT, no good deed ever goes unpunished
    When being helpful can mean being shown the door

    Who, Me? Going above and beyond in IT can sometimes lead to also going directly out of the door, as one Register reader found when discovering that sometimes efficiencies can be less than rewarding.

    A reader Regomised as "Will" told of us his days working at a now-defunct company that produced large telephone switches. In those days whenever a major software revision occurred, customers were expected to send in their configurations and Will's group would merge them into the latest and greatest. A new load would then be returned to the customers.

    It was not a fun process, not least because of constant hardware and software failures during the merge process. "When I first started, there was a constant grumble about how unreliable the machine used for the merging was," Will told us.

    Continue reading

Biting the hand that feeds IT © 1998–2022