It's always DNS, especially when a sysadmin makes a hash of their semicolons

Remember the days when 'we made it up as we went along'?


Who, Me? DNS (or the Devil's Naming Service as we've heard it called) takes centre stage in this week's tale from the Who, Me? vaults: a warning of the terrors of the forgotten typo.

A Register reader, "Hugo", shared today's story, which takes us back to the late 1990s, when the commercial internet was an optimistic glimpse of the future and outages were a thing that happened when someone accidentally picked up an extension elsewhere in the house.

Hugo was a senior sysadmin for the UK division of a certain global ISP (let's call it "BigNet", for that was certainly not its name).

"BigNet," he told us, "hadn't invested much in tools and automation, and for many things we made it up as we went along.

punchcards - '70s era

Panic in the mailroom: The perils of an operating system too smart for its own good

READ MORE

"There was no customer web portal for anything and they had to raise a ticket, by email, for things like DNS changes which were then actioned by Customer Services."

Happier and simpler times. Until the day Hugo came into work and found the place in uproar.

"DNS was down for every customer, primary domains and secondary," he told us. "The brown stuff really had hit the rotating air displacement machines."

Hugo sprang into action, pulling the DNS server logs and swiftly finding errors. At first they made no sense whatsoever until an awful, creeping realisation dawned.

Remember how he told us that there had been precious little investment in automation? Included in that sacrifice on the altar of corporate perks were tools to edit the DNS. A few helper scripts were used, which basically invoked Vi to edit the zone files.

Vi, for those spared the editor wars, is a venerable text editor much beloved by Unix admins. Others swear by Emacs (others still have been known to just swear at Emacs, but we'll step away from that argument).

As far as the scripts were concerned, there was some simple templating to assist with creating a Start of Authority (SOA) record, but no actual validation of the zone occurred. There was also no history or versioning. There was only the date and owner of the file.

It turned out that Hugo had made the last edit, two weeks ago.

"I had probably been working on a perl or bash script," he told us, "and on the same day I edited the zone file for uk.bignet.net which was in the SOA record for every domain we hosted."

He went on: "In Bind zone files, the comment character is a semicolon, but I accidentally used a hash, and whilst Bind loaded the zone file, it decided it was no longer authoritative, and this went unnoticed."

This was all well and good until the default two weeks time-to-live expired. Since every other domain depended on that one being valid and authoritative, that expiration meant Bind stopped serving all the other domains.

Result: chaos.

The fix was trivial. Hugo switched the comment to a semicolon, hurriedly pushed out the update and restarted all the name servers. The relief was palpable as the services came back up.

Hugo's fate was, unsurprisingly, to create a DNS zone file validator to prevent further "accidents". Mindful of his own brush with a pink slip, Hugo upped the paranoia of the tool from merely warning of errors to issuing a full-on stop when validation failed.

The customer services and provisioning team hated it "for reasons I couldn't understand," he said, "until I also ran the checker across all the domains we hosted and found something like 15 per cent of them had basic errors."

Hugo, it seemed, was not alone when it came to cavalier treatment of critical files.

Ever been struck by the curse of the wrong comment character? Or a mistake made weeks ago rearing its head in a most unpleasant way? Share your tale of woe with an email to Who, Me? ®

Similar topics

Broader topics


Other stories you might like

  • An international incident or just some finger trouble at the console?
    All routers are equal, but some are more equal than others

    Who, Me? Welcome to an edition of Who, Me? where some configuration confusion left an entire nation cast adrift.

    Today's story is set in the early 2000s and comes from a reader Regomized as "Mikael" who was gainfully employed at a European ISP. The company had customers in multiple countries and Mikael's team was responsible for the international backbone.

    "Us senior network engineers were widely regarded as consummate professionals," he told us, before adding, "at least amongst ourselves."

    Continue reading
  • A discounting disaster averted at the expense of one's own employment
    I know what this process needs: Microsoft Access!

    Who, Me? A tale of discounts and process improvement via the magic of Excel, Access and a fair bit of electronic duct tape we imagine. Welcome to Who, Me?

    "James" is the Regomized reader of record today, and continues the theme of running the risk of doing a job just that little bit too well with an ancedote from the end of the last century involving his first job out of university, at a certain telecommunications giant.

    The job involved a process of calculating the discount received by big customers (the ones with multiple branches). "For the life of me I can't remember what the main DB was called," he told us, "but it was the old style green writing on a black screen that took forever to download the necessary data."

    Continue reading
  • In IT, no good deed ever goes unpunished
    When being helpful can mean being shown the door

    Who, Me? Going above and beyond in IT can sometimes lead to also going directly out of the door, as one Register reader found when discovering that sometimes efficiencies can be less than rewarding.

    A reader Regomised as "Will" told of us his days working at a now-defunct company that produced large telephone switches. In those days whenever a major software revision occurred, customers were expected to send in their configurations and Will's group would merge them into the latest and greatest. A new load would then be returned to the customers.

    It was not a fun process, not least because of constant hardware and software failures during the merge process. "When I first started, there was a constant grumble about how unreliable the machine used for the merging was," Will told us.

    Continue reading
  • An early crack at network management with an unfortunate logfile
    It's a backronym, right?

    Who, Me? Come with us on a journey back to the glory days of Visual Basic 6, misplaced enthusiasm and an unfortunate naming incident. Welcome to Who, Me?

    Today's tale comes from a reader Regomised as "Stephen", who was working in the IT department of a Royal Air Force base. "My duties were many," he told us, "from running daily backups of an ancient engineering system using (I kid you not) reel-to-reel tapes to swapping out misbehaving printers."

    This being the early 2000s, his boss loaded up our hero with more tasks. He could change printers and tapes, so Visual Basic (and its bedfellow, Access) should present no problem.

    Continue reading

Biting the hand that feeds IT © 1998–2022