It's always DNS, especially when a sysadmin makes a hash of their semicolons

Remember the days when 'we made it up as we went along'?


Who, Me? DNS (or the Devil's Naming Service as we've heard it called) takes centre stage in this week's tale from the Who, Me? vaults: a warning of the terrors of the forgotten typo.

A Register reader, "Hugo", shared today's story, which takes us back to the late 1990s, when the commercial internet was an optimistic glimpse of the future and outages were a thing that happened when someone accidentally picked up an extension elsewhere in the house.

Hugo was a senior sysadmin for the UK division of a certain global ISP (let's call it "BigNet", for that was certainly not its name).

"BigNet," he told us, "hadn't invested much in tools and automation, and for many things we made it up as we went along.

punchcards - '70s era

Panic in the mailroom: The perils of an operating system too smart for its own good

READ MORE

"There was no customer web portal for anything and they had to raise a ticket, by email, for things like DNS changes which were then actioned by Customer Services."

Happier and simpler times. Until the day Hugo came into work and found the place in uproar.

"DNS was down for every customer, primary domains and secondary," he told us. "The brown stuff really had hit the rotating air displacement machines."

Hugo sprang into action, pulling the DNS server logs and swiftly finding errors. At first they made no sense whatsoever until an awful, creeping realisation dawned.

Remember how he told us that there had been precious little investment in automation? Included in that sacrifice on the altar of corporate perks were tools to edit the DNS. A few helper scripts were used, which basically invoked Vi to edit the zone files.

Vi, for those spared the editor wars, is a venerable text editor much beloved by Unix admins. Others swear by Emacs (others still have been known to just swear at Emacs, but we'll step away from that argument).

As far as the scripts were concerned, there was some simple templating to assist with creating a Start of Authority (SOA) record, but no actual validation of the zone occurred. There was also no history or versioning. There was only the date and owner of the file.

It turned out that Hugo had made the last edit, two weeks ago.

"I had probably been working on a perl or bash script," he told us, "and on the same day I edited the zone file for uk.bignet.net which was in the SOA record for every domain we hosted."

He went on: "In Bind zone files, the comment character is a semicolon, but I accidentally used a hash, and whilst Bind loaded the zone file, it decided it was no longer authoritative, and this went unnoticed."

This was all well and good until the default two weeks time-to-live expired. Since every other domain depended on that one being valid and authoritative, that expiration meant Bind stopped serving all the other domains.

Result: chaos.

The fix was trivial. Hugo switched the comment to a semicolon, hurriedly pushed out the update and restarted all the name servers. The relief was palpable as the services came back up.

Hugo's fate was, unsurprisingly, to create a DNS zone file validator to prevent further "accidents". Mindful of his own brush with a pink slip, Hugo upped the paranoia of the tool from merely warning of errors to issuing a full-on stop when validation failed.

The customer services and provisioning team hated it "for reasons I couldn't understand," he said, "until I also ran the checker across all the domains we hosted and found something like 15 per cent of them had basic errors."

Hugo, it seemed, was not alone when it came to cavalier treatment of critical files.

Ever been struck by the curse of the wrong comment character? Or a mistake made weeks ago rearing its head in a most unpleasant way? Share your tale of woe with an email to Who, Me? ®

Similar topics


Other stories you might like

  • Share your experience: How does your organization introduce new systems?

    The answer is rarely obvious. Take part in our short poll and we'll find out together

    Reg Reader Survey The introduction of new systems into an organization is essential. If we stay still, if we continue to rely on legacy systems, if we fail to innovate – well, we (or, in reality, the company) will die. As business guru Sir John Harvey-Jones once put it: “If you are doing things the same way as two years ago, you are almost certainly doing them wrong.”

    But who should lead innovation in our companies? Who should be introducing new systems? The answer is not obvious.

    On one hand, the introduction of new systems into the business should be led by the business. In principle, the people doing the work, dealing with the suppliers, selling to the customers, are best placed to be standing up and saying: “We need the system to do X,” whether their motivation be to reduce cost, increase revenues, make products more efficiently, or even bolster our environmental credentials.

    Continue reading
  • These Rapoo webcams won't blow your mind, but they also won't break the bank

    And they're almost certainly better than a laptop jowel-cam

    Review It has been a long 20 months since Lockdown 1.0, and despite the best efforts of Google and Zoom et al to filter out the worst effects of built-in laptop webcams, a replacement might be in order for the long haul ahead.

    With this in mind, El Reg's intrepid reviews desk looked at a pair of inexpensive Rapoo webcams in search for an alternative to the horror of our Dell XPS nose-cam.

    Rapoo sent us its higher-end XW2K, a 2K 30fps device and, at the other end of the scale, the 720p XW170. Neither will break the bank, coming in at around £40 and £25 respectively from online retailers, but do include some handy features, such as autofocus and a noise cancelling microphone.

    Continue reading
  • It's one thing to have the world in your hands – what are you going to do with it?

    Google won the patent battle against ART+COM, but we were left with little more than a toy

    Column I used to think technology could change the world. Google's vision is different: it just wants you to sort of play with the world. That's fun, but it's not as powerful as it could be.

    Despite the fact that it often gives me a stomach-churning sense of motion sickness, I've been spending quite a bit of time lately fully immersed in Google Earth VR. Pop down inside a major city centre – Sydney, San Francisco or London – and the intense data-gathering work performed by Google's global fleet of scanning vehicles shows up in eye-popping detail.

    Buildings are rendered photorealistically, using the mathematics of photogrammetry to extrude three-dimensional solids from multiple two-dimensional images. Trees resolve across successive passes from childlike lollipops into complex textured forms. Yet what should feel absolutely real seems exactly the opposite – leaving me cold, as though I've stumbled onto a global-scale miniature train set, built by someone with too much time on their hands. What good is it, really?

    Continue reading
  • Why Cloud First should not have to mean Cloud Everywhere

    HPE urges 'consciously hybrid' strategy for UK public sector

    Sponsored In 2013, the UK government heralded Cloud First, a ground-breaking strategy to drive cloud adoption across the public sector. Eight years on, and much of UK public sector IT still runs on-premises - and all too often - on obsolete technologies.

    Today the government‘s message boils down to “cloud first, if you can” - perhaps in recognition that modernising complex legacy systems is hard. But in the private sector today, enterprises are typically mixing and matching cloud and on-premises infrastructure, according to the best business fit for their needs.

    The UK government should also adopt a “consciously hybrid” approach, according to HPE, The global technology company is calling for the entire IT industry to step up so that the public sector can modernise where needed and keep up with innovation: “We’re calling for a collective IT industry response to the problem,” says Russell MacDonald, HPE strategic advisor to the public sector.

    Continue reading
  • A Raspberry Pi HAT for the Lego Technic fan

    Sneaking in programming under the guise of plastic bricks

    There is good news for the intersection of Lego and Raspberry Pi fans today, as a new HAT (the delightfully named Hardware Attached on Top) will be unveiled for the diminutive computer to control Technic motors and sensors.

    Continue reading
  • Reg scribe spends week being watched by government Bluetooth wristband, emerges to more surveillance

    Home quarantine week was the price for an overseas trip, ongoing observation is the price of COVID-19

    Feature My family and I recently returned to Singapore after an overseas trip that, for the first time in over a year, did not require the ordeal of two weeks of quarantine in a hotel room.

    Instead, returning travelers are required to stay at home, wear a government-issued tracking device, and stay within range of a government-issued Bluetooth beacon at all times for a week … or else. No visitors are allowed and only a medical emergency is a ticket out. But that sounded easy compared to the hotel quarantine we endured in 2020.

    Continue reading
  • Intel teases 'software-defined silicon' with Linux kernel contribution – and won't say why

    It might enable activation of entirely new features on existing Xeon CPUs … or, you know, not

    Intel has teased a new tech it calls "Software Defined Silicon" (SDSi) but is saying almost nothing about it – and has told The Register it could amount to nothing.

    SDSi popped up around three weeks ago in a post to the Linux Kernel mailing list, in which an Intel Linux software engineer named David Box described it as "a post-manufacturing mechanism for activating additional silicon features".

    "Features are enabled through a license activation process," he wrote. "The SDSi driver provides a per-socket, ioctl interface for applications to perform three main provisioning functions." Those provisioning functions are:

    Continue reading
  • Chip manufacturers are going back to the future for automotive silicon

    Where we're going, we don't need 5nm

    Analysis Cars are gaining momentum as computers on wheels, though chip manufacturers' auto focus isn't on making components using the latest and greatest fabrication nodes.

    Instead, companies that include Taiwan Semiconductor Manufacturing Co and Globalfoundries are turning back the clock and investing billions in factories that use older manufacturing techniques to make chips for vehicles.

    The rapid digitization and electrification of cars has created a giant demand for smaller, more power-efficient auto chips, said Jim McGregor, principal analyst at Tirias Research. He added that cars don't necessarily need the latest manufacturing processes, though, and many are still using analog-based components for various functions.

    Continue reading

Biting the hand that feeds IT © 1998–2021