Telstra's confession to DNS-messin' explains broadband borkage

Outage extends beyond a week so carrier ships free modems to the afflicted

Australia's dominant carrier, Telstra, has revealed that a botched fix for a DNS service is the reason for a week-long outage that has taken some of its broadband customers offline.

The company has explained that a seemingly-innocuous software update to Domain Name Servers (DNS) caused some of them fail and Telstra's DNS services went down.

Modems and gateways provided to customers did not enjoy the absence of DNS, because the heartbeat packets they sent were not acknowledged. The modems therefore reset themselves in an attempt to sort things out, which made for messes last Thursday and Friday until DNS returned and the network started behaving again. About 370,000 thousand customers copped that outage.

But some modems never recovered from losing DNS and “resetting continued for a small percentage of our customers across the weekend and into this week.”

Telstra twice called "all-clear" for its broadband services, but now acknowledges that some of those modems only started working again after a factory reset. Some did not re-connect and have been bricked. The latter modems are being replaced free of charge, for about 15,000 customers.

Telstra's explanation for the incident remorsefully regrets it has lost customers' trust and pledges to turn things around. Which will be hard to do: the company positions its networks as a premium product worthy of premium pricing. Which given the long list of recent outages is now rather hard to sustain.

A wider question is whether Telstra is just having a run of bad luck or if something has changed inside the company that makes it more prone to outages. The company has made significant redundancies, but most have been concentrated in call centres and there's little evidence staff responsible for network operations have departed.

Yet something is clearly wrong and meaningful change, not apologies and platitudes about the extent of the company's network investments, must surely be on managers' minds. ®