Countless Belkin routers go TITSUP in massive mystery meltdown
Maker says it's got it fixed now ... but what was it?
Updated A large number of Belkin routers were knocked off the internet on Tuesday, owing to an as-yet-unexplained glitch.
It's not known how many routers were affected, but it was a lot of them and quite possibly worldwide, all at once. Complaints began popping up on Twitter shortly after midnight Eastern time, with some customers reporting that they couldn't get through to Belkin phone support.
So far, Belkin has acknowledged that there was a problem but not why.
"We are aware of reports of an interruption to internet service when using some Belkin routers with several internet service providers," the company wrote on its status page early Tuesday morning. "Our team is diligently working to [find the] root cause the issue."
It later followed up that note with a temporary workaround, which involved manually configuring devices with static IP addresses and pointing them at Google's public DNS servers.
Belkin now says the problem should be fixed for all routers and that the workaround is no longer necessary, but it still hasn't explained what went wrong.
One early theory was that a firmware update borked the devices. Later, a Reddit thread revealed that the routers were attempting to ping a specific Belkin-owned internet server – heartbeat.belkin.com – and failing, which bizarrely triggered the outage.
Rerouting traffic destined for that host to a working machine solved the problem.
"I'm also able to ping that address from routers in the same IP space as some Belkins that are unable," Reddit user VictimOfAReload wrote. "So it's not a specific network issue. Just the destination not responding for some strange reason."
Belkin has not commented on any of this speculation, saying only, "We sincerely apologize for this inconvenience to our customers. We are taking a number of actions to eliminate this sort of incident from reoccurring."
Customers whose routers are still not working are advised to do a hard reset by powering down the devices and waiting one minute before plugging them back in. ®
A Belkin spokesperson emailed The Reg on Wednesday afternoon with a statement that offered more insight into the outage:
One of our cloud services associated with maintaining router operations was negatively impacted by a change made in our data center that caused a false denial of service. Normal operations were restored by 3PM PST, however, some users might still be required to reset their router and/or cable modem to regain connectivity. Moving forward, we will continue to monitor, improve and validate the system to ensure our routers continue to work properly in the event connectivity to our cloud environment is not available.
How "false" the outage seemed to affected Reg readers, however, we leave to individual interpretation.