Countless Belkin routers go TITSUP in massive mystery meltdown

Maker says it's got it fixed now ... but what was it?


Updated A large number of Belkin routers were knocked off the internet on Tuesday, owing to an as-yet-unexplained glitch.

It's not known how many routers were affected, but it was a lot of them and quite possibly worldwide, all at once. Complaints began popping up on Twitter shortly after midnight Eastern time, with some customers reporting that they couldn't get through to Belkin phone support.

So far, Belkin has acknowledged that there was a problem but not why.

"We are aware of reports of an interruption to internet service when using some Belkin routers with several internet service providers," the company wrote on its status page early Tuesday morning. "Our team is diligently working to [find the] root cause the issue."

It later followed up that note with a temporary workaround, which involved manually configuring devices with static IP addresses and pointing them at Google's public DNS servers.

Belkin now says the problem should be fixed for all routers and that the workaround is no longer necessary, but it still hasn't explained what went wrong.

One early theory was that a firmware update borked the devices. Later, a Reddit thread revealed that the routers were attempting to ping a specific Belkin-owned internet server – heartbeat.belkin.com – and failing, which bizarrely triggered the outage.

Rerouting traffic destined for that host to a working machine solved the problem.

"I'm also able to ping that address from routers in the same IP space as some Belkins that are unable," Reddit user VictimOfAReload wrote. "So it's not a specific network issue. Just the destination not responding for some strange reason."

Belkin has not commented on any of this speculation, saying only, "We sincerely apologize for this inconvenience to our customers. We are taking a number of actions to eliminate this sort of incident from reoccurring."

Customers whose routers are still not working are advised to do a hard reset by powering down the devices and waiting one minute before plugging them back in. ®

Update

A Belkin spokesperson emailed The Reg on Wednesday afternoon with a statement that offered more insight into the outage:

One of our cloud services associated with maintaining router operations was negatively impacted by a change made in our data center that caused a false denial of service. Normal operations were restored by 3PM PST, however, some users might still be required to reset their router and/or cable modem to regain connectivity. Moving forward, we will continue to monitor, improve and validate the system to ensure our routers continue to work properly in the event connectivity to our cloud environment is not available.

How "false" the outage seemed to affected Reg readers, however, we leave to individual interpretation.

Similar topics

Broader topics


Other stories you might like

  • Stolen university credentials up for sale by Russian crooks, FBI warns
    Forget dark-web souks, thousands of these are already being traded on public bazaars

    Russian crooks are selling network credentials and virtual private network access for a "multitude" of US universities and colleges on criminal marketplaces, according to the FBI.

    According to a warning issued on Thursday, these stolen credentials sell for thousands of dollars on both dark web and public internet forums, and could lead to subsequent cyberattacks against individual employees or the schools themselves.

    "The exposure of usernames and passwords can lead to brute force credential stuffing computer network attacks, whereby attackers attempt logins across various internet sites or exploit them for subsequent cyber attacks as criminal actors take advantage of users recycling the same credentials across multiple accounts, internet sites, and services," the Feds' alert [PDF] said.

    Continue reading
  • Big Tech loves talking up privacy – while trying to kill privacy legislation
    Study claims Amazon, Apple, Google, Meta, Microsoft work to derail data rules

    Amazon, Apple, Google, Meta, and Microsoft often support privacy in public statements, but behind the scenes they've been working through some common organizations to weaken or kill privacy legislation in US states.

    That's according to a report this week from news non-profit The Markup, which said the corporations hire lobbyists from the same few groups and law firms to defang or drown state privacy bills.

    The report examined 31 states when state legislatures were considering privacy legislation and identified 445 lobbyists and lobbying firms working on behalf of Amazon, Apple, Google, Meta, and Microsoft, along with industry groups like TechNet and the State Privacy and Security Coalition.

    Continue reading
  • SEC probes Musk for not properly disclosing Twitter stake
    Meanwhile, social network's board rejects resignation of one its directors

    America's financial watchdog is investigating whether Elon Musk adequately disclosed his purchase of Twitter shares last month, just as his bid to take over the social media company hangs in the balance. 

    A letter [PDF] from the SEC addressed to the tech billionaire said he "[did] not appear" to have filed the proper form detailing his 9.2 percent stake in Twitter "required 10 days from the date of acquisition," and asked him to provide more information. Musk's shares made him one of Twitter's largest shareholders. The letter is dated April 4, and was shared this week by the regulator.

    Musk quickly moved to try and buy the whole company outright in a deal initially worth over $44 billion. Musk sold a chunk of his shares in Tesla worth $8.4 billion and bagged another $7.14 billion from investors to help finance the $21 billion he promised to put forward for the deal. The remaining $25.5 billion bill was secured via debt financing by Morgan Stanley, Bank of America, Barclays, and others. But the takeover is not going smoothly.

    Continue reading

Biting the hand that feeds IT © 1998–2022