What a pain in the Azzz-ure: Microsoft Azure, SharePoint, etc knocked offline by DNS blunder

Technical term for today's three-hour outage is TITSUP: Total Inability To Support Users' Packets


For at least the past hour or two, Microsoft's Azure cloud has been up and down globally due to a DNS configuration mishap.

The platform-wide outage has knackered all sorts of Redmond-hosted systems around the world, from Azure SQL databases and App Services to multi-factor authentication, Microsoft 365 and Teams, Dynamics, SharePoint Online and OneDrive.

As we type, the cloud giant is gradually getting back on its feet, with Azure regions recovering one by one, though your mileage may vary. It appears trouble started around 1945 UTC.

"Customers may experience intermittent connectivity issues with Azure and other Microsoft services (including M365, Dynamics, DevOps, etc)," the Azure status page reads right now at 2128 UTC.

"Engineers are investigating DNS resolution issues affecting network connectivity. Connectivity issues are resulting in downstream impact to Compute, Storage, and Database services, and some customers may be unable to file support requests.

"More information will be provided as it becomes available. Some customers may start to see recovery."

In other words, the all-clear hasn't been given yet, and anything could happen in the next half hour, as they say.

Today's DNS outage

How the Azure status board looked earlier ... Click to enlarge

Over on the Microsoft 365 status page, Redmond's techies blamed an internal DNS configuration error for the downtime:

Users may be unable to access Microsoft 365 services or features.

More info: Affected services include SharePoint Online, OneDrive for Business, Microsoft Teams, Stream, Power BI, Planner, Forms, PowerApps, Dynamics 365, Intune and Office Licensing.

We've identified and corrected a DNS configuration issue that prevented users from accessing Microsoft 365 services and features. We've observed an increase in successful connections and our telemetry indicates that all services are recovering. We're continuing to monitor the environment to validate that service has been restored.

It wouldn't be the first time DNS problems have knackered Azure – the last time it happened, by our estimation, a few customers' databases went missing, so cross your fingers, folks. ®

Updated to add at 2300 UTC

Microsoft says it's fixed its broken systems, ending today's three-hour outage, and Azure's network infrastructure should more or less be back to normal: "Mitigation has been applied, and most services have recovered, with the exception of a small subset of services who may still experience some impact."

The tech goliath added that a bungled migration of its DNS systems was ultimately to blame: "Engineers identified the underlying root cause as a nameserver delegation change affecting DNS resolution and resulting in downstream impact to Compute, Storage, App Service, AAD, and SQL Database services.

"During the migration of a legacy DNS system to Azure DNS, some domains for Microsoft services were incorrectly updated. No customer DNS records were impacted during this incident, and the availability of Azure DNS remained at 100% throughout the incident. The problem impacted only records for Microsoft services.

"To mitigate, engineers corrected the nameserver delegation issue. Applications and services that accessed the incorrectly configured domains may have cached the incorrect information, leading to a longer restoration time until their cached information expired."

Similar topics


Other stories you might like

  • Stolen university credentials up for sale by Russian crooks, FBI warns
    Forget dark-web souks, thousands of these are already being traded on public bazaars

    Russian crooks are selling network credentials and virtual private network access for a "multitude" of US universities and colleges on criminal marketplaces, according to the FBI.

    According to a warning issued on Thursday, these stolen credentials sell for thousands of dollars on both dark web and public internet forums, and could lead to subsequent cyberattacks against individual employees or the schools themselves.

    "The exposure of usernames and passwords can lead to brute force credential stuffing computer network attacks, whereby attackers attempt logins across various internet sites or exploit them for subsequent cyber attacks as criminal actors take advantage of users recycling the same credentials across multiple accounts, internet sites, and services," the Feds' alert [PDF] said.

    Continue reading
  • Big Tech loves talking up privacy – while trying to kill privacy legislation
    Study claims Amazon, Apple, Google, Meta, Microsoft work to derail data rules

    Amazon, Apple, Google, Meta, and Microsoft often support privacy in public statements, but behind the scenes they've been working through some common organizations to weaken or kill privacy legislation in US states.

    That's according to a report this week from news non-profit The Markup, which said the corporations hire lobbyists from the same few groups and law firms to defang or drown state privacy bills.

    The report examined 31 states when state legislatures were considering privacy legislation and identified 445 lobbyists and lobbying firms working on behalf of Amazon, Apple, Google, Meta, and Microsoft, along with industry groups like TechNet and the State Privacy and Security Coalition.

    Continue reading
  • SEC probes Musk for not properly disclosing Twitter stake
    Meanwhile, social network's board rejects resignation of one its directors

    America's financial watchdog is investigating whether Elon Musk adequately disclosed his purchase of Twitter shares last month, just as his bid to take over the social media company hangs in the balance. 

    A letter [PDF] from the SEC addressed to the tech billionaire said he "[did] not appear" to have filed the proper form detailing his 9.2 percent stake in Twitter "required 10 days from the date of acquisition," and asked him to provide more information. Musk's shares made him one of Twitter's largest shareholders. The letter is dated April 4, and was shared this week by the regulator.

    Musk quickly moved to try and buy the whole company outright in a deal initially worth over $44 billion. Musk sold a chunk of his shares in Tesla worth $8.4 billion and bagged another $7.14 billion from investors to help finance the $21 billion he promised to put forward for the deal. The remaining $25.5 billion bill was secured via debt financing by Morgan Stanley, Bank of America, Barclays, and others. But the takeover is not going smoothly.

    Continue reading

Biting the hand that feeds IT © 1998–2022