Microsoft not a Teams player as admin center, 365 service suffer partial outage

It’s not the caching. There’s no way it’s the caching. It was the caching

Final update Microsoft is investigating an issue that is keeping some users from being able to fully use the Microsoft 365 cloud service and Teams admin centers on Thursday.

The problem kicked off this morning with Redmond saying it was looking into errors within its caching infrastructure. In an advisory, the Windows goliath wrote "some users may be intermittently unable to view or access web apps in Microsoft 365."

A range of Microsoft 365 online services are affected, such as Excel, the company wrote, adding "the search bar may not appear in any Office Online service." Others impacted include Teams admin centers, SharePoint Online (users may not be able to view the settings gear, search bar, and waffle), and Planner.

According to DownDetector, complaints of the outage began to spike before 0900 ET (1300 UTC). There's no sign of any resumption in services for the time being.

The software giant initially indicated the problem was linked to an "unusually high number of timeout exceptions within our caching and our Azure Active Directory (AAD) infrastructure." It soon updated that its engineers had narrowed down a cause.

"We determined that a section of caching infrastructure is performing below acceptable performance thresholds, causing calls to gather user licensing information to bypass the cache and go directly to Azure Active Directory infrastructure, resulting in high resource utilization, resulting in throttling and impact," Redmond wrote in an advisory.

"We're analyzing forensics within the caching infrastructure to identify the cause of the issue."

According to netizens, some found that when they logged into their Microsoft 365 accounts, they were unable to access the services. One admin in the UK wrote on Reddit that several of their users reported the problem to them and that they were able to reproduce the faults in several tenants.

"Same symptoms here across our org (Midwest, US) and a few potentially related tickets with clients this morning," wrote another user. "Thought I was going crazy."

Someone writing from Australia said they experienced similar problems as they tried to install Office apps.

"I disabled and re-enabled the 'Microsoft 365 apps for Business' component of a Business Standard license, then refreshed and all was good," they wrote. "Logged in just now and it looks like the problem's returned."

They then logged into the unlicensed Admin account, "repeated the same disable component, save, re-enabled component, save process – then refreshed the licensed user browser and all appears OK again."

Microsoft has battled its share of outages in recent months. A code change caused a four-hour outage of Azure Resource Manager in Europe in March and a month earlier Outlook was knocked out for a while.

In January, Microsoft had to roll back a network change in its WAN after it cause problems a range of cloud services, including Exchange Online, Teams, Outlook, and OneDrive for Business.

The Register contacted Microsoft and will update the story as more information comes in. ®

Updated to add

In an update early this afternoon, on the US East Coast, Microsoft said the focus of its investigation into the outage is on high CPU utilization occurring on “the components that facilitate the back-end API calls for the navigation headers and features. We’re changing some configurations which were part of the service update that occurred on Monday, April 17, 2023, to see if this provides relief whilst we continue to investigate the source of the high CPU utilization.”

The users impacted are those served by the affected infrastructure, with most of the cases being in North and South America, according to the vendor.

Redmond also expanded the list of the affected services to include Yammer (the search bar is missing from the user interface) and Outlook on the web, where users could seee slowness or latency when accessing or using it.

The biz noted that while the Microsoft 365 apps may not render, users can still try to access the applications directly through the URLs, such as Outlook at outlook.office.com and Word Online at microsoft365.com/word/launch.

Final update

The outage is over, and everyone caught up in it should now have a full service, Redmond said at 1310 PT (2010 UTC).

"We've received positive confirmation through our internal telemetry and impacted users that service has been restored," the biz disclosed. "Additional information is available in the admin center under MO544156."

Judging from the feedback online not everyone has got back to normal in Redmond's cloud yet. Let us know in the forums if you're still having issues.