Nearly quarter of a million O2 customers struggled to make calls, send texts and use their mobile internet connections yesterday.
The outage, triggered during upgrade work on the operator's 2G and 3G systems, also hit Tesco Mobile and came only a day after O2-owned GiffGaff suffered similar problems.
Tesco and GiffGaff are both virtual mobile operators camped on O2's phone network, so they share some of their parental infrastructure. Both had partial outages this week, though GiffGaff was struggling back to its feet just as O2 stumbled.
O2 attributes the outage to "routine upgrade work" leading to "unexpected congestion" resulting in "intermittent problems" lasting all day, but isn't saying any more. It claims one per cent of its customers fell off the network.
The characteristically open GiffGaff has admitted to botching an MSC/HRL (Mobile Switching Centre, Home Location Register) upgrade which led to overloading on Wednesday evening, but GiffGaff was up and running well before O2 dropped out.
GiffGaff doesn't share O2's MSC/HLR so it would seem the problem is unrelated, if coincidental. Tesco Mobile, which is half owned by O2, does share more infrastructure with its parent and Tesco customers were hit with problems lasting all Thursday.
O2 pegs the outage running from 1045AM BST to 1145PM BST - 13 hours of intermittent coverage hitting "around 1 per cent" of customers, which would be 220,000 based on their latest figures and not including those on Tesco's network.
The official line is that the problems were caused by upgrades to the 2G and 3G network, but the problems were was national so clearly nothing to do with building more base stations or pushing coverage into more homes - it was a signalling problem in the core that left customers disconnected.
And it's not the first time. Last year O2's attempt to install Ericsson's Centralized User Database led to two national outages, including the unprecedented 21 hours of down time that whacked O2 customers last July. Back then the company's chief operating officer promised: "We are not prepared to risk this happening to our customers for a third time ... performance will now be measured on customers’ confidence in our network."
Yesterday that confidence took another knock, and it will take more than canned promises to rebuild it. ®
Sponsored: Webcast: Simplify data protection on AWS