Google blames Gmail outage on data centre collapse

Domino effect crashes through the cloud


Google has apologised for yesterday’s major Gmail meltdown after some of its data centres in Europe failed to cope with a routine maintenance event.

The company’s web-based email service was flat as a pancake for about three hours yesterday morning as Mountain View engineers attempted to fix the problem, which affected vast swathes of the firm’s 113m strong global user base.

Businesses and individuals were unable to gain access to their email accounts via the internet until around 12:30 GMT on Shrove Tuesday.

“This morning, there was a routine maintenance event in one of our European data centers," said Google’s site reliability boss Acacio Cruz yesterday. "This typically causes no disruption because accounts are simply served out of another data centre.

“Unexpected side effects of some new code that tries to keep data geographically close to its owner caused another data centre in Europe to become overloaded, and that caused cascading problems from one data center to another. It took us about an hour to get it all back under control.”

Google said it had fixed the bugs and also shared the pain with its users.

“We run Google on Gmail, so outages like this affect us the same way they affect you. We always investigate the root causes of rare outages like this one, so we can prevent similar problems in the future,” said Cruz.

Many complained on the Google Apps and Gmail forums about the lengthy outage, while others worked around the problem by accessing their email using Imap on a desktop client or through their iPhone or other handset apps.

Premier Edition Google Apps customers such as The Guardian and Salesforce have been promised at least 99.9 per cent Gmail uptime per month. If Google fails to deliver on the terms of its service level agreement it has to cough up a penalty.

That translates as a commitment to no more than 45 minutes of downtime a month - so presumably, affected businesses will be expecting to see a cheque in the post.

In fact Google is offering paying customers who cough up $50 per user, per year for the service, 15 days free credit, according to Associated Press. ®

Broader topics


Other stories you might like

  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading
  • Amazon investors nuke proposed ethics overhaul and say yes to $212m CEO pay
    Workplace safety, labor organizing, sustainability and, um, wage 'fairness' all struck down in vote

    Amazon CEO Andy Jassy's first shareholder meeting was a rousing success for Amazon leadership and Jassy's bank account. But for activist investors intent on making Amazon more open and transparent, it was nothing short of a disaster.

    While actual voting results haven't been released yet, Amazon general counsel David Zapolsky told Reuters that stock owners voted down fifteen shareholder resolutions addressing topics including workplace safety, labor organizing, sustainability, and pay fairness. Amazon's board recommended voting no on all of the proposals.

    Jassy and the board scored additional victories in the form of shareholder approval for board appointments, executive compensation and a 20-for-1 stock split. Jassy's executive compensation package, which is tied to Amazon stock price and mostly delivered as stock awards over a multi-year period, was $212 million in 2021. 

    Continue reading
  • Confirmed: Broadcom, VMware agree to $61b merger
    Unless anyone out there can make a better offer. Oh, Elon?

    Broadcom has confirmed it intends to acquire VMware in a deal that looks set to be worth $61 billion, if it goes ahead: the agreement provides for a “go-shop” provision under which the virtualization giant may solicit alternative offers.

    Rumors of the proposed merger emerged earlier this week, amid much speculation, but neither of the companies was prepared to comment on the deal before today, when it was disclosed that the boards of directors of both organizations have unanimously approved the agreement.

    Michael Dell and Silver Lake investors, which own just over half of the outstanding shares in VMware between both, have apparently signed support agreements to vote in favor of the transaction, so long as the VMware board continues to recommend the proposed transaction with chip designer Broadcom.

    Continue reading

Biting the hand that feeds IT © 1998–2022