Sysadmins in Sydney had a horrible Sunday, while their CEOs bent their attention to wondering why their Foxtel Rugby sportscasts weren't working.
The Amazon Web Services AWS "power event" AP-SOUTHEAST-2 Region was almost certainly caused by a massive storm system that ran from Brisbane to the NSW South Coast, leading to a weekend of widespread flooding, blocked road and coastal erosion. A lot of usually dry places between Byron Bay and Sydney became navigable waterways.
Affected services included EC2, Elastic Load Balancing, ElastiCache, Redshift, Relational Database Service, Route 53 Private DNS, CloudFormation, CloudHSM, Database Migration Service, Elastic Beanstalk and Storage Gateway.
The outages reached six hours or more.
Twitter users noticed the troubles.
What is happening with AWS?— Jonathan Ferguson (@jonoabroad) June 5, 2016
To Aus customers. Wild weather in Sydney causing power issues https://t.co/ep0Z9bHFPp apologies for any disruption.— My Film Bag (@MyFilmBag) June 5, 2016
AWS EC2 in Sydney was in such bad shape you couldn't even post a support ticket...— geirmagnusson (@geirmagnusson) June 5, 2016
Now my #AWS instance which are still having issues won’t respond to stop/start— Rachael (@halfwrittentale) June 5, 2016
About now we hope AWS finally starts thinking about running an AZ in Australia outside of Sydney— Dawnstar Australis (@dawnstarau) June 5, 2016
There was a bit of “what's going on?” discussion on the Australian Network Operators Group's mailing list (AusNOG), which drew this thundering warning for media not to report list chatter that is posted in plain sight (The Register presumes is in response to a previous Vulture South story that drew from Ausnog posts).
So we won't tell you that some network CEOs were on the job on a Sunday dealing with the problem.
AWS still hadn't completely recovered at 7:00PM, Sydney time, on Sunday night, if Twitter posts were anything to go by.
Amazon Sydney EC2 still down, sorry folks #AWS— FlexiTime (@FlexiTimeTeam) June 5, 2016
The water will drain away, eventually. ®