You had one job: Akamai's Prolexic Denial-of-Service protection system fingered after users in Australia denied, er, services
Major banks, websites, gaming services, and more taken down
Updated A wide range of internet-connected services in Australia, including banking systems, are experiencing an outage – and it looks like a hiccup at Akamai was at the heart of the problem.
Reports of issues with news sites, gaming services, and – more critically – banking systems began to spread on Australian social media early Thursday afternoon local time. Uptime-tracking service Downdetector concurred, showing a massive spike in issues at Commonwealth Bank, Westpac, Bankwest, St.George Bank, Bank Australia, Bank of Melbourne, and others.
"We're aware some of you are experiencing difficulties accessing our services and we're urgently investigating," Commonwealth Bank told customers. "We apologise and thanks for your patience, we’ll provide an update soon."
The root cause of the outage is still unknown but fingers soon began pointing to Akamai Technologies, a multinational cloud service company which provides – among other things – a system for defending against Distributed Denial-of-Service (DDoS) attacks.
Akamai told The Reg this morning: "We are aware of the issue and actively working to restore services as soon as possible."
- Cloudflare network outage disrupts Discord, Shopify
- AWS Frankfurt experiences major breakdown that staff couldn't fix for hours due to 'environmental conditions' on data centre floor
- Fastly 'fesses up to breaking the internet with an 'an undiscovered software bug' triggered by a customer
- IBM Cloud resets 'Days Since Last Major Incident' clock to zero – after just five days
Its defence system, dubbed Prolexic, may be to blame: local Aussie outlet ITnews has posted a support chat session in which an Akamai agent advised a customer to "kindly turn off/route off [the] Prolexic solution" in order to resolve the issue.
While fault tracking indicates that services are beginning to return to normal, users still appear to be experiencing difficulties connecting to banking systems. "I've been stuck at Coles for two hours now," one bank user complained. "Could I get a Zinger Stacker Box coupon for the wait? Cheers."
"Services are still down," said another, a full hour after Commonwealth Bank had stated it was "starting to see services return to normal". "Can't use the app. Can't login online. Come on CBA, I've been trying all afternoon, for the last three hours, to check my account." ®
Updated to add at 09:01 UTC 18 June 2021:
Akamai confirmed in a statement last night that it had indeed experienced an outage for one of its Prolexic DDoS services, specifically "version 3.0 of [its] routed service" starting at 4:20 AM UTC.
"Many of the approximately 500 customers using this service were automatically rerouted, which restored operations within a few minutes. The large majority of the remaining customers manually rerouted shortly thereafter."
The issue was not caused by a system update or a cyberattack. A routing table value used by this particular service was inadvertently exceeded. The effect was an unanticipated disruption of service.
We restored the service by 8:47 AM UTC, and customers began the process of routing back on to the service at that time.
The firm said it had done some prophylactic work to prevent a recurrence and would be ensuring "every Akamai customer is set up for automatic rerouting" in the future.