Updated Parts of Amazon Web Services were effectively shoved off the internet today – at times breaking some customers' websites – after the cloud giant came under attack.
Unlucky netizens were intermittently unable to reach sites and other online services relying on the internet goliath's technology as a result of the ongoing outage.
Specifically, according to Amazon's support agents, the AWS DNS servers are being hampered by a distributed denial-of-service (DDoS) attack, which is when miscreants attempt to overwhelm systems with junk network traffic, rendering services inaccessible.
In this case, Amazon's DNS systems are being jammed by a flood of packets, with some legit domain-name queries being inadvertently dropped as part of the mitigations. That means attempts by websites and apps to contact back-end Amazon-hosted systems, such as S3 storage buckets, may fail, resulting in error messages or blank pages for users.
AWS DNS network hijack turns MyEtherWallet into ThievesEtherWalletREAD MORE
For example, if your web application or software tries to talk to your storage bucket at mycloudydata.s3.amazonaws.com, the DNS query to convert that human-readable address into an IP address may not get through to Amazon, and could cause your code to fail. One workaround is to insert the region of the bucket into the address, eg: mycloudydata.s3.us-east-2.amazonaws.com, which should, we're told, resolve correctly.
The partial downtime started about 10 hours ago, or around 0900 US East Coast time, and AWS's DNS servers are still, at time of writing, under siege. This affects more than just S3: it will hamper any connections to Amazon services that rely on external DNS queries, such as the Amazon Relational Database Service (RDS), Simple Queue Service (SQS), CloudFront, Elastic Compute Cloud (EC2), and Elastic Load Balancing (ELB). These are services countless sites and applications rely on to handle visitors and process customer information.
A note on the Jeff-Bezos-run cloud titan's status page right now reads:
Intermittent DNS Resolution Errors
We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution.
Meanwhile, earlier today, AWS customers said they received the following note from support agents indicating the US corp is under a DDoS cyber-assault:
We are investigating reports of occasional DNS resolution errors. The AWS DNS servers are currently under a DDoS attack.
Our DDoS mitigations are absorbing the vast majority of this traffic, but these mitigations are also flagging some legitimate customer queries at this time.
We are actively working on additional mitigations, as well as tracking down the source of the attack to shut it down. Amazon S3 customers experiencing impact from this event can update the configuration of their clients accessing S3 to specify the specific region that their bucket is in when making requests to mitigate impact.
For example, instead of "mybucket.s3.amazonaws.com" a customer would instead specify "mybucket.s3.us-west-2.amazonaws.com" for their bucket in the US-WEST-2 region. If you are using the AWS SDK, you can specify the region as part of the configuration of the Amazon S3 client to make sure your requests use this region-specific endpoint name.
The DNS resolution issues are also intermittently affecting other AWS Service endpoints like ELB, RDS, and EC2 that require public DNS resolution.
We've asked Amazon for more information. About an hour ago, its cloud support desk tweeted: "We're investigating reports of intermittent DNS resolution errors with Route 53 & our external DNS providers."
Digital Ocean, for one, has documented the impact of the DNS outage on its own systems here. ®
Updated to add at 0630 UTC October 23
Amazon now claims the outage is over. In a status page update, it said:
Between 10:30 AM and 6:30 PM PDT, we experienced intermittent errors with resolution of some AWS DNS names. Beginning at 5:16 PM, a very small number of specific DNS names experienced a higher error rate. These issues have been resolved.
However, some customers complain their AWS S3 resources at times remain unreachable. The tech giant's spokespeople are staying silent.
Meanwhile, if AWS's DNS servers go down again, you can deploy a DNS cache to keep a handy copy of queries around and prevent applications from falling over – just make sure you override Amazon's default TTL of a few seconds.