AWS customer faces staggering charges over S3 bucket misfire

Open source tool fingered for 100 million PUT requests and $1,300 in a day

Updated AWS looks set to intervene after a customer highlighted a flaw that allows S3 bucket owners to be stung with potentially massive charges for attempted accesses they have no control over.

Amazon's Simple Storage Service (S3) was the first and one of the most widely used of the cloudy giant's online services, and also regularly crops up in the news because of breaches caused by poorly configured security settings.

This latest incident also stemmed from misconfiguration, but not of S3 itself; the service was performing exactly as it was designed.

In an article posted on Medium this week, a software engineer complained that an S3 bucket he created as part of a proof-of-concept had managed to run up charges of over $1,300 in a single day. A check of the AWS billing console showed that the cause was nearly 100 million PUT requests to add data to the bucket, he said.

Maciej Pocwierz, a senior software engineer at Warsaw-based cloud services company Semantive, writes that he created a single S3 bucket in Amazon's eu-west-1 region and uploaded some files there for testing. Two days later, he checked the billing page to make sure this was still within the free-tier limits and discovered the charges.

The source of all the PUT requests, according to Pocwierz, is a popular open source tool that he doesn't identify. This tool stores backup data in S3 by default, and the placeholder bucket name it uses just happens to be identical to the one that he chose for his project.

Where this becomes a problem – apart from your bucket filling up with other people's data if those PUT requests were successful – is that Amazon charges for unauthorized incoming requests. He claims this was confirmed by AWS in exchanges he had with its support team regarding the matter.

Standard S3 PUT requests are priced at just $0.005 per 1,000 requests, which may seem like a trifling amount, but Pocwierz points out that a single machine can easily execute thousands of such requests per second.

To demonstrate the security implications of this, Pocwierz said that he opened up his S3 bucket for public writes, and in less than 30 seconds it amassed over 10 GB of data from numerous sources.

That's 10 GB of data that the owners are likely to be completely unaware was being exfiltrated to a random S3 bucket by some open source tool they are using, all because they didn't configure its backup function.

But it didn't take long for this complaint to get noticed, especially when people started posting links to the Medium article on Twitter. In response, AWS chief evangelist Jeff Barr indicated in a tweet that company would do something about the situation:

Thank you to everyone who brought this article to our attention. We agree that customers should not have to pay for unauthorized requests that they did not initiate. We'll have more to share on exactly how we'll help prevent these charges shortly.

We asked AWS for an official statement on this, but the company declined to say anything beyond Jeff Barr's message.

Pocwierz said he informed the maintainers of the open source tool about the issue and that they have fixed it in the code, but this doesn't fix the many instances of the tool that are still running in the wild.

The takeaway is that anyone who knows the name of an S3 bucket can send it PUT requests, and potentially rack up massive charges for the AWS account that owns it.

Until AWS comes up with a fix, customers will have to attempt to alleviate this risk by avoiding short or common names for S3 buckets, and making them less easy to guess by adding random characters. ®

Updated to add on May 14:

AWS announced on May 13 that S3 will make a change so that "unauthorized requests that customers did not initiate are free of charge."

The tweak won't happen immediately for everyone, although the billing changes will apply in all AWS Regions, "including the AWS GovCloud Regions and the AWS China Regions." The cloud giant added: "This deployment is starting today and we will post another update in a few weeks when it is completed."

The cloud giant said S3 will no longer charge for "several HTTP error codes," adding: "With this change, bucket owners will never incur request or bandwidth charges for requests that return an HTTP 403 (Access Denied) error response if initiated from outside their individual AWS account or AWS Organization.

"To see the full list of error codes that are free of charge, visit Billing for Amazon S3 error responses. This billing change requires no changes to customer applications and applies to all S3 buckets."