Google Cloud (over)Run: How a free trial experiment ended with a $72,000 bill overnight

Billing budget? Free plan? All useless when buggy code went into overdrive

Sudeep Chauhan, founder of startup Milkie Way, suffered a bad case of bill shock when a test with a $7.00 billing budget and a free database plan on Google Cloud platform (GCP) generated a $72,000 invoice overnight.

"I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000," Chauhan wrote on his company's blog. "Super stressed, and not sure what happened, I clicked around, trying to figure out what was happening. I also started thinking of what may have happened, and how we could possibly pay the $5K bill. The problem was, every minute the bill kept going up. After two hours, it settled at a little short of $72,000."

It was especially surprising that it happened to Chauhan, who is ex-Google and even spent two years as a payments technical program manager. What happened?

The idea was to build a system that scraped web pages and stored the results in a database. His team picked Google Cloud Run, a GCP service that runs containers, for the job. They then found their code in each instance would timeout and stop as it scraped one page after the other. So, they set up a many-instance system that processed pages in parallel to get each page fetched and stored within the run-time limit.

Devs invited to bake 'Run on Google Cloud' button into git repos... By Google, of course


Chauhan wrote: "To overcome the timeout limitation, I suggested using POST requests (with URL as data) to send jobs to an instance, and [to] use multiple instances in parallel instead of using one instance serially. Because each instance in Cloud Run would only be scraping one page, it would never time out, process all pages in parallel (scale), and also be highly optimized because Cloud Run usage is accurate to milliseconds."

The ex-Googler reflected that he missed the possibility of pages that link back to each other, causing "infinite recursion." It should not have mattered too much, though: he set a billing budget of $7.00 and had a Firebase database on a free plan. "The worst case we imagined was exceeding the daily free Firestore limits," he said. Further, the credit card for the account had a spending limit of $100.

Unfortunately, a billing budget "does not automatically cap Google Cloud or Google Maps Platform usage/spending," according to the docs.

While Chauhan was asleep after a day of testing, Google sent an automated email informing him that his free Firebase plan had been "upgraded due to activity in Google Cloud," and that this "initiated billing" for the project.

He discovered multiple issues with the GCP cost controls. "Billing takes about a day to be synced, and that's why we noticed the charges the next day," Chauhan said. Next, the "Firebase Dashboard took more than 24 hours to update," he said. This meant that the dashboard showed usage within the daily limit, when it was, he said, "86 million percentage points" more than what was shown.

Billing takes about a day to be synced, and that's why we noticed the charges the next day

The GCP Cloud Run defaults also played their part. "The max-instances is preset to 1,000, and concurrency set to 80," he said. If he had corrected this to small values like 2 and 1, the bill shock would not have occurred.

Thanks to these settings, "running [out] this version of Hello World deployment on Cloud Run made 116 billion reads and 33 million writes to Firestore," said Chauhan.

Most of the cost was down to Firebase read operations, even at just $0.06 per 100,000. Multiply that by 116 billion and you get $69,600. There was also the small matter of 16,000 hours of Cloud Run Compute time, partly because the application did not delete the services but left them "in background process".

The performance of the buggy code was impressive in its way. "At the peak, Firebase was able to handle about one billion reads per minute," he said, while Cloud Run with concurrency "can handle 9 million requests per minute".

"Fail fast, learn fast with cloud is a bad idea," Chauhan concluded. "If you count the number of pages in GCP documentation, it's probably more than pages in [a] few novels. Understanding pricing, usage, is not only time consuming, but requires a deep understanding of how cloud services work."

There is a happy ending. "After going through our lengthy doc on this incident sharing our side of the story, various consults, talks, and internal discussions, Google let go of our bill as a one-time gesture," said Chauhan.

Such leniency cannot be relied upon. Auto-scaling and on-demand computing has downsides, and working out what something will cost is challenging. Caution is advised. ®

Send us news

AWS wins yet another UK public-sector contract – this time to provide £15m health data system for NHS Scotland

Amazon's cloud factory hits £300m under 'One Government' agreement

NHS Education for Scotland has awarded AWS a £15m contract to host its National Digital Platform, an architecture to share data across the nation's health service.

The education and training body within NHS Scotland said that the platform would be designed to "create and deploy real time data at the point of care", "operate to a predictable architecture to enable new and innovative products to be developed and implemented" as well as "enable the use of data at scale for quality improvement and to support research and innovation", according to a tender notice.

The cloud infrastructure biz is set to host the data platform, including repositories of structured and unstructured clinical data, web services to power web and mobile applications, an integration layer, and web app. The platform is intended to enable the creation and use of information at source and facilitate the interoperability of existing and new healthcare technologies following the publication of the Digital Health and Care Strategy for Scotland in 2018.

Continue reading

Hong Kong floats doxxing laws that would let it force big tech to take down content

And make publishing personal data as an act of protest illegal into the bargain

Hong Kong’s Legislative Bureau has proposed amendments to local laws that that strengthen penalties for doxxing, and empower its Privacy Commissioner to request content removal from platforms and legally enforce compliance.

Doxxing became an issue in Hong Kong after the 2019 introduction of a law that would have made it easier for locals to be tried in Chinese courts. Protests against the law saw activists publish personal information about police and court staff.

From June 2019 until September 2020, the Office of the Privacy Commissioner for Personal Data handled over 4,700 doxxing cases. Of those doxxed, 35 percent where police officers or their family, four percent were public servants and government officials, 30 percent expressed support to the government or the police, and 32 percent voiced opposition to the government or police. Another 1,000 cases have popped up between September 2020 and April 2021.

Continue reading

Blessed are the cryptographers, labelling them criminal enablers is just foolish

Preserving privacy is hard. I know because when I tried, I quickly learned not to play with weapons

Column Nearly a decade ago I decided to try my hand as a cryptographer. It went about as well as you might expect. I’d gotten the crazy idea to write a tool that would encrypt Twitter’s direct messages - sent in the clear - so that your private communications would truly be private, visible to no one, including Twitter.

Writing the code turned out to be surprising easy; as I wrote it all in Python, I had libraries to handle the Twitter integration, and the cryptography. I read up a bit on the theory, put the pieces together, and with a bit of debugging “CrypTweet” was up and running.

Next step: sharing my brand-new code with the world, spruiking it as the privacy solution every Twitterer needed.

Continue reading

IBM compiles dataset to teach software how software is made: 14m code samples, half of which actually work

Big Blue hopes to create the ImageNet of training resources for AI-powered programming tools

Think IBM has assembled a massive silo of source code for teaching machine-learning programs about programming.

Dubbed Project CodeNet, the set contains, we're told, 14 million code samples totaling 500 million lines in more than 55 programming languages, from Java, C, and Go to COBOL, Pascal, and FORTRAN. Truth be told, more than three-quarters of it all is in C++ and Python.

This source code wasn't taken from production nor in-development applications: it was collected from entries submitted to two programming contests organized in Japan: Aizu and AtCoder. In these contests, competitors are challenged to write the necessary code to turn a given set of inputs into a set of desired outputs. About half of the samples work as expected, and the rest are labeled as either wrong solutions, non-building, or buggy.

Continue reading

Google gets into the international money transfer business, one-way out of the USA

India and Singapore are first destinations, teams with Western Union and Wise to target 200 nations

The verb “To Google” may soon have new meaning, as the ads-and-search giant has added the ability to “Google” money across borders with its Pay app.

The new offering is only for US-based users, for now, and allows the Pay app to hook into payment networks operated by Western Union and Wise.

Google hasn’t explained how the experience works for recipients. The Register will assume it doesn’t differ from Western Union’s options to have cash dispensed by an agent or deposited in a bank account, or Wise’s requirement to open an account. For Pay users, Western Union and Wise will appear as just another destination option.

Continue reading

Beijing twirls ban-hammer at 84 more apps it says need to stop slurping excess data

Online lending apps and more given fifteen days to ‘rectify’ behaviour

China’s Central Cyberspace Affairs Commission has named 84 apps it says breach local privacy laws and given their developers 15 days to “rectify” their code.

The Commission has posted two lists of apps it says need fixing, fast.

The first names 36 apps that breach user security by gathering and/or sharing more data than they need or doing so without users’ consent. Top of the list is web giant Tencent’s “mobile phone manager”, accused of harvesting and sharing more data than it needs. Most of the other apps on the first list share similar problems.

Continue reading

South Korea orders urgent review of energy infrastructure cybersecurity

No prizes for guessing why, as Colonial Pipeline outage stretches patience and looks like lasting a week

South Korea’s Ministry of Trade, Energy and Infrastructure has ordered a review of the cybersecurity preparedness of the nation’s energy infrastructure.

Minister of Trade, Industry and Energy Moon Seung-wook convened a meeting yesterday, saying it was needed considering the ransomware attack on the Colonial Pipeline that shuttered one of the USA’s main oil transport facilities.

“In the wake of the disruption, it is necessary to thoroughly examine whether cybersecurity preparations and countermeasures for our energy-related infrastructure are properly in place,” the minister said, before calling on operators of oil pipelines, power grids, gas pipelines, and emergency response systems to check the status of their systems and report back on their findings.

Continue reading

Salesforce fell over so hard today, it took out its own server status page

It’s not DNS. There is no way it’s DNS. It was DNS

Salesforce is digging itself out of a multi-hour outage right now that it has blamed on a DNS issue.

At one point today, the IT breakdown was so severe that its status page was pretty much inaccessible for netizens, and staff resorted to posting updates on their help and training sub-site.

"Salesforce is experiencing a major disruption due to what we believe is a DNS issue causing our service to be inaccessible," CTO Parker Harris said in a statement. "We recognize the significant impact on our customers and are actively working on resolution.

Continue reading

Tech industry quietly patches FragAttacks Wi-Fi flaws that leak data, weaken security

Dozen design, implementation blunders date back 24 years

A dozen Wi-Fi design and implementation flaws make it possible for miscreants to steal transmitted data and bypass firewalls to attack devices on home networks, according to security researcher Mathy Vanhoef.

On Tuesday, Vanhoef, a postdoctoral researcher in computer security at New York University Abu Dhabi, released a paper titled, "Fragment and Forge: Breaking Wi-Fi Through Frame Aggregation and Fragmentation" [PDF].

Scheduled to be presented later this year at the Usenix Security conference, the paper describes a set of wireless networking vulnerabilities, including three Wi-Fi design flaws and nine implementation flaws.

Continue reading

Microsoft says Outlook hit by 'email visibility issues' – as in, they're blank

Here's an unofficial fix for those who need their messages now

Microsoft says its Outlook desktop client is suffering serious “email visibility issues” today, with a fix yet to be rolled out. Users have reported either whole emails missing, chunks of data gone, or just seeing the first line of messages.

Folks can use the web or mobile client of Outlook, or the Windows desktop client in "safe mode." Otherwise, you're out of luck for the next few hours.

"We’re investigating an issue with email message visibility in Outlook. Outlook on the web appears to be unaffected," the Windows giant said a couple of hours ago.

Continue reading

WhatsApp: Share your data with Facebook, or we'll make our own app useless to you

Zuck gets tough just as Germany blocks privacy policy roll-out

WhatsApp users who refuse to accept its new privacy policy will slowly but surely be cut off from the chat app, the social network has confirmed.

In January, WhatsApp users were told if they wanted to keep using the software, they must agree to an updated fine print that, among other things, allows their data to be passed onto not only WhatsApp's parent Facebook but also its subsidiaries as and when decided by the tech giant.

This information includes names, profile pictures, status updates, phone numbers, contacts lists, and details about mobile devices and connections, though not the contents of encrypted messages and calls. Those who did not accept the terms and conditions would not be allowed to use the application from February.

Continue reading