Off-Prem

SaaS

Google Cloud (over)Run: How a free trial experiment ended with a $72,000 bill overnight

Billing budget? Free plan? All useless when buggy code went into overdrive


Sudeep Chauhan, founder of startup Milkie Way, suffered a bad case of bill shock when a test with a $7.00 billing budget and a free database plan on Google Cloud platform (GCP) generated a $72,000 invoice overnight.

"I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000," Chauhan wrote on his company's blog. "Super stressed, and not sure what happened, I clicked around, trying to figure out what was happening. I also started thinking of what may have happened, and how we could possibly pay the $5K bill. The problem was, every minute the bill kept going up. After two hours, it settled at a little short of $72,000."

It was especially surprising that it happened to Chauhan, who is ex-Google and even spent two years as a payments technical program manager. What happened?

The idea was to build a system that scraped web pages and stored the results in a database. His team picked Google Cloud Run, a GCP service that runs containers, for the job. They then found their code in each instance would timeout and stop as it scraped one page after the other. So, they set up a many-instance system that processed pages in parallel to get each page fetched and stored within the run-time limit.

Devs invited to bake 'Run on Google Cloud' button into git repos... By Google, of course

READ MORE

Chauhan wrote: "To overcome the timeout limitation, I suggested using POST requests (with URL as data) to send jobs to an instance, and [to] use multiple instances in parallel instead of using one instance serially. Because each instance in Cloud Run would only be scraping one page, it would never time out, process all pages in parallel (scale), and also be highly optimized because Cloud Run usage is accurate to milliseconds."

The ex-Googler reflected that he missed the possibility of pages that link back to each other, causing "infinite recursion." It should not have mattered too much, though: he set a billing budget of $7.00 and had a Firebase database on a free plan. "The worst case we imagined was exceeding the daily free Firestore limits," he said. Further, the credit card for the account had a spending limit of $100.

Unfortunately, a billing budget "does not automatically cap Google Cloud or Google Maps Platform usage/spending," according to the docs.

While Chauhan was asleep after a day of testing, Google sent an automated email informing him that his free Firebase plan had been "upgraded due to activity in Google Cloud," and that this "initiated billing" for the project.

He discovered multiple issues with the GCP cost controls. "Billing takes about a day to be synced, and that's why we noticed the charges the next day," Chauhan said. Next, the "Firebase Dashboard took more than 24 hours to update," he said. This meant that the dashboard showed usage within the daily limit, when it was, he said, "86 million percentage points" more than what was shown.

Billing takes about a day to be synced, and that's why we noticed the charges the next day

The GCP Cloud Run defaults also played their part. "The max-instances is preset to 1,000, and concurrency set to 80," he said. If he had corrected this to small values like 2 and 1, the bill shock would not have occurred.

Thanks to these settings, "running [out] this version of Hello World deployment on Cloud Run made 116 billion reads and 33 million writes to Firestore," said Chauhan.

Most of the cost was down to Firebase read operations, even at just $0.06 per 100,000. Multiply that by 116 billion and you get $69,600. There was also the small matter of 16,000 hours of Cloud Run Compute time, partly because the application did not delete the services but left them "in background process".

The performance of the buggy code was impressive in its way. "At the peak, Firebase was able to handle about one billion reads per minute," he said, while Cloud Run with concurrency "can handle 9 million requests per minute".

"Fail fast, learn fast with cloud is a bad idea," Chauhan concluded. "If you count the number of pages in GCP documentation, it's probably more than pages in [a] few novels. Understanding pricing, usage, is not only time consuming, but requires a deep understanding of how cloud services work."

There is a happy ending. "After going through our lengthy doc on this incident sharing our side of the story, various consults, talks, and internal discussions, Google let go of our bill as a one-time gesture," said Chauhan.

Such leniency cannot be relied upon. Auto-scaling and on-demand computing has downsides, and working out what something will cost is challenging. Caution is advised. ®

Send us news
115 Comments

Cloudflare slams AWS egress fees to convince web giant to join its discount data club

Lower your prices and play nicer, CDN goliath suggests

Cloudflare on Friday accused competitor Amazon Web Services of massive markups and hindering customer data portability, even as it invited the cloud services giant to join its discount data initiative known as the Bandwidth Alliance.

"AWS’s bandwidth pricing is bonkers," said CEO Matthew Prince, via Twitter. "And they stand alone in the industry not discounting when their customers send traffic to peered networks."

Prince and Nitin Rao, SVP of global infrastructure at Cloudflare, elaborated on that claim in a blog post that argues AWS is charging customers orders of magnitude more than its costs and makes a mockery of its parent company's mission statement that Amazon strives "to offer our customers the lowest possible prices…"

Continue reading

With Alphabet's legendary commitment to products, we can't wait to see what its robotics biz Intrinsic achieves

Google parent hopes to inject AI into factory machines

Alphabet today launched its latest tech startup, Intrinsic, which aims to build commercial software that will power industrial robots.

Intrinsic will focus on developing software control tools for industrial robots used in manufacturing, we're told. Its pitch is that the days of humans having to manually program and adjust a robot's every move are over, and that mechanical bots should be more autonomous and smart, thanks to advances in artificial intelligence and leaps in training techniques.

This could make robots easier to direct – give them a task, and they'll figure out the specifics – and more efficient – the AI can work out the best way to achieve its goal.

Continue reading

Google fixes 'Chromebork' one-character code typo that prevented Chrome OS logins

Programming blunder is the second such snafu this month

Bug of the week Google has fixed a bug in Chrome OS version 91.0.4472.165 that surfaced on Monday and prevented some users from being able to login to their systems.

Chrome OS downloads updates automatically but doesn't apply them until reboot, so only those who restarted their Chromebooks to ingest the force-fed broken update were affected.

Earlier this week, the internet titan on its Google Workplace status page said, "Our engineering team has identified an issue on Chrome OS 91.0.4472.165. The rollout of this version was halted."

Continue reading

Rackspace literally decimates workforce: One in ten staffers let go this week

85% of those jobs will be rehired, just in cheaper countries

Updated Around 10 per cent of Rackspace staff, predominantly in the US it seems, got an unwelcome email this week informing them they were being let go.

Not that the work they do isn't needed. In an paperwork submitted to the SEC on Wednesday, Rackspace disclosed that 85 per cent of the positions being cut will be backfilled by workers in "offshore service centers." That'll be where wages are lower and labor laws more lax, presumably.

"The rebalance in workforce is a component of a broader strategic review of the Company’s operations that is intended to more effectively align the Company’s resources with its business priorities in high growth areas," Rackspace said.

Continue reading

Punchy Italian kartist gets 15-year ban for trackside rampage... and other stories

An unexpectedly vehicular collection of chaos and confusion for your consideration

Welcome back for another compendium of tomfoolery from this week for those who enjoy a bit of light-hearted piffle. And let's face it, who doesn't?

Continue reading

Latest Windows 11 Preview a well-rounded update – literally

What else is round? Oh yes, holes

While the Windows of today may have more holes in it than a 20-year-old pair of underpants, Microsoft has continued plugging away at previews for the upcoming iteration, Windows 11.

Having got the excitement of integrated Teams chat out the way earlier this week, it was business as usual for build 22000.100, released to Dev Channel Insiders last night.

This week's modifications are all about soothing users whose nerves have likely been shredded by the recent arrival of HiveNightmare.

Continue reading

Apologetic Audacity rewrites privacy policy after 'significant lapse in communication'

Of course kids are allowed. Whatever gave you the impression they weren't?

Open-source audio editor Audacity this week posted an apology on GitHub in response to the entirely predictable furore over the platform's privacy policy.

An updated privacy policy accompanied the apology, in which the team insisted it had just been misunderstood, and that a look at the source would have shown its intentions.

"We are deeply sorry for the significant lapse in communication caused by the original privacy policy document," it said. The fact that it didn't regret the actual document itself seemed to alarm a good many users.

Continue reading

eBay cyberstalking victims sue internet tat bazaar over former staff members' campaign of harassment

We endured enormous cruelty and abuse and feared for our lives, say couple

A couple from the US who run a small ecommerce publication have launched legal action against eBay accusing the company of a "coordinated effort to intimidate, threaten to kill, torture, terrorize, stalk and silence" them to muzzle their coverage.

The allegations – made in a complaint lodged in the US District Court of Massachusetts this week – are the latest chapter in a long-running case that has already resulted in guilty pleas from a number of former employees in what has become known as the "eBay cyberstalking case".

Lawyers acting on behalf of the owners of EcommerceBytes – an online trade publication that covers the ecommerce industry run by journalists Ina and David Steiner - said the intimidation was so bad they were in fear for their lives.

Continue reading

Anyone fancy a Snowmobile full of Bags O'Crap? It'll be on the list somewhere

Reg reader reveals colossal 821-item collection of Amazon trademarks tucked away on its site

Recently, a Reg reader* contacted us at Vulture (virtual) Towers with something odd they'd found online – a page tucked away in the little-visited “Legal Policies” section of Amazon's website containing a "non-exhaustive" list of all the trademarks held by the company.

The list is massive: 821 trademarks, sorted alphabetically and listed entirely free of context or explanation.

On first glance, the contents can be baffling, or will induce flights of fancy as to their purpose. When simply plucked out of a list of plain-text words, the purpose of, say, "6PM", "BAG O'CRAP" or "MAD DOGS" are difficult to discern.

Continue reading

Subcontractors working on CityFibre's £45m Derby rollout threaten to 'rip up tarmac' in dispute over payments

Main contractor J McCann insists it takes its obligations 'very seriously'

Contractors helping to lay fibre cables under streets in Derby have threatened to scrap their work and "rip up tarmac" they've laid – unless they get paid.

A report by Construction Enquirer claims that subcontractors have also downed tools following the payment row.

The cables are being laid for digital infrastructure outfit CityFibre, which is spending £45m to install digital infrastructure in Derby.

Continue reading

Reserve Bank of India official suggests country may soon have a digital currency pilot

CBDC would be released in phases to prevent volatility

India may be launching a digital currency, an official from the Reserve Bank of India (RBI) said today.

Speaking at a panel discussion held by the Vidhi Centre for Legal Policy think tank, RBI deputy governor T Rabi Shankar described the potential Central Bank Digital Currencies (CBDCs) have for India, including smoother international transactions and protections from volatility.

Addressing whether CBDCs are needed in India, Shankar said: "It is important that all central banks get on the CBDC arrangements and coordinate effectively within themselves to actually maximize the immense potentials that CBDCs carry."

Continue reading