Off-Prem

SaaS

Google Cloud (over)Run: How a free trial experiment ended with a $72,000 bill overnight

Billing budget? Free plan? All useless when buggy code went into overdrive


Sudeep Chauhan, founder of startup Milkie Way, suffered a bad case of bill shock when a test with a $7.00 billing budget and a free database plan on Google Cloud platform (GCP) generated a $72,000 invoice overnight.

"I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000," Chauhan wrote on his company's blog. "Super stressed, and not sure what happened, I clicked around, trying to figure out what was happening. I also started thinking of what may have happened, and how we could possibly pay the $5K bill. The problem was, every minute the bill kept going up. After two hours, it settled at a little short of $72,000."

It was especially surprising that it happened to Chauhan, who is ex-Google and even spent two years as a payments technical program manager. What happened?

The idea was to build a system that scraped web pages and stored the results in a database. His team picked Google Cloud Run, a GCP service that runs containers, for the job. They then found their code in each instance would timeout and stop as it scraped one page after the other. So, they set up a many-instance system that processed pages in parallel to get each page fetched and stored within the run-time limit.

Devs invited to bake 'Run on Google Cloud' button into git repos... By Google, of course

READ MORE

Chauhan wrote: "To overcome the timeout limitation, I suggested using POST requests (with URL as data) to send jobs to an instance, and [to] use multiple instances in parallel instead of using one instance serially. Because each instance in Cloud Run would only be scraping one page, it would never time out, process all pages in parallel (scale), and also be highly optimized because Cloud Run usage is accurate to milliseconds."

The ex-Googler reflected that he missed the possibility of pages that link back to each other, causing "infinite recursion." It should not have mattered too much, though: he set a billing budget of $7.00 and had a Firebase database on a free plan. "The worst case we imagined was exceeding the daily free Firestore limits," he said. Further, the credit card for the account had a spending limit of $100.

Unfortunately, a billing budget "does not automatically cap Google Cloud or Google Maps Platform usage/spending," according to the docs.

While Chauhan was asleep after a day of testing, Google sent an automated email informing him that his free Firebase plan had been "upgraded due to activity in Google Cloud," and that this "initiated billing" for the project.

He discovered multiple issues with the GCP cost controls. "Billing takes about a day to be synced, and that's why we noticed the charges the next day," Chauhan said. Next, the "Firebase Dashboard took more than 24 hours to update," he said. This meant that the dashboard showed usage within the daily limit, when it was, he said, "86 million percentage points" more than what was shown.

Billing takes about a day to be synced, and that's why we noticed the charges the next day

The GCP Cloud Run defaults also played their part. "The max-instances is preset to 1,000, and concurrency set to 80," he said. If he had corrected this to small values like 2 and 1, the bill shock would not have occurred.

Thanks to these settings, "running [out] this version of Hello World deployment on Cloud Run made 116 billion reads and 33 million writes to Firestore," said Chauhan.

Most of the cost was down to Firebase read operations, even at just $0.06 per 100,000. Multiply that by 116 billion and you get $69,600. There was also the small matter of 16,000 hours of Cloud Run Compute time, partly because the application did not delete the services but left them "in background process".

The performance of the buggy code was impressive in its way. "At the peak, Firebase was able to handle about one billion reads per minute," he said, while Cloud Run with concurrency "can handle 9 million requests per minute".

"Fail fast, learn fast with cloud is a bad idea," Chauhan concluded. "If you count the number of pages in GCP documentation, it's probably more than pages in [a] few novels. Understanding pricing, usage, is not only time consuming, but requires a deep understanding of how cloud services work."

There is a happy ending. "After going through our lengthy doc on this incident sharing our side of the story, various consults, talks, and internal discussions, Google let go of our bill as a one-time gesture," said Chauhan.

Such leniency cannot be relied upon. Auto-scaling and on-demand computing has downsides, and working out what something will cost is challenging. Caution is advised. ®

Send us news
115 Comments

Japanese bloke collared after using AI software to uncensor smut and flogging it

Plus: Explore the limits of language models in bizarre research experiment, and more

In brief A man was detained in Japan for selling uncensored pornographic content that he had, in a way, depixelated using machine-learning tools.

Masayuki Nakamoto, 43, was said to have made about 11 million yen ($96,000) from peddling over 10,000 processed porn clips, and was formally accused of selling ten hardcore photos for 2,300 yen ($20). He pleaded guilty to violating Japan's copyright and obscenity laws, NHK reported this month.

Explicit images of genitalia are forbidden in Japan, and as such its porn is partially pixelated. Don't pretend you don't know what we're talking about. Nakamato flouted these rules by downloading smutty photos and videos, and reportedly used deepfake technology to generate fake private parts in place of the pixelation.

Continue reading

Florida man accused of breaking Mastodon's open-source license with botched social network launch

Golf enthusiast given 30 days to cough up code

A Florida man has been accused of breaking the copyleft license of Mastodon by running an online instance of the software without providing its source code as required.

And not only that, the real-estate baron and wannabe tech tycoon has been told he has a month to fall in line with the fine print or put himself potentially at risk of further action.

Mastodon is a Twitter-like microblogging service that you host yourself. Servers running this software can form a larger, decentralized social network.

Continue reading

Antitrust battle latest: Google, Facebook 'colluded' to smash Apple's privacy protections

Amended Texas complaint alleges backroom efforts to maintain ad dominance and more

Several years ago, to deal with the competitive threat of header bidding – a way for multiple ad exchanges to get a fair shot at winning an automated auction for ad space – Google allegedly hatched a plan called "Jedi" to ensure that its ad exchange always won.

And in 2017, after Facebook announced plans to support header bidding, Google, it's claimed, struck a deal with Facebook – dubbed "Jedi Blue" – in which the two internet behemoths would "work together to identify users using Apple products," and set up "quotas for how often Facebook would win publishers’ auctions."

The Jedi project is described in an amended complaint, filed Friday, that expands the December 2020 antitrust claim against Google, brought by Texas, 14 other US states, and the Commonwealth of Puerto Rico.

Continue reading

Microsoft under fire again from open-source .NET devs: Hot Reload feature pulled for sake of Visual Studio sales

Windows giant has a funny way of 'loving' Free software

Updated Microsoft has enraged the open-source .NET community by removing flagship functionality from open-source .NET to bolster the appeal of Visual Studio, not least against its cross-platform cousin Visual Studio Code.

The two key pieces in this latest unrest are this pull request in the open-source .NET SDK repository on GitHub, in which 2,500 lines of code implementing a feature called Hot Reload are removed from a tool called dotnet watch; and this blog post in which Principal Program Manager Dmitry Lyalin revealed "we’ve decided that starting with the upcoming .NET 6 GA release, we will enable Hot Reload functionality only through Visual Studio 2022."

Hot Reload is a feature whereby developers can modify source code while an application is running, apply the changes, and see the results in the running application. It speeds the development process because it is quicker than rebuilding the code, stopping the application, applying the changes, and then firing it up again.

Continue reading

It's 'near-impossible to escape persistent surveillance' by American ISPs, says FTC

Watchdog finds dubious data gathering, illusory solicitations for consent

The US Federal Trade Commission on Thursday said many internet service providers are sharing data about their customers, in defiance of expectations, and are failing to give subscribers adequate choices about whether or how their data is shared.

The trade watchdog's findings arrived in the form of a report [PDF] undertaken in 2019 to examine the data and privacy practices of major US broadband providers, including AT&T Mobility, Charter Communications, Google Fiber, T-Mobile US, Verizon Wireless, and Comcast's Xfinity.

"[T]hese findings underscore deficiencies of the 'notice-and-consent' framework for privacy, especially in markets where users face highly limited choices among service providers," said FTC boss Lina Khan in a statement [PDF].

Continue reading

While the iPhone's repairability is in the toilet, at least the Apple Watch 7 is as fixable as the previous model

Component swaps still a thing – for now

Apple's seventh-gen Watch has managed to maintain its iFixit repairability rating on a par with the last model – unlike its smartphone sibling.

The iFixit team found the slightly larger display of the latest Apple Watch a boon for removal via heat and a suction handle. Where the previous generation required a pair of flex folds in its display, the new version turned out to be simpler, with just the one flex.

Things are also slightly different within the watch itself. Apple's diagnostic port has gone and the battery is larger. That equates to a slight increase in power (1.094Wh from 1.024Wh between 40mm S6 and 41mm S7) which, when paired with the slightly hungrier display, means battery life is pretty much unchanged.

Continue reading

Better late than never: Microsoft rolls out a public preview of E2EE in Teams calls

Only for one-to-one voice and video, mind

Microsoft has finally kicked off the rollout of end-to-end-encryption (E2EE) in its Teams collaboration platform with a public preview of E2EE for one-to-one calls.

It has been a while coming. The company made the promise of E2EE for some one-to-one Teams calls at its virtual Ignite shindig in March this year (https://www.theregister.com/2021/03/03/microsoft_ups_security/) and as 2021 nears its end appears to have delivered, in preview form at least.

The company's rival in the conference calling space, Zoom, added E2EE for all a year ago, making Microsoft rather late to the privacy party. COO at Matrix-based communications and collaboration app Element, Amandine Le Pape, told The Register that the preview, although welcome, was "long overdue."

Continue reading

Recycled Cobalt Strike key pairs show many crooks are using same cloned installation

Researcher spots RSA tell-tale lurking in plain sight on VirusTotal

Around 1,500 Cobalt Strike beacons uploaded to VirusTotal were reusing the same RSA keys from a cracked version of the software, according to a security researcher who pored through the malware repository.

The discovery could make blue teams' lives easier by giving them a clue about whether or not Cobalt Strike traffic across their networks is a real threat or an action by an authorised red team carrying out a penetration test.

Didier Stevens, the researcher with Belgian infosec firm NVISO who discovered that private Cobalt Strike keys are being widely reused by criminals, told The Register: "While fingerprinting Cobalt Strike servers on the internet, we noticed that some public keys appeared often. The fact that there is a reuse of public keys means that there is a reuse of private keys too: a public key and a private key are linked to each other."

Continue reading

Microsoft investor urges shareholders to vote for a deep dive into pay gap and harassment policies

More transparency and reporting needed, says Arunja Capital

Updated Accusations of harassment and concerns over pay gaps continue to dog Microsoft as shareholders were urged by investor Arunja Capital to vote for the software giant to release transparency reports.

The form PX14A6G filed with the US Securities and Exchange Commission is sent to shareholders outlining why the sender wants them to vote a certain way. In this case, for a proposal to release a transparency report regarding the effectiveness of Microsoft's workplace sexual harassment policies, and another to have the company report on median pay gaps across race and gender.

Investment management firm Arunja Capital put forward the resolution earlier this year, stating: "Microsoft needs independent and transparent investigation of gender discrimination, [and alleged] sexual harassment by former CEO Gates and others."

Continue reading

US drops tariff threat against nations who dished out digital taxes to American tech giants as OECD members hash out new deal

15% tax minimum to hit tech firms

The US government and administrations in Europe have come to an agreement that will drop the threat of tariffs in response to policies on digital services taxes (DSTs).

The Department of the Treasury announced the deal would mean Austria, France, Italy, Spain, and the United Kingdom could keep their DSTs while multinational rules negotiated with the Organization for Economic Co-operation and Development (OECD) were introduced.

Back in June, nations where digital services taxes were deemed to disproportionately affect the US tech industry were threatened with 25 per cent tariffs on up to $2bn of their goods by the Office of the US Trade Representative (USTR). At the same time, it announced an immediate 180-day suspension of the tariffs so G20 and OECD nations had time to complete their negotiations on a global tax law, which would also hit tech companies.

Continue reading

Nobody cares about DAB radio – so let's force it onto smart speakers, suggests UK govt review

Britain's anti Amazon and Google war gets a second front

The UK may require smart speakers such as Amazon Echo and Google Home devices to broadcast UK DAB radio stations, over government fears that Brits aren't consuming enough of the unloved radio tech.

Under the guise of "protecting UK radio stations' accessibility" the Department for Culture, Media and Sport (DCMS) has published a report calling for smart speakers to rebroadcast domestic radio stations' output. The recommendation is as follows:

Continue reading