G Suite'n'sour: Google resets passwords after storing some unhashed creds for months, years

Biz app login details encrypted at rest, though, ad giant insists

Google admitted Tuesday its paid-for G Suite of cloudy apps aimed at businesses stored some user passwords in plaintext albeit in an encrypted form.

Administrators of accounts affected by the security blunder were warned via email that, in certain circumstances, passwords had not been hashed. Hashing is a standard industry practice that protects credentials by scrambling them using a one-way encryption algorithm.

Google was at pains to stress it was the enterprise non-consumer version of G Suite affected, that there were no signs of misuse of the passwords, and that the passwords were encrypted at rest on disk – though, we note, hashing them would have fully secured the sensitive info.

Before we get to the threat model part of this, there are essentially two security cockups at play here. The first involves a G Suite feature available from 2005 that allowed organizations' admins to set their G Suite users' passwords via the Google account admin console. That feature, designed for IT staff to help new colleagues set their passwords and log in, did not hash these passwords.

The second involves recording some user passwords in plaintext on disk, as they logged in, and keeping these unhashed credentials around for 14 days at a time, again encrypted at rest. This practice started in January this year, during attempts by Googlers to troubleshoot their login system, and has been stopped.

On the first issue, Suzanne Frey, Google veep of engineering and cloud trust, explained:

In our enterprise product, G Suite, we had previously provided domain administrators with tools to set and recover passwords because that was a common feature request. The tool (located in the admin console) allowed administrators to upload or manually set user passwords for their company’s users. The intent was to help them with onboarding new users; e.g., a new employee could receive their account information on their first day of work, and for account recovery. The functionality to recover passwords this way no longer exists.

We made an error when implementing this functionality back in 2005: The admin console stored a copy of the unhashed password. This practice did not live up to our standards. To be clear, these passwords remained in our secure encrypted infrastructure. This issue has been fixed and we have seen no evidence of improper access to or misuse of the affected passwords.

On the second issue, Frey continued:

In addition, as we were troubleshooting new G Suite customer sign-up flows, we discovered that starting in January 2019 we had inadvertently stored a subset of unhashed passwords in our secure encrypted infrastructure. These passwords were stored for a maximum of 14 days. This issue has been fixed and, again, we have seen no evidence of improper access to or misuse of the affected passwords. We will continue with our security audits to ensure this is an isolated incident.

Here's some examples of today's email alerts sent to G Suite admins:

Legacy functionality lead to some G Suite passwords being stored unhashed. Here is a sample of the email from Google pic.twitter.com/6z2dvHIsiE

— Thexyz (@thexyz) May 21, 2019

It looks like @gsuite have been storing unhashed passwords. Ooops! 😬

If you are a GSuite admin check your inbox!#GSuite pic.twitter.com/Prq9UzzvqF

— Paul Barton (@barton_paul) May 21, 2019

Passwords are hashed so that if someone breaks into, say, Google's servers, they can't usefully make off with people's login credentials: the passwords will be scrambled in such a way the miscreant can't figure out the originals and can't use them to log into other websites where the passwords are reused. If a hacker gets into Google's infrastructure, passwords hashed or not hashed, it's potentially game over anyway: it's possible they could access other sensitive info.

The passwords weren't hashed in this G Suite case, reminding us of Facebook, but were apparently stored encrypted at rest, meaning a hacker should not, in the best case scenario, be able to access them. However, it may be possible that a rogue staffer, or a skilled intruder, could still access the logged passwords as plaintext – they have to be decrypted at some stage by the ad giant's backend software to be used. This is why it's highly preferable to hash passwords, and then store them encrypted at rest, to be totally sure.

In other words, it's sloppy, dangerous, and embarrassing, though keep it all in perspective: to exploit this security blunder, an attacker would have to break into Google’s key networks, and obtain and exfiltrate the decrypted data, or subvert a staffer with enough seniority to decrypt the passwords, and in both cases, users would be severely screwed anyway. Hashing would protect the account passwords from snooping, sure, but, er, there would be rather bigger problems to solve: bad people on the network with all sorts of servers and data to potentially leaf through... until they trip one of many intrusion detection systems, of course.

From Wednesday, Google will begin changing passwords for affected accounts that have not already done so. So if you see things have changed, don’t panic. Just keep calm and carry on. ®