Updated Unlucky Salesforce customers have been unable to reach the service since 0956 PDT (1656 UTC) on Friday, thanks to a ham-handed database deployment.
Shortly thereafter, the cloud CRM biz said that it's looking into an issue linked to current or past users of its Pardot B2B marketing automation system.
It seems the US tech giant granted Pardot customers access privileges they should not have, which is to say access to everything and alter any data. As was observed on Reddit, "One of our projects had all its profiles modified to enable modify all, allowing all users access to all data."
To deal with the mess, Salesforce's IT team has denied all access to more than 100 cloud instances that host Pardot users, shutting out everyone else using those same systems, whether or not they were using Pardot.
"The deployment of a database script resulted in granting users broader data access than intended," Salesforce said in a note posted at 1033 PDT (1733 UTC). "To protect our customers, we have blocked access to all instances that contain affected customers until we can complete the removal of the inadvertent permissions in the affected customer orgs."
Salesforce says customers with no ties to Pardot may thus experience service disruption. The biz insists it's working to restore things as quickly as it can.
Social media of course has risen to the occasion with a litany of complaints. Some people report that their entire company has ended the week early and gone home on account of the cloud service outage.
Via Twitter, Salesforce CTO and co-founder Parker Harris apologized for the screwup.
To all of our @salesforce customers, please be aware that we are experiencing a major issue with our service and apologize for the impact it is having on you. Please know that we have all hands on this issue and are resolving as quickly as possible.— Parker Harris (@parkerharris) May 17, 2019
In a statement emailed to The Register, Balaji Parimi, CEO of security biz CloudKnox, cautioned that companies need to understand over-provisioned privileges represent a more likely threat than external attacks or insider threats.
"Security teams need to make sure that privileges with massive powers are restricted to a small number of properly trained personnel," he said. "Until companies better understand which identities have the privileges that can lead to these types of accidents and proactively manage those privileges to minimize their risk exposure, they’ll be vulnerable to devastating incidents like the one we’re seeing with Salesforce right now."
The Register asked Salesforce to comment but we've not heard back. No doubt they're rather busy at the moment. ®
Updated to add on May 18
Salesforce claims it was able to mostly restore access to services by 0104 PDT after 15 hours of disruption. Some organizations may still be locked out, though, and administrators may have to spend some time over the weekend repairing user account permissions, it seems.
In other words, it sounds as though Salesforce staff ran a script on a database that went wild and changed people's permissions to read-write all, then scrambled to remove all permissions and shut down instances to avoid data theft or tampering, and then brought back services and restored known administrators' access permissions allowing those admins to check or fix up their users' permissions as needed.
The biz said in an update:
We have restored administrators' access to all affected orgs as of 08:04 UTC. We have prepared a set of instructions for admins that may need guidance on how to manually restore those user permissions. We notified admins via an email that contained a link to the instructions.
A subset of admins may still be experiencing issues such as logging in to their orgs, modifying perms that are uneditable, or timeouts.
Come Monday morning, most customers' access has been restored though not everyone is back up to speed.