Data Centre

Cloud

We sense a great disturbance in the Salesforce: File-slinging feature breaks down for more than 12 hours

It felt as if millions of SaaS customers suddenly cried out in frustration, then headed off for the weekend early

6 Got Tips?

It has been a rough morning for Salesforce here in San Francisco: part of the cloud giant's sprawling empire fell over and stayed down for more than 12 hours. In fact, it stumbled over so hard, CEO Marc Benioff's techies are having to apply fixes to servers manually.

The SaaS titan told customers on Thursday night its Files service, used to store and share documents, was struggling worldwide to, well, read and write files. It's now Friday lunchtime, and some data remains inaccessible as engineers continue to roll out repairs.

"The technology team continues to manually update a subset of servers that need to have the fix implemented as well as test and validate the resynchronization process, which is the next phase of the resolution path," Salesforce told customers at 1030 PT on Friday.

"We continue to see a reduction in read/write alerts, and some customers are reporting that they can now read and write to new files.

"However, there is still a subset of customers being impacted by the issue. Files that were created after the incident began and before the fix was applied, however, may remain unavailable until resynchronization is complete."

The systems crashed on Thursday night at roughly 2330 PT. This meant just as Western Europe was beginning to wake up and start their Fridays, the Files part of Salesforce fell over.

This outage would carry over into the US East Coast and Midwest working day, and the morning for the West Coast where Salesforce is based.

By 1100 PT, the cloud giant said it was nearly through the tedious process of hand-fixing servers to address the problem.

"The technology team now has less than 5 per cent of the affected servers to manually apply the fix to, and the resynchronization process is being implemented to a subset of production hosts," the biz explained. "If the resynchronization completes successfully on those hosts, the team will then start the process of implementing resynchronization across the rest of the affected environment."

As we were preparing to publish this piece, shortly after 1300 PT, Salesforce posted the following revised update, indicating the outage is ongoing as is the repair work:

The technology team continues to manually apply the fix to a subset of the affected servers in three of the impacted data centers. All servers in the other five affected data centers have had the fix implemented.

The synchronization process is the next phase of the resolution path that will ensure files that were created after the incident began and before the fix was applied are once again available to customers. That process was initiated within the first of the eight data centers, and on successful completion of that activity and after the technology team validates the implementation, we will then initiate that process across the remaining data centers.

Inconvenient as it is, Friday's problem pales in comparison to the nightmare outage in May that caused portions of the service to go down over a span of three days.

Hopefully, most of those affected were able to find an alternate way to move files and, barring that, at least head out for an early start to the weekend. ®

Sign up to our NewsletterGet IT in your inbox daily

6 Comments

Keep Reading

Contact-tracing is basically CRM so we think we've got it sorted, says Salesforce

Coming real soon now as-a-service. Details on data collection and price? Coming soon, promise

Pipped to the post: Google Cloud nabs Salesforce exec to lead UK and Ireland business

Pip White among bunch of new EMEA hires at Chocolate Factory offshoot

Salesforce publishes self-themed activity book to keep your kids ‘Appy

A new use for App Exchange, with that slightly greasy Ronald McDonald vibe

Salesforce plans to write Slack integration out of the equation by rolling its own messaging and collaboration app

Salesforce Anywhere beta due to drop in July, full release in Q4

Uber turns SaaS vendor with deal that bakes public transport authority into its app

Given the scoundrels in the world of software, could Uber be worse?

UK housing association Places for People hands £21m to Salesforce to look after CRM and job scheduling

Tender loving care

Salesforce's revolving door spins amid shopping spree: Co-CEO out, BT alumnus G-Patz up, Vlocity slurped

Plus: World Tour Sydney event goes online only over coronavirus

Salesforce founder Marc Benioff tossed $2.3m in return for bumper company growth

Enterprise cloud rains bonus down on software supremo... that's this year's holiday sorted then

CIOs will force SaaS vendors to limber up and get more flexible about contracts in the post-pandemic world

No more paying for non-existent employees, users, page views, or transactions, says Forrester

UK.gov confirms: Yes, our former DWP perm sec will join Salesforce

No lobbying for at least another five months

Tech Resources

Islamic Republic of Iran Cybersecurity Profile from Anomali Threat Research

Iran’s regional influence has been operationalized through the use of proxy military activity, information operations as well as offensive cyber activity.

Unlocking the Cloud-Native Data Layer

Being able to exceed customer expectations is essential to a successful business.

The Evolution of Ransomware

There’s no question that ransomware is a growing threat.

Dark Reading Report: The State of IT Operations & Cybersecurity Operations

Your enterprise’s cyber risk may depend upon the relationship between the IT team and the security team.