We sense a great disturbance in the Salesforce: File-slinging feature breaks down for more than 12 hours

It felt as if millions of SaaS customers suddenly cried out in frustration, then headed off for the weekend early

It has been a rough morning for Salesforce here in San Francisco: part of the cloud giant's sprawling empire fell over and stayed down for more than 12 hours. In fact, it stumbled over so hard, CEO Marc Benioff's techies are having to apply fixes to servers manually.

The SaaS titan told customers on Thursday night its Files service, used to store and share documents, was struggling worldwide to, well, read and write files. It's now Friday lunchtime, and some data remains inaccessible as engineers continue to roll out repairs.

"The technology team continues to manually update a subset of servers that need to have the fix implemented as well as test and validate the resynchronization process, which is the next phase of the resolution path," Salesforce told customers at 1030 PT on Friday.

"We continue to see a reduction in read/write alerts, and some customers are reporting that they can now read and write to new files.

"However, there is still a subset of customers being impacted by the issue. Files that were created after the incident began and before the fix was applied, however, may remain unavailable until resynchronization is complete."

The systems crashed on Thursday night at roughly 2330 PT. This meant just as Western Europe was beginning to wake up and start their Fridays, the Files part of Salesforce fell over.

This outage would carry over into the US East Coast and Midwest working day, and the morning for the West Coast where Salesforce is based.

By 1100 PT, the cloud giant said it was nearly through the tedious process of hand-fixing servers to address the problem.

"The technology team now has less than 5 per cent of the affected servers to manually apply the fix to, and the resynchronization process is being implemented to a subset of production hosts," the biz explained. "If the resynchronization completes successfully on those hosts, the team will then start the process of implementing resynchronization across the rest of the affected environment."

As we were preparing to publish this piece, shortly after 1300 PT, Salesforce posted the following revised update, indicating the outage is ongoing as is the repair work:

The technology team continues to manually apply the fix to a subset of the affected servers in three of the impacted data centers. All servers in the other five affected data centers have had the fix implemented.

The synchronization process is the next phase of the resolution path that will ensure files that were created after the incident began and before the fix was applied are once again available to customers. That process was initiated within the first of the eight data centers, and on successful completion of that activity and after the technology team validates the implementation, we will then initiate that process across the remaining data centers.

Inconvenient as it is, Friday's problem pales in comparison to the nightmare outage in May that caused portions of the service to go down over a span of three days.

Hopefully, most of those affected were able to find an alternate way to move files and, barring that, at least head out for an early start to the weekend. ®

Send us news

Europe's largest caravan club admits wide array of personal data potentially accessed

Experts also put an end to social media security updates

Oxide reimagines private cloud as... a 2,500-pound blade server?

Rackscale system can be had with up to 2,048 cores, 32TB of RAM, and nearly a petabyte of flash

Cloudflare joins the 'we found ways to run our kit for longer' club

Finds modest savings, but isn't modest about ability to land big customers or sell AI

Arm share price bulges after AI pumped revenue to a new record

Smartphones surge, but chip design champ isn't ready to predict its cloudy future

Alibaba Cloud posts modest growth, mostly thanks to other Alibaba business units

Customers from beyond the Chinese giant are being let go if they've signed for low-margin contracts

Bank boss has pay slashed after presiding over tech outages in 2023

Singapore's DBS wants accountability for digital dropouts

Restrictive licensing keeps businesses grounded in cloud vendor vortex

Microsoft named – again – by EU trade groups seeking to level the playing field

Amazon extends the life of its servers to six years, expects $900m benefit in 90 days

Cloud optimization efforts ebb, and migrations resume

Amid Broadcom's subscription push, VMware killed a SaaS product

Another 50-plus products also binned in move to big bundles

Alphabet just banked $3B by stretching life of its servers

Q4 results reveal AI is the answer to – or the reason for – everything: a cloud profit, wobbly ads, boosting subscriptions

Snow day in corporate world thanks to another frustrating Microsoft Teams outage

Network rollback fails to resolve issue in Americas as Redmond scrambles to optimize its way out of the problem

Japanese government finally bids sayonara to the 3.5" floppy disk

Businesses can at long last submit digital docs to government agencies