Rackspace runs short of Cloud Files storage in LON region

Rackspace? More like Lackspace as customers face upload and delete problems

Updated Rackspace is running out of, er, space. At least as far as a portion of its Cloud Files customers served by the LON (London) region are concerned.

According to the company, problems began on November 17 at 1545 UTC, as some customers experienced 503 errors when attempting to access their files.

On November 24 – yes, a week later – the company said its engineers increased storage capacity in the region. However, problems have continued this week. A Register reader, working for a SaaS biz that uses Rackspace's Cloud Files product, told us: "Since Monday we've been seeing a big jump in errors as they tried to upload files, and have had to delay a release because we can't publish assets to their CDN.

"To be clear, some operations succeed, but we're seeing between 1 and 6 percent fail in any given hour, and it's not getting any better."

By November 28, Rackspace told customers - in a communication seen by The Reg - that its engineers continued working and introducing additional storage "at a deliberate pace" to avoid worsening the situation. It said "customers should be able to access already uploaded content but may experience issues processing new uploads or delete actions."

As of November 29, more storage was added – and will continue to be added – to, in the words of Rackspace, "restore the environment's health."

The company said: "Engineers continue to recommend that customers retry upload and delete actions as needed as this is the only workaround currently available."

The embarrassing event is yet another reminder of the fallibility of cloud storage, although unlike the problems reported with Google Drive earlier this week, Rackspace has at least admitted that, yes, it isn't your imagination. Something has gone wrong.

However, the question remains – how did this happen? Storage needs are relatively easy to predict when paying customers are involved. Cloud Files itself is powered by OpenStack. Rackspace says of the product: "Whether your storage needs are modest - or monumental - you enjoy built-in redundancy."

Unless, it seems, the disk fills up.

The Register contacted Rackspace for comment on the incident and will update this piece should the company respond. ®

Updated at 16.32 UTC on December 1, 2023, to add:

A Rackspace spokesperson told us that no data has been lost in the incident and the issue was related to LON Cloud Files customers uploading new objects.

"Actions regarding existing objects are not affected by this issue."

As for how the company failed to plan for storage running out, the spokesperson added: "Reasonably foreseeable storage needs were forecasted and accounted for. High demand for Rackspace's Object Storage services significantly exceeded those forecasts."

Updated to add on December 6:

Rackspace since declared, in an update, that it had made a "series of changes" yesterday in response to what it described as a "usage-generated error spike on 4 December."

It said: "Engineers sped up the removal of expired objects to allow the environment to clear up space which reduces the error rates caused by over utilization.

"Engineers also made configuration changes to better utilize larger size drives which have now begun being introduced into the environment, and also continued the sourcing and provisioning of both a new cabinet of storage gear and additional drives. These are to replace the older drives currently in production. Further updates will be provided as the situation progresses."