Customers of cloud hoster VPS.net are having problems, after a routine SAN upgrade went wrong and forced a haphazard equipment migration in the company's Chicago data center.
Several days ago, a SAN within the company's Chicago facility started to have intermittent iSCSI connectivity, so VPS.net replaced its networking stack. This did not fix the problem, and the company was forced to swap out the SAN's networking card.
The SAN then went offline at 12:21am Pacific Time on Wednesday, leading to 10 hours (and counting) of problems for customers.
"We are in a process of moving customers with virtual machines on this SAN onto another, stable SAN," Steve Holford, marketing director of VPS.net's parent company UK2Group, told The Register via email. "This migration is currently underway... no customer data has been lost."
The outage comes after multiple reports of spotty performance at the company's "Chi-E" data center in February, along with a bevy of outages at its London, UK, and Utah, USA data centers.
The SAN upgrade was required due to degrading performance, and problems could go on for as long as a "a day or so," a VPS.net support engineer told The Register.
A vocal minority of VPS.net customers have experienced outages of three hours and even seven hours, according to posts on Twitter. Many have made complaints about VPS.net's customer service response times.
As of 10:34am Pacific Time, 51 per cent of customer virtual machines have been migrated to the new SAN, and VPS.net reckons it will take another three hours or so to get everyone back online.
It all goes to show that although cloud hosters like to present their services as fundamentally more advanced than on-premise IT, a routine equipment upgrade can lead to exactly the same type of cascading cock-up that happens on premise as well.
At the time of writing, VPS.net had not responded to various Register requests for more information about the root cause of its intermittent technical woes. ®