Exclusive PlusNet has deleted hundreds of gigabytes of its customers' email during a storage update. The blunder also left about half its 140,000 punters unable to send and receive new email until this morning.
Data recovery engineers have been called in to try and retrieve the more than 700GB of data that was lost by the Sheffield-based ISP.
A key stage of implementing a new storage solution was set to go ahead at 8AM on Sunday. An engineer was in the process of bringing the new back-up storage server online, which meant wiping the disks.
An explanatory email from PlusNet told punters: "At the time of making this change the engineer had two management console sessions open – one to the backup storage system and one to live storage. These both have the same interface, and until Friday it was impossible to open more than one connection to any part of the storage system at once."
"The patches we installed on Friday evening removed this limitation, but unaware of this, the engineer made an incorrect presumption that the window he was working in was the back-up rather than the live server. Subsequently the command to reconfigure the disk pack and remove all data therein was made to the wrong server."
Whoops. Plusnet said: "We would like to make it clear that we are treating this as a critical service issue." Well, yeah.
They continued: "In line with the ISP industry we will not be offering compensation due to a break in service. Just as BT has never offered compensation for loss of broadband service."
PlusNet has a running account of the scramble to fix the calamity here.
Dissatisfied users at the official PlusNet customer forum have already launched a campaign to get the ISP put measures in place to ensure it doesn't delete their data or crock their email access again.
PlusNet would know by Friday evening whether the deleted emails are gone for good.®