Europe's largest online storage provider, Humyo, has been offline all afternoon due to a multi-terabyte database rebuild.
Humyo is an online, "store-your-data-in-the-cloud" service that was started up by Dan Conlon in January 2007. It has more than 300,000 customers using its facilities.
Conlon funded it himself using money he made when sold his hosting company, DonHost, to Pipex in a £5.9m deal in September 2004. Conlon then spent two years as MD of Pipex Hosting before leaving to start something up again. A few days ago the company announced a $1.15m investment from other investors to fund expansion from its current operations in the UK and Germany.
Users see Humyo as just another drive to which they can drag and drop files. Unfortunately, when it isn't there, neither is their data. Although Humyo encrypts user data and keeps it in a data centre in a former bullion vault that's all very well, the vital thing is that the bloody data centre is available, 24 by 7, all the time, every second of every day, just like a local hard drive. When it isn't, then the whole cloud storage proposition goes tits up in a major way.
Dan Conlon said the outage was due to "An incorrect set of database queries input as part of a software release. A table in the database was dropped and it replicated through the database server. We had to restore from last night's backup."
The good news is that the backup is present and correct and no user will lose any data. The bad news is that, at 5.30pm, it will take another 60 to 120 minutes to complete because the restore software has to roll the log file forward and there are multiple terabytes of data involved in the SQL database.
Conlon said: "It's unfortunate and we regret the inconvenience to our customers. This is the first outage since the company was formed and we very much regret this."
A possible fix might be to have a third database server running a few minutes behind the others and this could be used to restore from if a similar problem happened in the future, radically reducing the restoration time needed.
A user said of the outage: "It's certainly annoying me as a paid subscriber, half of my work data is on there." Accesses to the Humyo website get this response: "We are currently running a system maintenance check, please check back shortly. We apologise for any inconvenience this may cause."
Ho ho, ho ho, Humyo no go, ho hum, ho hum, it's time to twiddle my thumb. ®