Wells Fargo? Well fscked at the moment: Data center up in smoke, bank website, app down

Something something dark, billowing cloud computing: Massive multi-hour outage across US, and it's still down

Wells Fargo customers have been unable to access their online bank accounts for more than seven hours today – after smoke knackered one of its data centers.

Starting around 6am Pacific Time (2pm UTC), the American bank's online portal and its mobile application have been totally unusable. There are also reports of cards being rejected by cash machines and stores.

"We’re experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance," the bank said in a statement. "We’re working to restore services as soon as possible. We apologize for the inconvenience."

It added on Twitter: "We apologize to our customers who may be experiencing an issue with our online banking and mobile app. Thanks for your patience while we research this issue. If you are impacted, please check back here for updates."

The problems appear to have started early on Thursday morning, US West Coast time, and show no signs of abating. Customers at America's fourth largest bank are still, at time of writing, locked out of their accounts online.

According to fire chief Tim Boehlke, of Lake Johanna Fire Department, the downtime kicked off at a Wells Fargo data center in Shoreview, Minnesota. We're told a fire suppression system was activated at around 0500 local time (1100 UTC), forcing a power shutdown, and switching off the facilities' servers. When the fire department got there, though, they found no evidence of a blaze.

Fire suppression systems tend to be rather punishing to data center machines, particularly their hard drives, as sysadmins in this Reddit thread on the outage point out. It could take a while to restore power and undo the effects of the suppression system.

Whatever happened, it has clearly had an unfortunate knock-on effect. If there are any backup or redundant systems, they're not working. There are some rumors Wells Fargo's servers may be offline until Friday.

Whether the backups failed, or the shutdown caused a cascading fault that affected other data centers and took out the rest of the bank's online presence, isn't known at this time. It is baffling that a single incident has set off such a chain reaction that it knocked out the entire internet-facing infrastructure of the bank.

Wells Fargo did not respond to questions at time of publication. ®