This article is more than 1 year old

Fire, flood and vomit: Defeating the Great White Whale of Fail

Got a plan? Better get one quick

When to declare a crisis

Your BC process should be designed to ensure that where possible you declare a crisis before you're actually in the middle of one. Sometimes this isn't possible, of course: if someone whacks your building with a JCB and makes it unsafe then you'd be forgiven for not owning a crystal ball.

But what if, say, a core site goes down at 10pm on Saturday evening and the spare widget you keep on-site turns out to be faulty? You're not open on Sunday, so nothing's spoiling immediately, but if it's still down after 6am on Monday your customers will suffer.

Don't wait until 5am Monday to declare a crisis – make sure the process helps you do it early so that the essential bodies have as much time as possible to deal with the problem. And make sure your management sign up to the concept of nobody getting shot for invoking crisis management when things are smouldering instead of waiting for the inferno.

Testing the setup

You wouldn't test a new IT system before putting it live, would you? Neither should you run a BC regime without testing it. And just as you'd check your backups occasionally (particularly if you seldom had a real need to restore files from one) so you should test your BC processes and systems frequently to make sure they still work.

System testing should be a matter of routine. When you first put a system in you'll often be a bit nervous until you've failed it over and back a couple of times, but when you're comfortable it works you'll relax a bit.

I used to run various resilient firewall pairs around the world in a previous life, for instance, and once we'd confirmed everything was stable I had no qualms about forcing a failover from primary to secondary because everything was configured to a standard and we'd done enough tests to be confident that they just worked.

Do the same with your BC processes: test them at least annually, or if this would be too inconvenient or costly then at least test part of them each year (and do a different part each year so you cover everything over time). There are parts you can validate without business interruption (checking the cascade call-out list against the HR employee database, for instance, to ensure those on the list actually still work for you) so do so frequently.

BC is a business in its own right: there are companies out there that will work with you to write your processes for you, and will also come along and run entire BC simulations.

It's not a cheap option, but if you're serious about BC then give them strong consideration.

Simulations are bloody hard work but beneficial and enjoyable: in the simulation I mentioned earlier where the entire board was stuck overseas, we had an impending flood and a virulent virus, and then for good measure they threw in some other staff disasters and a potential PR disaster for good measure.

Next page: Conclusion

More about

TIP US OFF

Send us news


Other stories you might like