Surviving Hurricane Katrina: A sysadmin's epic DR (as in Didn't Realise) odyssey

25 days of refugee sex and guns in odd places

On-Call Jim Thompson got in touch with The Register about the mother of all On-Call stories, recalling the time he received a message asking him to come back to New Orleans because a storm called “Hurricane Katrina” was on its way and looked bad.

It's nearly 10 years since Katrina raged, so Jim kindly retrieved his jottings on the storm from his now-defunct LiveJournal blog. It's a long piece, but a richly rewarding read. You'll learn about extreme disaster recovery, what sex is like during a disaster, what not to take with you in an evacuation ... and much, much more.

Thanks for sharing, Jim, and over to you.

* * *

On Friday, 26 August 2005, I boarded a flight from New Orleans to Boise, Idaho. I had a job interview the following day. Hurricane Katrina was supposed to hit the Florida pan-handle sometime over the weekend.

On Saturday morning I was getting ready for my interview when I received a phone call from one of the officers of the company I worked for at the time. He asked me if I'd seen the weather channel recently and suggested that I “do whatever it is you do in this situation”.

After 30 seconds of watching the weather channel I turned on my laptop and VPN'd into the office. I executed a single shell script and logged off.

I was the only sysadmin working for this company and that was one of the reasons I was going to quit.

I tried calling the only other technical person that lived near the office, an officer of the company, let's call him Henry, with a sharp technical mind. No answer.

I called the airline to arrange a flight back to New Orleans. $1,700 later I had a one-way first class ticket to New Orleans with two layovers. If everything went according to plan I'd be home around 20:00 Central time.

I called my contact for the interview and told him I needed to cancel as I had some DR work to do back home. He offered to drive me to the airport and we did the interview on the drive over. He offered me the job on the spot. I told him I'd be in touch on Monday.

I never got a chance to speak with him again.

In 2004 Hurricane Ivan scared the hell out of me professionally. I evacuated for Ivan and spent my whole time in Memphis terrified because I had no DR.

After I came back I spent quite a bit of time trying to get a DR plan together and funded. It didn't work out. I did manage to work out a deal with a buddy of mine in Arizona where I got the company to pay for a DSL line at his house and let me send him a couple of servers.

I never gave this the attention it deserved, all I had running out of the remote location were secondary DNS and mail services. I also had some scripts to push what I thought were critical data on a daily/weekly basis. I'd never really thought it all the way through but it was better than what we had for Ivan.

In short, it was enough to let me sleep at night.