Never trust a developer who says 'I can fix this in a few minutes'

And never assume a project manager can calm a cranky client

On-call In this week's instalment of On-call, our weekend thing in which we share readers' tales of odd things that happen at odd hours, we bring you the adventures of “Foredeck” who once “worked for a small UK ISV ... as a Support Manager, which meant I had a lot of different hats to wear, including being on-call one week in three".

“My most memorable call out (and there were lots) actually happened when I wasn't supposed to be on-call,” recalls Foredeck.

“I was on the West Coast of the US working with a customer for several weeks to stand-up a new system for ~20,000 users and got back to the hotel at about 18:00 on Friday looking forward to a quiet beer.”

“Just as I was starting to relax, my phone went and when I answered one of our UK project managers was on the line.”

Said project manager was a bit panicked “because the go-live of a new release for one of our customers in the UK had gone badly pear-shaped".

The customer was a TV station, so was rather sensitive to delays.

“Our change windows and SLAs were really aggressive; we'd blown all those limits some time before and the whole development team were still at work (2:00 AM in the UK) trying to fix it whilst the main customer exec was demanding half-hourly update calls and generally shouting.”

Foredeck points out that it was a little hard to do much about it from 9,000 miles away, but did volunteer to phone the customer, explain that the situation was going nowhere much and that, since they were now off-air anyway, there was no point keeping an exhausted team at work any longer.

“In the morning (9:00AM UK, 01:00AM for me – there went my evening!) they were able to get things running again. The development team spent the next week analysing the problem and the roll-out the next weekend was successful.”

“In the wash-up afterwards my main question was why the PM had not just gone for a back-out when it went wrong? That would have restored service, albeit on the old version, well within the change window ... Her answer was that the customer had been shouting, and developers had said they 'thought they could fix it'!”

“Lesson learned: never believe a developer when he says 'I can fix this in a few minutes' and never rely on a project manager to handle a customer fire-storm.”

What have you experienced or learned when working on-call? Drop me a line here and you may well end up in your very own On-call story. One to frame for the grandkids, eh? ®