In-house techies fixed faults before outsourced help even noticed they'd happened

60-minute SLA was effectively useless and the contractor admitted it

On Call Welcome to another instalment of On Call, The Register's weekly reader-contributed column that shares your stories of weird and wonderful tech support jobs.

This week, meet a reader we'll Regomize as "Callum" who sent us a story from his time working for a company in the north of England that ran an 8-CPU Sun server which used pairs of processors stored on removable cards.

"It was running the Oracle database that underpinned their sales platform, and was considered sufficiently critical that there was a five-figure leased-line connection to the support vendor who 'constantly monitored' the server for issues and was paid to send an engineer within an hour," Callum explained. A nearby cache of spare parts meant that replacement hardware would usually arrive before the support tech!

This arrangement was sensible because the server was old and occasionally unreliable.

"We would have issues from time to time, generally CPU cards having tantrums," Callum wrote. "But no problem, that's what the support contract was for!"

The arrangement also had two flaws.

One was that the designated contracted on call support lived sufficiently far away that the slightest bit of bad weather – and there's plenty of that in the north – meant road conditions became so bad he could not safely arrive within an hour as required.

The second was that the monitoring system wasn't very good at noticing when the servers went down but was excellent at detecting startups.

Callum told us those quirks meant incidents usually unfolded as follows:

1. A CPU card would experience a fault;
2. The server's OS responded to losing a quarter of its CPUs by rebooting;
3. The server would not reboot, because one of its CPU cards was broken;
4. The contact center would complain to IT;
5. Callum, or whichever other IT worker was on call, would drive in to remove the faulty card and reboot the server;
6. The server would resume operations;
7. The support contractor would call to report a server outage and promise to send someone within an hour.

After this sequence played out two or three times, Callum said they told the contracted IT support guy not to risk rushing in to fix the server.

"Most of the time we could get the box up and running ourselves, and in any case the world wasn't going to end if we couldn't sell stuff for a couple of hours," Callum wrote. "So we told him not to risk killing himself and/or others by attempting to get to us in an hour."

The support vendor came to the party, too, by waiving the cost of the leased line for two years in recognition that it wasn't doing its job!

Have you outpaced outsourced help? If so, hurry up and click here to send The Register an email so we can become your third-party narrator in a future edition of On Call! ®