So, that vast IT disaster you may have caused? Come in, sit down

You won't be forced to dig your own grave

I have a little sideline in interrogating IT professionals who are suspected of doing bad things.

Sometimes it is quite hard to objectively tell the difference between incompetence and malice. In fact it is rare that either are the root cause of the worst screw-ups. The most dangerous techie in your firm is not the disaffected sysadmin nor the under-performing developer but someone trying hard to do their job. The problems are being caused by him or her trying too hard.

Given my analysis of RBS's overnight transaction processing cock-up, it’s not likely they will ask me in to diagnose what happened, but this is what I’d do.

Ground rules

“Interrogation” is of course exactly the wrong word and if I’m dealing with a firm that has handled this sort of situation before, one that is utterly forbidden.

Often lawyers are involved by this point and they know that anything that can even be slightly represented as bullying can undermine the employer’s position big time and make a bad situation worse. Also if you're an employee in a position to cause the level of harm that justifies my high-but-excellent-value fees, losing you unnecessarily may well damage the business.

You will have political support either from the boss or colleagues and must be seen to be treated fairly. Of course I have no legal power to compel you to do anything. Since I’m far more polite in these meetings than I am in real life and not harassing you, any attempt to refuse to have a chat looks fatally bad.

I get to set some of the conditions, of which tea and biscuits are more important than some people realise. Ideally I’d be doing this in a bar, where the two of us relax, check each other out and get to an approximation of the truth that will allow a conclusion to be reached. That’s never going to happen, so the next best thing is to be a good host, preferably off-site with as few members of the firm’s management as I can engineer.

The exact nature of what happened often is an ingredient in a political spat. It’s not always clear to me what the client “wants” the answer to be, since it could well turn into ammunition in an internal fight. The killer is that this can be attached to a legal process where I may end up in court as a witness and no way am I lying under oath.

Please lie to me

It may seem odd to share some of my tricks with the almost 7 million people who read El Reg in any given month, given that I expect some of you to be involved. But they are integrity checks and I use “integrity” in both the ethical and database sense of the words.

I give the interviewee chances to lie to me. In my experience good techies are really uncomfortable with actually lying. They may skip details or exaggerate but very few can construct a consistent framework of untruths without the stress showing.

Trying to catch them out is the best fun I get with my clothes on: a multi-level puzzle with prizes as well as consequences. Either you know I’m a fellow geek because I’m nearly famous, or it comes as an unpleasant shock that the schmoozing City headhunter-type expresses “surprise” at the arcane details of your code or that you didn’t notice this particular flaw in the backup script.

You may be smarter than me - quite a lot of people are - but you have to be lucky all the time, and I need to catch you just once. You feeling lucky, punk?

A common defence, used by people who really ought to know better is that “it wasn’t me, it was someone using my ID”, which presumably sounds better in your head than out loud. In these days of CCTV, it rarely gives much protection. If you’re using that line, it’s hard not to believe you haven’t done something bad.

Lies make my job easier and make me look good to the people who have hired me. Let's be very clear here: they want a clear result. If I catch you in a lie then to the best of my ability I won’t react, but I recalibrate the other things you tell me in a very different light.

A provable lie means your bosses get that clear result. Your political support will not only vanish but your supporters will feel betrayed and turn on you in a way that rabid wolves would regard as harsh.


I don’t do forensics. Yes, I can do sector-level drive scans and know more about SQL server logs than is good for me, but if you are suspected of being very naughty I will refuse point blank to even touch your PC. Instead, I'll call up someone like Guidance Software to provide an evidence trail that can be used in court. Though, it usually doesn’t go that far.

I do need to look at what’s been going on, which does cause considerable discomfort since that means looking at the rather less polished parts of the operation - and is why I have serious non-disclosure agreements and no real specifics are in this article.

Why me?

I sometimes ask that myself. I certainly get a feeling of “there but for the grace of small gods go I” during this process. If I was a skeleton with a scythe I’d probably get a warmer reception from the rest of the team, who know any number of people in their firm who can evaluate what you did better than I can.

The problem is that they are conflicted and may be implicated themselves. A good boss will defend his staff, and your colleagues know that “shortcuts” are necessary to meet deadlines and resource constraints - as well as the standard defence of “everyone else was doing it” - which is both crap and true. Someone may want to get you out of a petty personality squabble that you see in any organisation, and it may be that someone is trying to shift blame. In total there’s not likely to be anyone inside the firm that can be seen to be entirely objective, so I get a call.

Next page: Judgement

Other stories you might like

  • Cloudflare's outage was human error. There's a way to make tech divinely forgive
    Don't push me 'cos I'm close to the edge. And the edge is safer if you can take a step back

    Opinion Edge is terribly trendy. Move cloudy workloads as close to the user as possible, the thinking goes, and latency goes down, as do core network and data center pressures. It's true  – until the routing sleight-of-hand breaks that diverts user requests from the site they think they're getting to the copies in the edge server. 

    If that happens, everything goes dark – as it did last week at Cloudflare, edge lords of large chunks of web content. It deployed a Border Gateway Protocol policy update, which promptly took against a new fancy-pants matrix routing system designed to improve reliability. Yeah. They know. 

    It took some time to fix, too, because in the words of those in the know, engineers "walked over each other's changes" as fresh frantic patches overwrote slightly staler frantic patches, taking out the good they'd done. You'd have thought Cloudflare of all people would be able to handle concepts of dirty data and cache consistency, but hey. They know that too. 

    Continue reading
  • You need to RTFM, but feel free to use your brain too
    But I was only following the procedures!

    Who, Me? Monday is here, and with it a warning that steadfast determination to ignore instructions might not be such a silly thing after all. Welcome to Who, Me?

    Today's story comes from a reader Regomized as "Sam" and takes us back to his first proper IT job following his departure from the education system.

    Sam found himself on the mainframe operations team for a telecommunications company. The work was, initially, pretty manual stuff. The telco wasn't silly, and had its new recruits start by performing offline duties, such as gathering tapes and job tickets for batch runs, handling payslips, "basically anything involving a bit of leg work," he told us.

    Continue reading
  • Tropical island paradise ponders tax-free 'Digital Nomad Visa'
    Live and work in Bali, pay tax at home

    The government of Indonesia has once again raised the idea of creating a "digital nomad visa" that would allow foreign workers to live and work in the tropical paradise of Bali, tax free, for five years.

    The idea was raised before the COVID-19 pandemic, but understandably shelved as borders closed and the prospect of any digital nomads showing up dropped to zero.

    But in recent interviews Sandiaga Uno, Indonesia's minister for Tourism and the Creative Economy, said the visa was back on the drawing board.

    Continue reading

Biting the hand that feeds IT © 1998–2022