Comment The stupidest thing I’ve ever said was “if it was a jet, the tower would have collapsed” on September 11th and I feel the same about RBS. As I pass it most days, part of me expects to see crowds outside, perhaps including the police and TV camera crews, because I can’t understand why it still functions.
The Reg has covered in depth the antics of its offshored systems but they’ve left both Reg readers and, for lack of a better word, RBS “management” with the wrong idea about the threats to it.
As a headhunter I hear just too many stories about the people they are hiring. OK, the outfit to which they’d outsourced HR/recruiting didn't help matters much, but the core problem is that bright people are extremely reluctant to work there, seeing the worst of two worlds where you have to work bank hours but with what are reportedly unappetizing pay levels.
There is some raw talent there since newish grads are being put in positions of responsibility that normally would have taken years to achieve, which in some ways is good … so long as you are hiring from them, rather than actually trusting RBS with your money in the wake of the mainframe cockup.
Ironically the recession that the chimps helped create is helping protect RBS for now because there are some very smart people still on board waiting for the right time to jump ship and the slowdown in banking means that has taken longer to appear. So the worst at RBS may be yet to come, when defections hit some critical level and juniors make mistakes when there’s no grownups left to bail them out.
Shafting Stephen Hester may have kept the arts graduates in Parliament and the BBC happy, but the RBS clusterfuck really wasn’t his fault. He arrived afterwards to help sort it out and screwing him out of his bonus then “letting him resign” sent a clear and bad message to everyone. You probably aren’t sympathetic to bankers, but unless the government introduces conscription to fight the tech/finance Vietnam at RBS, then you the taxpayer will own a huge bank that simply doesn’t work.
The rest of the City is also running risks with most of the consequences so far being kept within the banks themselves and since the regulators are far behind the curve, the pressure won’t really increase until something goes bad enough to get on TV.
Investment banking IT is quite different from the retail banks both in the types of technology and the complexity with which it is strung together. Although RBS retail banking has nightmarishly complex data flows, the systems are more homogenous, which means that a smart operator can deal with multiple types of issue because there are few types of basically different systems. An investment bank is not a single business like Sainsburys.
It’s lots of business units varying in size from six to a few hundred people and each has different tech needs, to the extent that there will be multiple networks, every major server O/S and more different applications than staff, with Excel/VBA playing a far larger part than any rational person would choose or even imagine. At RBS retail (or Sainsburys), it would be astonishing for a branch to hire some C++ programmers or to use FPGAs when x86 CPUs were far short of ideal for a purpose that they could not explain to central IT.
Gather round, children: I tell a tale of SmallTalk
There’s also industrial-scale grids, big data mining and some ancient legacy code as well. Not only Cobol, but SmallTalk, which for you younger people (I graduated in 1984) is an interesting bit of recent history. JP Morgan, for example, is critically dependent on it.
Unlike retail banks which haven’t innovated for decades, internet banks are constantly trying to find an angle and IT is whipped up to support it. Back when I ran a group that had to interface to JPM’s IT, I was shocked by the shabby state of its core technology, but they seem to have handled the SmallTalk issue better than most banks. But if you suddenly needed a SmallTalk programmer who understands banking, where would you get one?
My first direct experience of this was when I ran the group at the grand old age of 37. I was told “Dominic, you’re old” when I came into work and asked why my desk was surrounded by children in their 20s. I hobbled over to a desk, as befitted my age, and saw BTrieve error messages scrolling up the command Windows.
“Your indexes are screwed,” I said. The children were impressed. This was why I was the most expensive guy in the room.
“Can you fix it?”
I’d deduced that “NDX” meant indexes in the viciously terse Btrieve messages and remembered from my misspent youth that this was the most common problem, and one reason it wasn’t used any more.
Turns out that they’d fired the one guy who still understood it and to make it even more fun the children asked me to look at the source code. On a server. Somewhere. In one of the largest banks on the planet. Somewhere. The last developer of this stupidly useful system had connected to the server dynamically from the command line using “NET USE S:…” No one knew which one. This apparently was my problem and I lost great credibility by not being able to fix a system without source code that I’d barely been able to understand.
That leads us to the main vulnerability at investment banks. Their diversity increases their earnings (or makes their losses more newsworthy) but the number of things that go wrong goes up at least as quickly. But using so many different types of technology stacked on top of each other means that there are people whose main skill is holding them together. This is so much so that when I do careers counselling I often refer to the “Perl Pothole”, where an ace IT Pro finds himself jammed in a dead end, but doing a critical job pumping data from system A to system B.
This is an issue because banks have been managing down headcount, which is very different from managing down cost, so that IT managers have to make hard and risky decisions. It's now much harder to say “we’ve only got one person who understands the shredding of Bloomberg downloads” without getting a response of “so what’s the problem”?
Contractors will fill these holes, but no matter how sharp their skills the combinations and reasons will take time to master. I like to think of myself as someone who can wing any kind of programming, but when writing this I simply couldn’t think of anything to write about SmallTalk interfacing to Perl. Or for that matter Cobol. Can you?