RBS Mainframe Meltdown: A year on, the fallout is still coming

When the totally brand new kit comes on ... what do you think will happen?


A year ago, RBS experienced its Chernobyl moment – an incident when a case of simple human error by those running critical systems resulted in a crisis.

IT staff badly botched routine maintenance of the IBM mainframe handling millions of customers' accounts – a system processing 20 million transactions a day. The mistake was compounded by their inability to recover the situation quickly enough.

The fallout saw up to 16.7 million customers at three banks in the group – RBS, NatWest and Ulster Bank – unable to access their money for four days.

RBS couldn’t hide and MPs monitoring the City pounced, demanding immediate answers from senior management on what went wrong with the bank's computers.

The press of middle England weighed in, too, pillorying the bank’s already unpopular chairman as he gave a grovelling apology to MPs for the whole episode.

A year on and the fallout is still landing as the Financial Conduct Authority (FCA) are deciding whether action is needed against RBS.

The bank is splashing out £450m on top of its £2bn annual IT spend to replace the mainframe that failed and on new backup. RBS told The Reg that it's instituting a “complete refresh of the mainframe” system in Edinburgh.

It’s an unprecedented move: RBS has had computer problems in the past but nothing has warranted a complete rip and replace of entire systems on this scale.

A year in, though, has RBS learned its lessons? Can throwing money at new hardware save customers from future problems? It’s still unclear that RBS has truly reversed course on its policy of wholesale outsourcing of IT jobs - a policy that helped ignite last year's crisis.

Then it's quite possible that what happened at RBS might be replicated elsewhere as old and overloaded mainframes like the one at RBS hold millions of accounts at other banks who’ve also sent their IT jobs overseas.

RBS is spending nearly half a billion to replace the system that failed in June 2012. The group is buying a new IBM mainframe and making “significant change to many of the systems to improve disaster recovery and automated error recovery” a bank spokesperson tells the Reg. “This is a result of the IT problems [last] June.”

One former RBS IT insider told us the usual procedure for outages at the bank:

“I've dealt with a few outages at RBS in the past. There will have been standard 'lessons learned' or 'drains up' type of investigation. The exact reason for the problem have been found and poured over in a tedious level of detail, then process will have been put in place to prevent the set of circumstances repeating. Often this is to tighten security and/or process, so for example you may have found individual user groups being tied down more or process documentation required to be more granular. Potentially more post-change reviewing to make sure that people did what they said they would.”

RBS faces a Herculean job in bringing online a new mainframe operating in a core part of its day-to-day business. It must plan and execute the job without interrupting the existing service by taking the old mainframe offline during the transition.

RBS did not say when it plans to bring the new mainframe online.

But hardware is only one thing: RBS must also determine what to do with the existing apps running on the system. Either it must port existing apps to the new system - which is likely - or write or buy new apps. If the former, RBS must design, write, test and then shift. If the latter, RBS must make sure the new apps work on the new mainframe and interoperate with other RBS’s other, connected systems.

RBS did not say whether the old CA-7 software would be ported to run on the new mainframe.

Our ex-RBS techie outlines the complexity of the challenge facing the RBS team making the switchover:

Being a bank everything is by nature hung together with schedules and batches, this is really the correct way of doing things. You make a product, make it stable, then if it needs to do something else you add another system and a dataflow … I once saw a diagram of all the dataflow in RBS, just the ones between major systems fairly impressively filled a projection screen, then all the minor systems were added and it was just a black screen!

It’s a measure of just how bad things were for RBS that it’s spending £450m to be more or less back where it started - on a mainframe, just a newer and, fingers crossed, more reliable mainframe. There have been reports of companies dumping mainframes, but the mainframe remains a standard for banks: 25 of the world’s top banks use mainframes from IBM, according to Gartner.

Another ex-RBS IT staffer told us:

“A lot of talk has been had in the news about how these systems are too complex and bound to fail, but I guarantee that re-writing the systems and making them monolithic programs would result in some serious pain and cost in the short to medium term and in the long term you'd just end up with the same satellite batch controlled systems when the appetite to change the central system runs out.”

Mainframes are embedded thanks to their history, starting out with the S/360 in the mid 1960s giving companies access to the kinds of fast computer power that had only been available to governments and academics building machines on a project-by-project basis. Over the years, IBM has extended and upgraded the family, through S/390 to the z-series.


Other stories you might like

  • China reveals its top five sources of online fraud
    'Brushing' tops the list, as quantity of forbidden content continue to rise

    China’s Ministry of Public Security has revealed the five most prevalent types of fraud perpetrated online or by phone.

    The e-commerce scam known as “brushing” topped the list and accounted for around a third of all internet fraud activity in China. Brushing sees victims lured into making payment for goods that may not be delivered, or are only delivered after buyers are asked to perform several other online tasks that may include downloading dodgy apps and/or establishing e-commerce profiles. Victims can find themselves being asked to pay more than the original price for goods, or denied promised rebates.

    Brushing has also seen e-commerce providers send victims small items they never ordered, using profiles victims did not create or control. Dodgy vendors use that tactic to then write themselves glowing product reviews that increase their visibility on marketplace platforms.

    Continue reading
  • Oracle really does owe HPE $3b after Supreme Court snub
    Appeal petition as doomed as the Itanic chips at the heart of decade-long drama

    The US Supreme Court on Monday declined to hear Oracle's appeal to overturn a ruling ordering the IT giant to pay $3 billion in damages for violating a decades-old contract agreement.

    In June 2011, back when HPE had not yet split from HP, the biz sued Oracle for refusing to add Itanium support to its database software. HP alleged Big Red had violated a contract agreement by not doing so, though Oracle claimed it explicitly refused requests to support Intel's Itanium processors at the time.

    A lengthy legal battle ensued. Oracle was ordered to cough up $3 billion in damages in a jury trial, and appealed the decision all the way to the highest judges in America. Now, the Supreme Court has declined its petition.

    Continue reading
  • Infusion of $3.5bn not enough to revive Terra's 'stablecoin'
    Estimated $42bn vanished with collapse of UST, Luna – we explain what all this means

    TerraUSD, a so-called "stablecoin," has seen its value drop from $1 apiece a week ago to about $0.09 on Monday, demonstrating not all that much stability.

    The cryptocurrency token, abbreviated UST, is supposed to be pegged to the price of the US dollar. Hence the "stable" terminology.

    But UST is not a "centralized stablecoin" that's exchangeable for a fiat currency; UST for USD (US dollars). Rather, it's a "decentralized stablecoin," meaning it can be exchanged for Luna (LUNA) tokens, another cryptocurrency tied to the Terra blockchain.

    Continue reading
  • DigitalOcean tries to take sting out of price hike with $4 VM
    Cloud biz says it is reacting to customer mix largely shifting from lone devs to SMBs

    DigitalOcean attempted to lessen the sting of higher prices this week by announcing a cut-rate instance aimed at developers and hobbyists.

    The $4-a-month droplet — what the infrastructure-as-a-service outfit calls its virtual machines — pairs a single virtual CPU with 512 MB of memory, 10 GB of SSD storage, and 500 GB a month in network bandwidth.

    The launch comes as DigitalOcean plans a sweeping price hike across much of its product portfolio, effective July 1. On the low-end, most instances will see pricing increase between $1 and $16 a month, but on the high-end, some products will see increases of as much as $120 in the case of DigitalOceans’ top-tier storage-optimized virtual machines.

    Continue reading
  • GPL legal battle: Vizio told by judge it will have to answer breach-of-contract claims
    Fine-print crucially deemed contractual agreement as well as copyright license in smartTV source-code case

    The Software Freedom Conservancy (SFC) has won a significant legal victory in its ongoing effort to force Vizio to publish the source code of its SmartCast TV software, which is said to contain GPLv2 and LGPLv2.1 copyleft-licensed components.

    SFC sued Vizio, claiming it was in breach of contract by failing to obey the terms of the GPLv2 and LGPLv2.1 licenses that require source code to be made public when certain conditions are met, and sought declaratory relief on behalf of Vizio TV owners. SFC wanted its breach-of-contract arguments to be heard by the Orange County Superior Court in California, though Vizio kicked the matter up to the district court level in central California where it hoped to avoid the contract issue and defend its corner using just federal copyright law.

    On Friday, Federal District Judge Josephine Staton sided with SFC and granted its motion to send its lawsuit back to superior court. To do so, Judge Staton had to decide whether or not the federal Copyright Act preempted the SFC's breach-of-contract allegations; in the end, she decided it didn't.

    Continue reading

Biting the hand that feeds IT © 1998–2022