Test Systems Better, IBM tells UK IT meltdown bank TSB

Hired to assess migration chaos, Big Blue suggests lack of rigorous go-live criteria

Updated A report into the IT meltdown at TSB has suggested the British bank did not carry out rigorous enough testing and that the problems went beyond previously reported middleware issues.

The chaos at the bank, a subsidiary of the Spanish Sabadell Group, saw many customers unable to access services for a week at the end of April after the bank bodged a long-planned migration off its former parent firm Lloyds Banking Group's systems. The IT issues were compounded by a wave of scams and an underwhelming response from execs.

Amid the crisis, the bank hired IBM in a systems integration role to identify and resolve the problems – which the bank's CEO Paul Pester told MPs was due to issues with its middleware systems.

Big Blue produced a short presentation for the bank four days after being appointed, offering a "preliminary work plan with very early hypotheses" – and the Treasury Committee has today published the slide deck.

In it, IBM suggested that the bank's testing was not up to scratch, saying it "has not seen evidence of the application of a rigorous set of go-live criteria to prove production readiness".

Emphasising the scale and complexity of the project, IBM said that a firm would need "longer than normal to prove the platform through incremental customer take-on to observe and mitigate any operational risks" – and warned that such projects bring a broad range of hard-to-diagnose technical and functional problems.

"To address this risk profile, IBM would expect world class design rigour, test discipline, comprehensive operational proving, cut-over trial runs and operational support set-up," it said.

However, a set of bullet points suggest that this was not the case – or that TSB was not able to demonstrate this to IBM.

"Performance testing did not provide the required evidence of capacity and the lack of active-active test environments have materialised risk due to issues with global load balancing (GLB) across data centres," IBM stated.

It said that a "limited number of services" – including mortgage origination and ATM and head office functions – had been launched on the new platform and a broader set of services to about 2,000 TSB partners.

The integrator added that it "has not seen evidence of technical information available to TSB", such as architectures, configuration and design documents, test outcomes or monitoring information.

A statement in italics at the bottom of this slide in an apparent attempt to contrast the situation at TSB with IBM's own work:

In a similar situation when IBM partnered with a financial organisation to migrate to a new core banking platform, multiple trial migrations were conducted, rolled back and then remediated prior to launch.

The production launch was done over a longer period, initially open to programme members only, then staff, then targeted customer groups, before full launch to new customers and subsequent migration.

from the great gatsby pic copyright: warner brothers

UK 'meltdown' bank TSB's owner: Our IT migration was a 'success'


The assessment – or, rather, its release into the public domain – adds to the pressure Pester is facing to stand down as CEO, as he told MPs that testing had been “extensive” three days after TSB had been handed the IBM report.

Pester also told the MPs that the issues at the bank were with middleware – despite the fact the IBM report outlined problems with custom and package applications and the network; although it did back-up TSB's statements that the underlying infrastructure was functional.

The report – compiled about a week after the failed go-live following two days of planned downtime for the migration – also set out recommendations for initial actions, advising the bank to focus on stability in the short term and ease pressure on some systems.

This includes a recommendation to throttle connections to try to limit inbound connections to "serve a proportion of customers more consistently", although it acknowledged this would have a bigger impact on internet and mobile users.

IBM also told TSB to prioritise telephony and branch channels, noting that large volumes of internet and mobile banking was "causing additional stress on systems". By getting the other two channels working properly, they could help customers and reduce complaints, IBM said.

It's not clear from the document how many of these recommendations were acted upon – for instance, the bank did say it was "limiting access" to the app and online banking, but this happened before IBM made the presentation.

However, TSB confirmed to The Register that it had not implemented one eyebrow-raising suggestion: to reduce the number of times Actimize – anti-fraud and money laundering software – was invoked in user journeys. IBM's report said this would "create more capacity", but might create "additional risk of fraud".

In a canned statement, TSB said: "The IBM document contained a preliminary work plan with very early hypotheses based on observations to date, that were produced after only three days of engagement with TSB.

"To present this document as a clear view on what went wrong wouldn't be a fair reflection. Similarly it isn't a fair reflection of what actions may or may not subsequently have been taken." ®

Updated to add

Three hours after issuing the above statement, the PR machine at TSB – possibly having realised just what sort of light the IBM report cast on the bank – cranked it up a notch, and got back in touch with an updated missive that aimed to downplay the report it commissioned from the firm it hired to help it deal with the meltdown:

The IBM document contained a preliminary work plan with very early hypotheses based on observations to date, that were produced after only three days of engagement with TSB, almost eight weeks ago. The content is therefore now very much out of date.

The hypotheses were not final, nor were they a validated view of what went wrong or of the actions that have subsequently been taken. Without this context, this document could be misinterpreted to the detriment of TSB's customers.

Similar topics

Narrower topics

Other stories you might like

  • IBM finally shutters Russian operations, lays off staff
    Axing workers under 40 must feel like a novel concept for Big Blue

    After freezing operations in Russia earlier this year, IBM has told employees it is ending all work in the country and has begun laying off staff. 

    A letter obtained by Reuters sent by IBM CEO Arvind Krishna to staff cites sanctions as one of the prime reasons for the decision to exit Russia. 

    "As the consequences of the war continue to mount and uncertainty about its long-term ramifications grows, we have now made the decision to carry out an orderly wind-down of IBM's business in Russia," Krishna said. 

    Continue reading
  • IBM AI boat to commemorate historic US Mayflower voyage finally lands… in Canada
    Nearly two years late and in the wrong country, we welcome our robot overlords

    IBM's self-sailing Mayflower Autonomous Ship (MAS) has finally crossed the Atlantic albeit more than a year and a half later than planned. Still, congratulations to the team.

    That said, MAS missed its target. Instead of arriving in Massachusetts – the US state home to Plymouth Rock where the 17th-century Mayflower landed – the latest in a long list of technical difficulties forced MAS to limp to Halifax in Nova Scotia, Canada. The 2,700-mile (4,400km) journey from Plymouth, UK, came to an end on Sunday.

    The 50ft (15m) trimaran is powered by solar energy, with diesel backup, and said to be able to reach a speed of 10 knots (18.5km/h or 11.5mph) using electric motors. This computer-controlled ship is steered by software that takes data in real time from six cameras and 50 sensors. This application was trained using IBM's PowerAI Vision technology and Power servers, we're told.

    Continue reading
  • IBM buys Randori to address multicloud security messes
    Big Blue joins the hot market for infosec investment

    RSA Conference IBM has expanded its extensive cybersecurity portfolio by acquiring Randori – a four-year-old startup that specializes in helping enterprises manage their attack surface by identifying and prioritizing their external-facing on-premises and cloud assets.

    Big Blue announced the Randori buy on the first day of the 2022 RSA Conference on Monday. Its plan is to give the computing behemoth's customers a tool to manage their security posture by looking at their infrastructure from a threat actor's point-of-view – a position IBM hopes will allow users to identify unseen weaknesses.

    IBM intends to integrate Randori's software with its QRadar extended detection and response (XDR) capabilities to provide real-time attack surface insights for tasks including threat hunting and incident response. That approach will reduce the quantity of manual work needed for monitoring new applications and to quickly address emerging threats, according to IBM.

    Continue reading
  • Compute responsibly: Yet another IT industry sustainability drive
    From greener datacenters to data transparency and 'conscious code', IBM, Dell, others push for better IT ops

    IBM and Dell are the founding members of a new initiative to promote sustainable development in IT by providing a framework of responsible corporate policies for organizations to follow.

    Responsible Computing is described as a membership consortium for technology organizations that aims to get members to sign up to responsible values in key areas relating to infrastructure, code development, and social impact. The program is also operating under the oversight of the Object Management Group.

    According to Object Management Group CEO Bill Hoffman, also the CEO of Responsible Computing, the new initiative aims to "shift thinking and, ultimately behavior" within the IT industry and therefore "bring about real change", based around a manifesto that lays out six domains the program has identified for responsible computing.

    Continue reading

Biting the hand that feeds IT © 1998–2022