Train-knackering software design blunder discovered after lightning sparked Thameslink megadelay

Official reports reveal 'as designed but not intended' snafu

133 Reg comments Got Tips?

Updated British electricity providers are paying £10.5m after a 2019 outage revealed a train-bricking software design flaw.

Companies behind the Hornsea One wind farm, off the Yorkshire coast, and the Little Barford conventional power station in Bedfordshire have between them coughed up £9m in "redress" to UK 'leccy regulator Ofgem after the August lightning strike and associated outage.

Distribution firm UK Power Networks is paying a further £1.5m for its role in the resulting power blackouts which spread across large parts of England, Wales and Scotland, according to Ofgem's report published today (PDF, 26 pages).

Yet in among the official reports of electrickery woes is an all-too-familiar tale of software design, including a problem that wasn't spotted until it was too late. In this case, the flaw was on Thameslink's Siemens Desiro City Class 700 and Class 717 trains.

On 9 August last year, a lightning strike on power lines north of London caused 2GW of "distributed" electrical generation to almost instantly disconnect from the National Grid. The spinning reserve available to grid controllers was "just over 1,000MW". Ofgem noted that "the cumulative loss of Hornsea-1 Windfarm, the steam turbine at Little Barford Station and a number of smaller embedded generators exceeded the reserve and response being held". As a result, the frequency of the electricity system began to fall and frequency response services were automatically triggered, which "initially arrested the frequency at 49.1Hz."

It added:

The system frequency began to recover, however the additional loss of the Little Barford gas turbine caused a second drop in system frequency to 48.8Hz, triggering the operation of the LFDD automatic protection system to restore the balance between generation and demand, by disconnecting demand. This resulted in the disconnection of over 1 million customers.

As for the Thameslink railway service, which runs mostly between Brighton, Bedford and Cambridge, it soon discovered a software flaw with its Class 700 and Class 717 electric trains: a frequency drop beyond 49Hz put the trains into a failure mode – which required manual reset by an engineer.

Although railway standards specified the lowest permitted frequency before shutdown as 47.0Hz, within 200ms of Network Rail's power supply frequency dropping past 49.0Hz the trains sat down, folded their arms and refused to go anywhere.

As designed but not as intended

The Office of Rail and Road (ORR), a regulator, published a detailed report explaining that "technicians with laptops" (PDF, 14 pages) had to be dispatched to 22 stranded trains. It found: "All Class 700 and 717 trains had been programmed by [manufacturer] Siemens to operate from a power supply of a nominal 50.0Hz, with a minimum frequency of 49.0Hz."

Helpfully, the trains were halfway through a new software deployment. Version 3.25.x went into "temporary lock-out", which the driver could reset, something that seven drivers did successfully. Trains running v3.27.x entered "permanent lock-out" needing a techie with a laptop to reset them. Permanent lock-outs, the ORR report explained, are a safety feature to prevent drivers from re-electrifying damaged components and exposing people to the risk of electric shock.

The ORR continued: "Power supply frequency excursions of the magnitude experienced are unusual, so the service-disrupting implications of imposing a lock-out requiring a driver or – as in this case – a technician to reset appear not to have been given weight when developing the protection parameters for the on-train software."

It appears therefore that the collective response of the Class 700 and 717 trains to the out-of-specification supply frequency was in accordance with the software design, but was not an explicit intention. Siemens accepts that the temporary reduction in frequency should not have been considered a situation that requires a permanent lock-out.

Siemens is said to be delivering a software patch to sort out the trains' tripping frequency.

According to Ofgem's report on the power outage, this caused 371 trains to be cancelled, 220 to be part-cancelled and 873 to be delayed, with problems reportedly knocking on for three days after the 45-minute incident. ®

Juicenote

Scottish Power "erroneously disconnected 22MW of demand in Scotland," Ofgem found, because its auto-disconnect gear had been set to trip at 48.8Hz instead of the correct 48.5Hz frequency.

Updated to add

Steve White, chief operating officer of Govia Thameslink Railway, told us in a statement: “Passengers are now protected from this rare but disruptive event because, following a full review with GTR, Siemens Mobility has modified all the trains so that none of them now need a technician to restart."

SUBSCRIBE TO OUR WEEKLY TECH NEWSLETTER


Biting the hand that feeds IT © 1998–2020