Tokyo Stock Exchange lets CEO resign to atone for October outage, other execs take pay cuts and rebukes

Wobbly NAS tended by Fujitsu crashed trading – and automatic failover failed

Miyahara Koichiro, the CEO of the Tokyo Stock Exchange (TSE), has resigned in the wake of its October outage. The exchange’s operator, Japan Exchange Group, accepted his request to stand down.

The IT breakdown saw the world’s third-largest stock exchange take an unplanned day off due to what was described as “a technical glitch occurred to distribution of market data.” Fujitsu, which provides some of the exchange’s core technology, apologized for its role in the downtime.

On Monday, the Japan Exchange Group published an analysis [PDF] of the kerfuffle, based on findings by the Financial Services Agency of Japan (FSA). The document stated the root cause of the outage was “a defect in a failed device,” and everything went pear-shaped because “a setting for the automatic switchover regarding the failed device was inadequate and TSE had not developed sufficient rules for trading resumption.”

A more extensive analysis [PDF] that’s only been published in Japanese, and which The Register has read after using online translation tools, reports that a network-attached storage box, which could not write data to a memory card, was the source of the problem.

Attempts to failover the NAS to a secondary unit proved difficult, and was only achieved after more than an hour of effort to manually switch over to a spare array. As work continued behind the scenes, the exchange opened.

The NAS fault meant that data flowing to market participants during that time was not accurate, yet market participants had placed orders and the exchange had accepted them.

That all added up to a situation in which participants’ trading intentions may not have been reflected in accurate trades, so the exchange pulled the plug for the day.

The short report found that the exchange hadn’t developed procedures to resume trading in the event of an outage of this sort, and lacked clear rules for doing so. Regulators have therefore called on TSE to develop those procedures and embark on an extensive resilience-building program to make sure its IT works as intended.

Koichiro resigned, and other executives were issued “stern warnings,” and/or had their pay reduced for periods of between four and six months.

Fujitsu’s Japanese and global websites offer no statements relating to the incident at the time of writing. ®