Disaster recovery blunder broke New York Stock Exchange this week

The irony, it burns

On Tuesday, the New York Stock Exchange (NYSE) failed to open properly, disrupting and invalidating auction trading for more than 250 securities.

The NYSE in a statement on Wednesday said that "a technical issue" prevented the opening auction process by which stocks get priced through the reconciliation of orders since the previous day's market close.

"The root cause was determined to be a manual error involving the exchange’s disaster recovery configuration at system start of day," the NYSE said.

The root cause was determined to be a manual error involving the exchange’s Disaster Recovery configuration

The problem, according to Bloomberg, is that an NYSE employee failed to shut down a disaster recovery system at the exchange's secondary Chicago data center. Because the system was left running overnight, the market software at the NYSE acted as if trading had already begun and prevented opening auction prices from being set correctly.

Based on its rule that "clearly erroneous trades" should be invalidated, the exchange "determined that approximately 4,341 trades in 251 symbols should be busted."

Fixing these errant trades is expected to be expensive. According to Reuters, the cost "is likely to be in the eight-figure range." NYSE rules [PDF] give investors three business days to claim damages due to a suspected system failure.

Affected companies [XLSX] include ExxonMobil, Morgan Stanley, Verizon, WalMart, Walt Disney Company, and Wells Fargo.

Brokerage Charles Schwab, listed on the NYSE but not among the stocks directly affected by the snafu, slammed the exchange for the way it has dealt with the situation.

The NYSE has not owned up to their full responsibility and investors will have to go through a lengthy process to correct orders

"We are disappointed with the way the NYSE handled their system failure [on Tuesday], which they admitted was the result of their own manual error," a company spokesperson said in a statement provided to The Register.

"Unfortunately, the NYSE has not owned up to their full responsibility and retail investors will have to go through a lengthy process to correct orders, with no guarantee of a reasonable outcome.

"In light of these most recent events coupled with past similar events, we strongly urge the SEC to carefully re-consider the risks of forcing retail orders through unproven auctions on exchanges. If exchanges will not accept accountability when they make an obvious mistake, it further heightens our concerns that routing even greater levels of retail orders to the exchanges will dramatically reduce the quality of the investing experience for America’s retail investors."

The US Securities and Exchange Commission – the SEC referenced above – is said to be looking into the screw-up.

Intercontinental Exchange Inc, which owns and operates the NYSE, did not immediately respond to a request for comment. ®