An outage that knocked out half of New York's subway system and left hundreds of passengers stuck in tunnels was likely caused by someone in a control center pressing the wrong button and turning off part of the network, according to New York Governor Kathy Hochul.
The incident occurred on the evening of August 29, when a number of pieces of equipment stopped functioning in the New York City Transit Rail Control Center due to a power dip at around 2025 ET. It is thought the "emergency power off" button was then pressed in error during attempts by control center staff to bring the affected equipment back online again.
Independent engineering firms HDR [PDF] and WSP [PDF] contributed to an investigation by Hochul, and the pair said there was a "strong possibility" the button had been pressed accidentally in the confusion following the power dip. Their suspicions in this regard were strengthened by the discovery that a plastic cover over the emergency power off button – designed to prevent exactly this eventuality – was missing.
Every electrical system connected to one of the power distribution units at the center subsequently lost power from 2106 ET onward, causing all of the numbered lines and the L train on the city's incomprehensible and counter-intuitive subway network to stop working.
The breakdown affected over 80 individual trains across eight different lines.
Power was fully restored by 2230 ET, though the lines involved continued to be affected for several hours afterwards. The restoration of the service was delayed partly because passengers on two stopped trains chose to get out and walk along the track rather than wait for the problem to be fixed.
We at The Register hope for their own safety that these impatient individuals are never stuck in a failed train on the Wuppertal Schwebebahn.
The blame has officially been attributed to human error, with the inability to restore power for 84 minutes put down to "inadequate organizational structure and a lack of guidelines," according to the Associated Press.
The Metropolitan Transit Authority CEO Janno Lieber has promised it will reorganise how it manages and maintains key equipment in the control center in order to reduce the risk of a reoccurrence.
"It appears that a button was pushed accidentally that was not supposed to be pushed," Lieber told Forbes, unflinchingly but obviously. "One: that can't happen, that somebody does something wrong because of lack of training or proper protocols. Two: there have to be interventions that prevent people from hitting the wrong button."
Presumably starting with a new cover for the switch. ®