Techies try to bypass damaged UPS, send 380V into air traffic system
New Year's Day debacle strands thousands of passengers, attributed to decade-old tech
While weather and workforce strikes have affected air travel in Europe and the US this holiday season, Southeast Asia experienced disruption of its own due to a New Year's Day power outage at Manila's Ninoy Aquino International Airport (NAIA) that shut down both flights and airspace.
"At around 9:49 am local time, the Air Traffic Management Center (ATMC) which serves as the facility for controlling and overseeing all inbound and outbound flights and overflights within the Philippine airspace, went down due to power outage, resulting to loss of communication, radio, radar, and internet," said transport secretary Jaime Bautista in an online press conference.
The outage went on for eight hours, with normal operations resuming around 18.00 the same day. While NAIA activated its crisis management team for a "multi-discipline approach to cushion the impact of the incident" and ordered emergency response teams to implement irregular operations procedures, more than 65,000 would-be passengers across 360-plus flights were stranded.
The glitch didn't just delay passengers coming in and out of Manila, it also disrupted the Manila Flight Information Region (FIR), which covers the entirety of the Philippines. This caused all commercial flights out of the country's airspace to be redirected.
There are currently no airborne commercial flights in the Philippines (MANILA FIR). pic.twitter.com/xUhyPJarmE— Flightradar24 (@flightradar24) January 1, 2023
MIAA general manager Cesar Chiong anticipated it would take approximately 72 hours to return to normal operations, with most flights coming in or out of Manila packed for the time being.
- Philippines logs on to Starlink for remote area internet services
- Southwest Airlines blames IT breakdown for stranding holiday travelers
- UK government in talks with datacenter operators over blackouts
- Tech supply chains brace for impact as China shifts from zero-COVID to rampant COVID
According to the Civil Aviation Authority of the Philippines (CAAP), the loss in power originated in the Communications, Navigation, and Surveillance/Air Traffic Management (CNS/ATM) system's electrical network. Of course, uninterruptible power supplies (UPS) exist to prevent this exact scenario, but neither of the system's two backup power supplies worked.
One because its blowers "conked out," and the other simply failed to come online, according to CAAP director general Manuel Tamayo.
"The primary cause identified was a problem with the power supply and the degraded uninterrupted power supply, which had no link to the commercial power, and had to be connected to the other manually. The secondary problem was the power surge due to the power outage, which affected the equipment," said Bautista.
That power surge occurred when technicians tried to bypass the damaged UPS and sent 380 volts into the system, instead of the intended 220 volts, thus frying the terminals that receive satellite data from airplanes and air traffic management systems.
To add to the shame of not one but two failed UPSes, both the CAAP and Bautista recognized the CMS/ATM was already outdated before it was even fully operational in July 2019. Currently, in 2023, it has been outdated for a full decade.
"Although it's a system that was introduced in 2010, we implemented this in 2018, so the system is already in its midlife, so we really need to improve or modernize it. Maybe we can still use it, but we need to upgrade this to a better system," said Bautista.
The transport secretary added that the country's air traffic management system was at least 10 years behind that of Singapore.
The CAAP said it has made recommendations to President Ferdinand "BongBong" Marcos Jr to improve the country's air traffic management system. Marcos has already ordered a proposal on system upgrades. An investigation is underway by CAAP's Aerodrome and Air Navigation Safety Oversight Office and some senators have called for heads to roll at the CAAP. Other politicians have urged to allocate funding for upgrades.
According to Batista, an upgrade would cost upward of $233 million (13 billion Philippine pesos), similar to what the 2018 system cost. That system was financed with loans from the Japan International Cooperation Agency.
Opportunists have already come in to offer assistance with the upgrade. Manuel Pangilinan, owner of the Philippines' largest telecom company, PLDT, suggested a colocation of multiple protective redundancies.
If our Group could be of any help to DOTr/CAAP, we’d be happy to participate - colocation of 2nd, even 3rd, redundancies in our nationwide data centers, required connectivities - fiber, satellite, wireless - robust even redundant power supply protection etc.Let’s all support.— Manny V. Pangilinan (@iamMVP) January 1, 2023
Pangilinan himself was on a flight from Tokyo to Manila when his plane was turned around and sent back to Haneda.
"Six hours of useless flying but inconvenience to travelers and losses to tourism and business are horrendous. Only in the PH. Sigh," tweeted Pangilinan.
Pangilinan is familiar with difficult system upgrades. PLDT experienced a P48 billion budget overrun after it over ordered network equipment, forcing the exec into a shareholder mea culpa just last month. ®