OK Google, what is African ISP Main One, and how did it manage to route your traffic into China through Russia?

Sub cable biz raises hand, 'fesses up to causing BGP hijack drama

Updated Monday's prolonged Google cloud and websites outage was triggered by a botched network update by a West Africa telco, it is claimed.

Main One, a biz ISP based in Lagos, Nigeria, that operates a submarine cable between Portugal and South Africa, said a misconfiguration at its end caused Google-bound traffic to be redirected to China Telecom for 74 minutes.

During that time, web browsers and apps that tried to connect to Google, YouTube, etc, or sites and platforms on Google Cloud, such as Spotify and Nest, were routed to the Chinese telco via Russian ISP TransTelekom, and dropped into a black hole.

The blunder was possible because Main One leaked details of one part of the internet's layout into the configuration of another, temporarily rewiring the spinal cord of the 'net. Packets working their way toward Google were sent down routes that took them around the globe.

Main One peers with Google, in that they agree to exchange traffic with each other through a peering point. Simply put, the ISP accidentally let slip details of its routes into Google's network in a way that caused the rest of the 'net to adjust its pathways so Google-bound traffic headed toward China Telecom.

"MainOne has a peering relationship with Google via IXPN in Lagos and has direct routes to Google, which leaked into China Telecom," explained Ameet Naik of cloud-monitoring biz ThousandEyes today. These leaked routes propagated from China Telecom, via TransTelecom to NTT and other transit ISPs. We also noticed that this leak was primarily propagated by business-grade transit providers and did not impact consumer ISP networks as much."

This eventually caused vast bucketfuls of internet traffic bound for Google in the US and posibly elsewhere to pour into a bottomless pit in China Telcom's network, effectively knocking the ad giant offline in the eyes of many netizens. It's understood no data was intercepted or handed over during the spill. Here's Main One 'fessing up:

Google also said that none of its servers or data was affected by the incident.

"We’re aware that a portion of internet traffic was affected by incorrect routing of IP addresses, and access to some Google services was impacted," a Chocolate Factory spokesperson told El Reg. "The root cause of the issue was external to Google and there was no compromise of Google services."


OK Google, why was your web traffic hijacked and routed through China, Russia today?


If nothing else, the disclosure will allay fears that the outage was the result of some sort of attack or other nefarious activity. At the same time, the realization that something as simple as a regional ISP misconfiguring a server could trigger a global outage does not sit particularly well either.

"This incident further underscores one of the fundamental weaknesses in the fabric of the internet," said Naik. "BGP was designed to be a chain of trust between well-meaning ISPs and universities that blindly believe the information they receive. It hasn’t evolved to reflect the complex commercial and geopolitical relationships that exist between ISPs and nations today. While verification methods like ROA [Route Origin Authorization] exist, few ISPs use them. Even corporations like Google with massive resources at their disposal are not immune from this sort of BGP leak or malicious hijacks."

As NSA advisor and former White House cyber security boss Rob Joyce noted, the incident should serve as a call to reassess the state of the BGP system.

"I hope this latest fiasco of traffic rerouting through China is the wakeup call for all of us to get serious about addressing the massive and unacceptable vulnerability inherent in today’s BGP routing architecture," Joyce said on Tuesday. ®

Updated to add

Cloudflare has some more technical info on the BGP blunder, and suggests it is possible to perform route filtering to cut out dodgy updates. Unfortunately, China Telecom's CN2 carrier did not perform any sanity checks on Main One's changes to the internet's pathways, leading to this week's cock-up.

Also, had the intermediate systems been able to withstand the volume of Google-bound traffic, the sites and cloud connections wouldn't have dropped. In the end, the machines that suddenly had Google traffic routed through them fell over, taking Google with it for a ton of netizens.

Broader topics

Other stories you might like

  • Lonestar plans to put datacenters in the Moon's lava tubes
    How? Founder tells The Register 'Robots… lots of robots'

    Imagine a future where racks of computer servers hum quietly in darkness below the surface of the Moon.

    Here is where some of the most important data is stored, to be left untouched for as long as can be. The idea sounds like something from science-fiction, but one startup that recently emerged from stealth is trying to turn it into a reality. Lonestar Data Holdings has a unique mission unlike any other cloud provider: to build datacenters on the Moon backing up the world's data.

    "It's inconceivable to me that we are keeping our most precious assets, our knowledge and our data, on Earth, where we're setting off bombs and burning things," Christopher Stott, founder and CEO of Lonestar, told The Register. "We need to put our assets in place off our planet, where we can keep it safe."

    Continue reading
  • Conti: Russian-backed rulers of Costa Rican hacktocracy?
    Also, Chinese IT admin jailed for deleting database, and the NSA promises no more backdoors

    In brief The notorious Russian-aligned Conti ransomware gang has upped the ante in its attack against Costa Rica, threatening to overthrow the government if it doesn't pay a $20 million ransom. 

    Costa Rican president Rodrigo Chaves said that the country is effectively at war with the gang, who in April infiltrated the government's computer systems, gaining a foothold in 27 agencies at various government levels. The US State Department has offered a $15 million reward leading to the capture of Conti's leaders, who it said have made more than $150 million from 1,000+ victims.

    Conti claimed this week that it has insiders in the Costa Rican government, the AP reported, warning that "We are determined to overthrow the government by means of a cyber attack, we have already shown you all the strength and power, you have introduced an emergency." 

    Continue reading
  • China-linked Twisted Panda caught spying on Russian defense R&D
    Because Beijing isn't above covert ops to accomplish its five-year goals

    Chinese cyberspies targeted two Russian defense institutes and possibly another research facility in Belarus, according to Check Point Research.

    The new campaign, dubbed Twisted Panda, is part of a larger, state-sponsored espionage operation that has been ongoing for several months, if not nearly a year, according to the security shop.

    In a technical analysis, the researchers detail the various malicious stages and payloads of the campaign that used sanctions-related phishing emails to attack Russian entities, which are part of the state-owned defense conglomerate Rostec Corporation.

    Continue reading

Biting the hand that feeds IT © 1998–2022