Google routing blunder sent Japan's Internet dark on Friday

Another big BGP blunder


Last Friday, someone in Google fat-thumbed a border gateway protocol (BGP) advertisement and sent Japanese Internet traffic into a black hole.

The trouble began when The Chocolate Factory “leaked” a big route table to Verizon, the result of which was traffic from Japanese giants like NTT and KDDI was sent to Google on the expectation it would be treated as transit.

Since Google doesn't provide transit services, as BGP Mon explains, that traffic either filled a link beyond its capacity, or hit an access control list, and disappeared.

The outage in Japan only lasted a couple of hours, but was so severe that Japan Times reports the country's Internal Affairs and Communications ministries want carriers to report on what went wrong.

BGP Mon dissects what went wrong here, reporting that more than 135,000 prefixes on the Google-Verizon path were announced when they shouldn't have been.

Since it leaked what the monitors call “a full table” to Verizon, the fat-thumb error also provided a “peek into what Google's peering relationships look like and how their peers traffic engineer towards Google”.

For example, BGP Mon explains how the mistake hit ISP Jastel (Jasmine Telecom) in Thailand:

“If we take a closer look at the AS paths involved starting at the right side, we see the prefix was announced by 45629 (Jastel) as expected. Since Jastel peers with Google (15169) that’s the next AS we see. The next AS in the path is 701 (Verizon) and this is where it is getting interesting as Verizon has now started to provide transit for Jastel via Google.

“Verizon (701) then announced that to several of it’s customers, some of them very large such as KPN (286) and Orange (5511). So by just looking at four example paths we can see it hit large networks in Europe, Latin America, the US, and India (9498 Airtel).”

BGP is the Internet's protocol for distributing routing information between networks. A BGP advertisement shouts out to the rest of the internet to announce things like “if you give me traffic for Verizon, it will reach its destination”.

Designed for a more trusting (and much smaller) Internet, BGP's most serious shortcoming is that it's up to network admins to check and filter information in route advertisements.

As BGP Mon notes, BGP leaks are “a great risk to the Internet's stability”, and both sides of an advertisement should be filtering them before accepting them.

Previous BGP incidents have sent YouTube traffic to Pakistan, blackholed Chinese traffic, made Belarus the default route for more traffic than it could handle, and redirected Level 3's traffic to Malaysia.

There are various proposals to tweak BGP to stop this sort of thing happening, but as is so often the case, implementation is lagging far behind requirement. ®


Other stories you might like

  • Google has more reasons why it doesn't like antitrust law that affects Google
    It'll ruin Gmail, claims web ads giant

    Google has a fresh list of reasons why it opposes tech antitrust legislation making its way through Congress but, like others who've expressed discontent, the ad giant's complaints leave out mention of portions of the proposed law that address said gripes.

    The law bill in question is S.2992, the Senate version of the American Innovation and Choice Online Act (AICOA), which is closer than ever to getting votes in the House and Senate, which could see it advanced to President Biden's desk.

    AICOA prohibits tech companies above a certain size from favoring their own products and services over their competitors. It applies to businesses considered "critical trading partners," meaning the company controls access to a platform through which business users reach their customers. Google, Apple, Amazon, and Meta in one way or another seemingly fall under the scope of this US legislation. 

    Continue reading
  • I was fired for blowing the whistle on cult's status in Google unit, says contractor
    The internet giant, a doomsday religious sect, and a lawsuit in Silicon Valley

    A former Google video producer has sued the internet giant alleging he was unfairly fired for blowing the whistle on a religious sect that had all but taken over his business unit. 

    The lawsuit demands a jury trial and financial restitution for "religious discrimination, wrongful termination, retaliation and related causes of action." It alleges Peter Lubbers, director of the Google Developer Studio (GDS) film group in which 34-year-old plaintiff Kevin Lloyd worked, is not only a member of The Fellowship of Friends, the exec was influential in growing the studio into a team that, in essence, funneled money back to the fellowship.

    In his complaint [PDF], filed in a California Superior Court in Silicon Valley, Lloyd lays down a case that he was fired for expressing concerns over the fellowship's influence at Google, specifically in the GDS. When these concerns were reported to a manager, Lloyd was told to drop the issue or risk losing his job, it is claimed. 

    Continue reading
  • End of the road for biz living off free G Suite legacy edition
    Firms accustomed to freebies miffed that web giant's largess doesn't last

    After offering free G Suite apps for more than a decade, Google next week plans to discontinue its legacy service – which hasn't been offered to new customers since 2012 – and force business users to transition to a paid subscription for the service's successor, Google Workspace.

    "For businesses, the G Suite legacy free edition will no longer be available after June 27, 2022," Google explains in its support document. "Your account will be automatically transitioned to a paid Google Workspace subscription where we continue to deliver new capabilities to help businesses transform the way they work."

    Small business owners who have relied on the G Suite legacy free edition aren't thrilled that they will have to pay for Workspace or migrate to a rival like Microsoft, which happens to be actively encouraging defectors. As noted by The New York Times on Monday, the approaching deadline has elicited complaints from small firms that bet on Google's cloud productivity apps in the 2006-2012 period and have enjoyed the lack of billing since then.

    Continue reading
  • Google recasts Anthos with hitch to AWS Outposts
    If at first you don't succeed, change names and try again

    Google Cloud's Anthos on-prem platform is getting a new home under the search giant’s recently announced Google Distributed Cloud (GDC) portfolio, where it will live on as a software-based competitor to AWS Outposts and Microsoft Azure Stack.

    Introduced last fall, GDC enables customers to deploy managed servers and software in private datacenters and at communication service provider or on the edge.

    Its latest update sees Google reposition Anthos on-prem, introduced back in 2020, as the bring-your-own-server edition of GDC. Using the service, customers can extend Google Cloud-style management and services to applications running on-prem.

    Continue reading
  • Google offers $118m to settle gender discrimination lawsuit
    Don't even think about putting LaMDA on the compensation committee

    Google has promised to cough up $118 million to settle a years-long gender-discrimination class-action lawsuit that alleged the internet giant unfairly pays men more than women.

    The case, launched in 2017, was led by three women, Kelly Ellis, Holly Pease, and Kelli Wisuri, who filed a complaint alleging the search giant hires women in lower-paying positions compared to men despite them having the same qualifications. Female staff are also less likely to get promoted, it was claimed.

    Gender discrimination also exists within the same job tier, too, the complaint stated. Google was accused of paying women less than their male counterparts despite them doing the same work. The lawsuit was later upgraded to a class-action status when a fourth woman, Heidi Lamar, joined as a plaintiff. The class is said to cover more than 15,000 people.

    Continue reading
  • US to help Japan make leading-edge 2nm chips, possibly by 2025
    Player Four has entered the game

    Japan is reportedly hoping to join the ranks of countries producing leading-edge 2nm chips as soon as 2025, and it's working with the US to make such ambitions a reality.

    Nikkei reported Wednesday that businesses from both countries will jointly research the design and manufacturing of such components for devices ranging from smartphones to servers as part of a "bilateral chip technology partnership" between America and Japan.

    The report arrives less than a month after US and Japanese leaders said they would collaborate on next-generation semiconductors as part of broader agreement that also calls for "protecting and promoting critical technologies, including through the use of export controls."

    Continue reading
  • Google: How we tackled this iPhone, Android spyware
    Watching people's every move and collecting their info – not on our watch, says web ads giant

    Spyware developed by Italian firm RCS Labs was used to target cellphones in Italy and Kazakhstan — in some cases with an assist from the victims' cellular network providers, according to Google's Threat Analysis Group (TAG).

    RCS Labs customers include law-enforcement agencies worldwide, according to the vendor's website. It's one of more than 30 outfits Google researchers are tracking that sell exploits or surveillance capabilities to government-backed groups. And we're told this particular spyware runs on both iOS and Android phones.

    We understand this particular campaign of espionage involving RCS's spyware was documented last week by Lookout, which dubbed the toolkit "Hermit." We're told it is potentially capable of spying on the victims' chat apps, camera and microphone, contacts book and calendars, browser, and clipboard, and beam that info back to base. It's said that Italian authorities have used this tool in tackling corruption cases, and the Kazakh government has had its hands on it, too.

    Continue reading

Biting the hand that feeds IT © 1998–2022