Google opens curtain on 'manual' search penalties

Whitelists still hidden in Mountain View dungeon


When the world learned that EU antitrust regulators were investigating claims that Google unfairly manipulated its search results at the expense of competitors, Google's European corporate counsel was unequivocal in her defense of the company. "We don’t whitelist or blacklist anyone,” Julia Holtz told journalists in Brussels last February, according to The Financial Times.

Holtz seemed to imply that Google does not manually override its search algorithms, that all sites are subject to the same mathematical rules. But Google does manually intervene, and this past week, for the first time, the company at least partially explained its rules for search penalties, including manual demotions.

"We have confirmed that Google's webspam team is willing to take action manually – for example, if we get a spam report, off-topic porn, things like that," Google's Matt Cutts said in a video posted to the web on Monday. He also discussed algorithmic penalties, and he said that whereas site owners could appeal manual interventions, they could not appeal the algorithmic variety. This was new information, and it provides some additional insight into the dispute under investigation by the EU.

The video arrived two days after a New York Times story detailing a campaign by American retailer JC Penney – or at least the search engine consulting firm hired by the retailer – to artificially boost the company's Google search-ranking by purchasing links to other sites across the web. The Times contacted Google about the JC Penney scheme, and the company immediately took action.

Thanks to the link-building campaign, JC Penney was the number-one Google search result on a wide variety of keywords, from "dresses" to "bedding" to "area rugs". But when Google intervened, its ranking on the keyword search "Samsonite carry on luggage" dropped from 1 to 71, and its "living room furniture" placement dropped from 1 to 68. Matt Cutts – who heads Google's webspam team – is quoted at length by The Times, and the story makes it clear that Google took what it calls "manual action".

Manual interventions are an important part of any search engine. Foundem, one of the EU complainants, says as much. "Whether manual interventions are required to override something that the algorithms have got wrong or to override something that the algorithms have overlooked, such interventions are an essential part of the process," Foundem CEO Shivaun Raff tells us. The problem is that for so many years, Google has largely avoided acknowledging that these interventions exist, and it has said almost nothing about how they work.

For years, Google told the world that its search engine was completely objective, and only now is the company beginning to freely explain that this is not the case.

In 2006, Foundem essentially vanished from Google's engine, after Mountain View took aim at vertical search engines. Foundem was not the victim of a "manual action". It was demoted by an algorithmic change. The rub was that other similar vertical search engines were not affected by the change. Other sites were apparently whitelisted out of the algorithms. Foundem has produced emails from Google's AdWords team indicating that there are whitelists on the search-ad side, but Google continues to deny that whitelists exist. Company spokesman Adam Kovacevich reiterated the denial on the phone with The Register late last year.

Foundem spent more than three years fighting the demotion – to no avail. But after the company took its story public in late 2009, the site returned to Google's "organic" search results. Foundem – which was granted an audience with Google's Search Quality Team – says its site was manually whitelisted.

In his video, Cutts did not mention whitelists. But he did say that if a site owner makes a "reconsideration request" – i.e., if it appeals Google's search ranking – the company will only examine the request if the site has been manually demoted. "If you do a reconsideration request, at least right now, we're checking against whether you have penalty in place manually," Cutts said. "But if something is only affected by our algorithms and not by any sort of manual action that flagged something that violated our guidelines, then normally you wouldn't be able to apply against the algorithm."

One of the apparent results of the EU investigation is that this sort of information is slowly trickling out of the company. But Google should have provided such guidelines years ago, and it's yet to come clean on such basic issues as whitelists. Whitelists, Foundem says, amount to a safety net, allowing Google to manually correct the inevitable mistakes that arise in its complex and ever-evolving penalty algorithms.

On one level, Foundem's EU antitrust complaint attacks Google's so-called Universal Search setup, which inserts links from other Google services – such as Google Maps and Google Product Search – into prominent positions on the company's main search result pages. Foundem says the setup is transforming Google's search engine into an "immensely powerful marketing channel" for its own services. Google Product Search (formerly known as Froogle) is a price-comparison service similar to Foundem.

But on another level, Foundem takes aim at the way Google handles manual interventions – and how little information we have about how they work. "The half of our Complaint detailing the anticompetitive power of Google's exclusionary penalties is as much about the systemic failures of Google's manual review process as it is about the legitimacy of the penalty algorithms themselves," Foundem CEO Shivaun Raff says.

Even though Foundem was returned to Google's search results, the company went ahead with its antitrust complaint. The change in ranking shouldn't have taken three years. It shouldn't have required pressure applied through the press. The process, Foundem says, is broken. And when you step back and examine the entire situation, the company contends, you see that Google has an unfair advantage when it expands into new markets, including price-comparison shopping or, well, travel search. Google's acquisition of flight-data outfit ITA Software is currently under review by the Department of Justice.

Raff sums up Foundem's EU complaint like this: "You have an overwhelmingly dominant search engine. If you add to that that search engine's ability to apply discriminatory penalties - they're discriminatory because some services are manually rendered immune through whitelists - and you add the ability of that search engine to preferentially insert its own services at or near the top of the search results, all of that adds up to an unparalleled and unassailable competitive advantage." ®

Similar topics


Other stories you might like

  • Prisons transcribe private phone calls with inmates using speech-to-text AI

    Plus: A drug designed by machine learning algorithms to treat liver disease reaches human clinical trials and more

    In brief Prisons around the US are installing AI speech-to-text models to automatically transcribe conversations with inmates during their phone calls.

    A series of contracts and emails from eight different states revealed how Verus, an AI application developed by LEO Technologies and based on a speech-to-text system offered by Amazon, was used to eavesdrop on prisoners’ phone calls.

    In a sales pitch, LEO’s CEO James Sexton told officials working for a jail in Cook County, Illinois, that one of its customers in Calhoun County, Alabama, uses the software to protect prisons from getting sued, according to an investigation by the Thomson Reuters Foundation.

    Continue reading
  • Battlefield 2042: Please don't be the death knell of the franchise, please don't be the death knell of the franchise

    Another terrible launch, but DICE is already working on improvements

    The RPG Greetings, traveller, and welcome back to The Register Plays Games, our monthly gaming column. Since the last edition on New World, we hit level cap and the "endgame". Around this time, item duping exploits became rife and every attempt Amazon Games made to fix it just broke something else. The post-level 60 "watermark" system for gear drops is also infuriating and tedious, but not something we were able to address in the column. So bear these things in mind if you were ever tempted. On that note, it's time to look at another newly released shit show – Battlefield 2042.

    I wanted to love Battlefield 2042, I really did. After the bum note of the first-person shooter (FPS) franchise's return to Second World War theatres with Battlefield V (2018), I stupidly assumed the next entry from EA-owned Swedish developer DICE would be a return to form. I was wrong.

    The multiplayer military FPS market is dominated by two forces: Activision's Call of Duty (COD) series and EA's Battlefield. Fans of each franchise are loyal to the point of zealotry with little crossover between player bases. Here's where I stand: COD jumped the shark with Modern Warfare 2 in 2009. It's flip-flopped from WW2 to present-day combat and back again, tried sci-fi, and even the Battle Royale trend with the free-to-play Call of Duty: Warzone (2020), which has been thoroughly ruined by hackers and developer inaction.

    Continue reading
  • American diplomats' iPhones reportedly compromised by NSO Group intrusion software

    Reuters claims nine State Department employees outside the US had their devices hacked

    The Apple iPhones of at least nine US State Department officials were compromised by an unidentified entity using NSO Group's Pegasus spyware, according to a report published Friday by Reuters.

    NSO Group in an email to The Register said it has blocked an unnamed customers' access to its system upon receiving an inquiry about the incident but has yet to confirm whether its software was involved.

    "Once the inquiry was received, and before any investigation under our compliance policy, we have decided to immediately terminate relevant customers’ access to the system, due to the severity of the allegations," an NSO spokesperson told The Register in an email. "To this point, we haven’t received any information nor the phone numbers, nor any indication that NSO’s tools were used in this case."

    Continue reading

Biting the hand that feeds IT © 1998–2021