Google opens curtain on 'manual' search penalties

Whitelists still hidden in Mountain View dungeon


When the world learned that EU antitrust regulators were investigating claims that Google unfairly manipulated its search results at the expense of competitors, Google's European corporate counsel was unequivocal in her defense of the company. "We don’t whitelist or blacklist anyone,” Julia Holtz told journalists in Brussels last February, according to The Financial Times.

Holtz seemed to imply that Google does not manually override its search algorithms, that all sites are subject to the same mathematical rules. But Google does manually intervene, and this past week, for the first time, the company at least partially explained its rules for search penalties, including manual demotions.

"We have confirmed that Google's webspam team is willing to take action manually – for example, if we get a spam report, off-topic porn, things like that," Google's Matt Cutts said in a video posted to the web on Monday. He also discussed algorithmic penalties, and he said that whereas site owners could appeal manual interventions, they could not appeal the algorithmic variety. This was new information, and it provides some additional insight into the dispute under investigation by the EU.

The video arrived two days after a New York Times story detailing a campaign by American retailer JC Penney – or at least the search engine consulting firm hired by the retailer – to artificially boost the company's Google search-ranking by purchasing links to other sites across the web. The Times contacted Google about the JC Penney scheme, and the company immediately took action.

Thanks to the link-building campaign, JC Penney was the number-one Google search result on a wide variety of keywords, from "dresses" to "bedding" to "area rugs". But when Google intervened, its ranking on the keyword search "Samsonite carry on luggage" dropped from 1 to 71, and its "living room furniture" placement dropped from 1 to 68. Matt Cutts – who heads Google's webspam team – is quoted at length by The Times, and the story makes it clear that Google took what it calls "manual action".

Manual interventions are an important part of any search engine. Foundem, one of the EU complainants, says as much. "Whether manual interventions are required to override something that the algorithms have got wrong or to override something that the algorithms have overlooked, such interventions are an essential part of the process," Foundem CEO Shivaun Raff tells us. The problem is that for so many years, Google has largely avoided acknowledging that these interventions exist, and it has said almost nothing about how they work.

For years, Google told the world that its search engine was completely objective, and only now is the company beginning to freely explain that this is not the case.

In 2006, Foundem essentially vanished from Google's engine, after Mountain View took aim at vertical search engines. Foundem was not the victim of a "manual action". It was demoted by an algorithmic change. The rub was that other similar vertical search engines were not affected by the change. Other sites were apparently whitelisted out of the algorithms. Foundem has produced emails from Google's AdWords team indicating that there are whitelists on the search-ad side, but Google continues to deny that whitelists exist. Company spokesman Adam Kovacevich reiterated the denial on the phone with The Register late last year.

Foundem spent more than three years fighting the demotion – to no avail. But after the company took its story public in late 2009, the site returned to Google's "organic" search results. Foundem – which was granted an audience with Google's Search Quality Team – says its site was manually whitelisted.

In his video, Cutts did not mention whitelists. But he did say that if a site owner makes a "reconsideration request" – i.e., if it appeals Google's search ranking – the company will only examine the request if the site has been manually demoted. "If you do a reconsideration request, at least right now, we're checking against whether you have penalty in place manually," Cutts said. "But if something is only affected by our algorithms and not by any sort of manual action that flagged something that violated our guidelines, then normally you wouldn't be able to apply against the algorithm."

One of the apparent results of the EU investigation is that this sort of information is slowly trickling out of the company. But Google should have provided such guidelines years ago, and it's yet to come clean on such basic issues as whitelists. Whitelists, Foundem says, amount to a safety net, allowing Google to manually correct the inevitable mistakes that arise in its complex and ever-evolving penalty algorithms.

On one level, Foundem's EU antitrust complaint attacks Google's so-called Universal Search setup, which inserts links from other Google services – such as Google Maps and Google Product Search – into prominent positions on the company's main search result pages. Foundem says the setup is transforming Google's search engine into an "immensely powerful marketing channel" for its own services. Google Product Search (formerly known as Froogle) is a price-comparison service similar to Foundem.

But on another level, Foundem takes aim at the way Google handles manual interventions – and how little information we have about how they work. "The half of our Complaint detailing the anticompetitive power of Google's exclusionary penalties is as much about the systemic failures of Google's manual review process as it is about the legitimacy of the penalty algorithms themselves," Foundem CEO Shivaun Raff says.

Even though Foundem was returned to Google's search results, the company went ahead with its antitrust complaint. The change in ranking shouldn't have taken three years. It shouldn't have required pressure applied through the press. The process, Foundem says, is broken. And when you step back and examine the entire situation, the company contends, you see that Google has an unfair advantage when it expands into new markets, including price-comparison shopping or, well, travel search. Google's acquisition of flight-data outfit ITA Software is currently under review by the Department of Justice.

Raff sums up Foundem's EU complaint like this: "You have an overwhelmingly dominant search engine. If you add to that that search engine's ability to apply discriminatory penalties - they're discriminatory because some services are manually rendered immune through whitelists - and you add the ability of that search engine to preferentially insert its own services at or near the top of the search results, all of that adds up to an unparalleled and unassailable competitive advantage." ®

Similar topics


Other stories you might like

  • Experts: AI should be recognized as inventors in patent law
    Plus: Police release deepfake of murdered teen in cold case, and more

    In-brief Governments around the world should pass intellectual property laws that grant rights to AI systems, two academics at the University of New South Wales in Australia argued.

    Alexandra George, and Toby Walsh, professors of law and AI, respectively, believe failing to recognize machines as inventors could have long-lasting impacts on economies and societies. 

    "If courts and governments decide that AI-made inventions cannot be patented, the implications could be huge," they wrote in a comment article published in Nature. "Funders and businesses would be less incentivized to pursue useful research using AI inventors when a return on their investment could be limited. Society could miss out on the development of worthwhile and life-saving inventions."

    Continue reading
  • Declassified and released: More secret files on US govt's emergency doomsday powers
    Nuke incoming? Quick break out the plans for rationing, censorship, property seizures, and more

    More papers describing the orders and messages the US President can issue in the event of apocalyptic crises, such as a devastating nuclear attack, have been declassified and released for all to see.

    These government files are part of a larger collection of records that discuss the nature, reach, and use of secret Presidential Emergency Action Documents: these are executive orders, announcements, and statements to Congress that are all ready to sign and send out as soon as a doomsday scenario occurs. PEADs are supposed to give America's commander-in-chief immediate extraordinary powers to overcome extraordinary events.

    PEADs have never been declassified or revealed before. They remain hush-hush, and their exact details are not publicly known.

    Continue reading
  • Stolen university credentials up for sale by Russian crooks, FBI warns
    Forget dark-web souks, thousands of these are already being traded on public bazaars

    Russian crooks are selling network credentials and virtual private network access for a "multitude" of US universities and colleges on criminal marketplaces, according to the FBI.

    According to a warning issued on Thursday, these stolen credentials sell for thousands of dollars on both dark web and public internet forums, and could lead to subsequent cyberattacks against individual employees or the schools themselves.

    "The exposure of usernames and passwords can lead to brute force credential stuffing computer network attacks, whereby attackers attempt logins across various internet sites or exploit them for subsequent cyber attacks as criminal actors take advantage of users recycling the same credentials across multiple accounts, internet sites, and services," the Feds' alert [PDF] said.

    Continue reading

Biting the hand that feeds IT © 1998–2022