Google opens curtain on 'manual' search penalties

Whitelists still hidden in Mountain View dungeon


When the world learned that EU antitrust regulators were investigating claims that Google unfairly manipulated its search results at the expense of competitors, Google's European corporate counsel was unequivocal in her defense of the company. "We don’t whitelist or blacklist anyone,” Julia Holtz told journalists in Brussels last February, according to The Financial Times.

Holtz seemed to imply that Google does not manually override its search algorithms, that all sites are subject to the same mathematical rules. But Google does manually intervene, and this past week, for the first time, the company at least partially explained its rules for search penalties, including manual demotions.

"We have confirmed that Google's webspam team is willing to take action manually – for example, if we get a spam report, off-topic porn, things like that," Google's Matt Cutts said in a video posted to the web on Monday. He also discussed algorithmic penalties, and he said that whereas site owners could appeal manual interventions, they could not appeal the algorithmic variety. This was new information, and it provides some additional insight into the dispute under investigation by the EU.

The video arrived two days after a New York Times story detailing a campaign by American retailer JC Penney – or at least the search engine consulting firm hired by the retailer – to artificially boost the company's Google search-ranking by purchasing links to other sites across the web. The Times contacted Google about the JC Penney scheme, and the company immediately took action.

Thanks to the link-building campaign, JC Penney was the number-one Google search result on a wide variety of keywords, from "dresses" to "bedding" to "area rugs". But when Google intervened, its ranking on the keyword search "Samsonite carry on luggage" dropped from 1 to 71, and its "living room furniture" placement dropped from 1 to 68. Matt Cutts – who heads Google's webspam team – is quoted at length by The Times, and the story makes it clear that Google took what it calls "manual action".

Manual interventions are an important part of any search engine. Foundem, one of the EU complainants, says as much. "Whether manual interventions are required to override something that the algorithms have got wrong or to override something that the algorithms have overlooked, such interventions are an essential part of the process," Foundem CEO Shivaun Raff tells us. The problem is that for so many years, Google has largely avoided acknowledging that these interventions exist, and it has said almost nothing about how they work.

For years, Google told the world that its search engine was completely objective, and only now is the company beginning to freely explain that this is not the case.

In 2006, Foundem essentially vanished from Google's engine, after Mountain View took aim at vertical search engines. Foundem was not the victim of a "manual action". It was demoted by an algorithmic change. The rub was that other similar vertical search engines were not affected by the change. Other sites were apparently whitelisted out of the algorithms. Foundem has produced emails from Google's AdWords team indicating that there are whitelists on the search-ad side, but Google continues to deny that whitelists exist. Company spokesman Adam Kovacevich reiterated the denial on the phone with The Register late last year.

Foundem spent more than three years fighting the demotion – to no avail. But after the company took its story public in late 2009, the site returned to Google's "organic" search results. Foundem – which was granted an audience with Google's Search Quality Team – says its site was manually whitelisted.

In his video, Cutts did not mention whitelists. But he did say that if a site owner makes a "reconsideration request" – i.e., if it appeals Google's search ranking – the company will only examine the request if the site has been manually demoted. "If you do a reconsideration request, at least right now, we're checking against whether you have penalty in place manually," Cutts said. "But if something is only affected by our algorithms and not by any sort of manual action that flagged something that violated our guidelines, then normally you wouldn't be able to apply against the algorithm."

One of the apparent results of the EU investigation is that this sort of information is slowly trickling out of the company. But Google should have provided such guidelines years ago, and it's yet to come clean on such basic issues as whitelists. Whitelists, Foundem says, amount to a safety net, allowing Google to manually correct the inevitable mistakes that arise in its complex and ever-evolving penalty algorithms.

On one level, Foundem's EU antitrust complaint attacks Google's so-called Universal Search setup, which inserts links from other Google services – such as Google Maps and Google Product Search – into prominent positions on the company's main search result pages. Foundem says the setup is transforming Google's search engine into an "immensely powerful marketing channel" for its own services. Google Product Search (formerly known as Froogle) is a price-comparison service similar to Foundem.

But on another level, Foundem takes aim at the way Google handles manual interventions – and how little information we have about how they work. "The half of our Complaint detailing the anticompetitive power of Google's exclusionary penalties is as much about the systemic failures of Google's manual review process as it is about the legitimacy of the penalty algorithms themselves," Foundem CEO Shivaun Raff says.

Even though Foundem was returned to Google's search results, the company went ahead with its antitrust complaint. The change in ranking shouldn't have taken three years. It shouldn't have required pressure applied through the press. The process, Foundem says, is broken. And when you step back and examine the entire situation, the company contends, you see that Google has an unfair advantage when it expands into new markets, including price-comparison shopping or, well, travel search. Google's acquisition of flight-data outfit ITA Software is currently under review by the Department of Justice.

Raff sums up Foundem's EU complaint like this: "You have an overwhelmingly dominant search engine. If you add to that that search engine's ability to apply discriminatory penalties - they're discriminatory because some services are manually rendered immune through whitelists - and you add the ability of that search engine to preferentially insert its own services at or near the top of the search results, all of that adds up to an unparalleled and unassailable competitive advantage." ®

Similar topics


Other stories you might like

  • Millions of people's info stolen from MGM Resorts dumped on Telegram for free
    Meanwhile, Twitter coughs up $150m after using account security contact details for advertising

    Miscreants have dumped on Telegram more than 142 million customer records stolen from MGM Resorts, exposing names, postal and email addresses, phone numbers, and dates of birth for any would-be identity thief.

    The vpnMentor research team stumbled upon the files, which totaled 8.7 GB of data, on the messaging platform earlier this week, and noted that they "assume at least 30 million people had some of their data leaked." MGM Resorts, a hotel and casino chain, did not respond to The Register's request for comment.

    The researchers reckon this information is linked to the theft of millions of guest records, which included the details of Twitter's Jack Dorsey and pop star Justin Bieber, from MGM Resorts in 2019 that was subsequently distributed via underground forums.

    Continue reading
  • DuckDuckGo tries to explain why its browsers won't block some Microsoft web trackers
    Meanwhile, Tails 5.0 users told to stop what they're doing over Firefox flaw

    DuckDuckGo promises privacy to users of its Android, iOS browsers, and macOS browsers – yet it allows certain data to flow from third-party websites to Microsoft-owned services.

    Security researcher Zach Edwards recently conducted an audit of DuckDuckGo's mobile browsers and found that, contrary to expectations, they do not block Meta's Workplace domain, for example, from sending information to Microsoft's Bing and LinkedIn domains.

    Specifically, DuckDuckGo's software didn't stop Microsoft's trackers on the Workplace page from blabbing information about the user to Bing and LinkedIn for tailored advertising purposes. Other trackers, such as Google's, are blocked.

    Continue reading
  • Despite 'key' partnership with AWS, Meta taps up Microsoft Azure for AI work
    Someone got Zuck'd

    Meta’s AI business unit set up shop in Microsoft Azure this week and announced a strategic partnership it says will advance PyTorch development on the public cloud.

    The deal [PDF] will see Mark Zuckerberg’s umbrella company deploy machine-learning workloads on thousands of Nvidia GPUs running in Azure. While a win for Microsoft, the partnership calls in to question just how strong Meta’s commitment to Amazon Web Services (AWS) really is.

    Back in those long-gone days of December, Meta named AWS as its “key long-term strategic cloud provider." As part of that, Meta promised that if it bought any companies that used AWS, it would continue to support their use of Amazon's cloud, rather than force them off into its own private datacenters. The pact also included a vow to expand Meta’s consumption of Amazon’s cloud-based compute, storage, database, and security services.

    Continue reading

Biting the hand that feeds IT © 1998–2022