Some stories just take forever to come true. 30 months ago, we revealed Google was going to introduce a weblog search engine - and this week, it finally did. The story, so obvious in retrospect, barely merits the term 'scoop'. But now, as then, it has been eclipsed by a raging debate about the implications for bloggers and for the web in general.
A great many people see this as the perfect opportunity to improve Google search - and introduce some innovation into the world of web user interface navigation - by removing weblogs from the main Google index, and giving them their own tab, as Usenet enjoys now.
Google, along with rival search engines which aped its link based algorithms, has to wrestle with the constantly evolving techniques deployed to trick it into promoting certain web pages. It's an arms race comparable to email spam, and one of the chief culprits is 'blog noise' - a catch-all term for the irrelevant blog entries and all the extraneous plumbing that props them up: RSS feeds, empty pages, duplicate pages, TrackBacks, and so on.
This inadvertently has the effect of skewing Google's results to promote the irrelevant pages, with a side consequence of conferring instant web celebrity on some of the bloggers. A huge and lucrative industry of SEOs sprang up overnight, mimicking many of the techniques bloggers had used to gain advantage, and Google began to index pages almost constantly. Google's index reached its nadir in October 2003, when almost every search result in the Top Ten for certain queries was an empty and useless page - in each case, the culprit being blog-related.
Readers pegged it as a "life or death" issue for Google. But to exclude anything messy, such as blogs or product catalogs, from the main index risks antagonizing vested interests. Some internet utopians have placed so much faith in blogs, and identify so strongly with them, that the act of exclusion is like being unplugged from a life support system. And maybe for such people, as Jaron Lanier has suggested, there is no core of subjective real world experience to fall back on, once the electronic representation of self has floated away.
Weblog evangelists like to see fellow practitioners as very special flowers indeed - uniquely representative of the rest of us, the one true authentic voice of the people, and the very cortex of the web's "hive mind". Remove weblog chatter from the web, they argue, and you lose the very essence of the internet. Like er... cat pictures.
Since the blog fad began the US media has been complicit. This has less to do with a new found infatuation with electronics networks and more to do with the fact that the professional media here is monopolistic and owned by the same corporations which own the US government, and will do absolutely anything to avoid improving the quality of its product So it's promoted, rather patronizingly, bloggers as unique citizens with almost telepathic properties. Last year's Presidential election regularly cut away to "hear what the bloggers are saying," as if it was some secret communication channel, populated by the highly evolved. One blogger yesterday encapsulated the anxieties about being estranged from Google's main search index very nicely in comments left at the website of Google hagiographer John Battelle:
"From a utilitarian perspective and a 'blogger' myself, I suppose I should be very excited about this move. But instead, I'm unhappy about what I perceive to be a rather slippery slope... moving more towards filtering by structure rather than message," he writes.
Alas, search engines have always filtered by structure, rather than the message, and so they should. Just as Usenet benefitted from being a web interface that respected its underlying structure, so could blogs. Blogs are so atomized they desperately need better navigation - and here, clearly, Google can help.