Sphinx - text search The Pirate Bay way

Like MySQL. But it can scale

Vowels are a scare resource

Before Sphinx, the other option for text search was Apache Solr. Solr, whose name-giver understands that vowels are a scarce resource and must be used sparingly, is a server that sits on top of Lucene. Solr is popular with the enterprise crowd, who love its Java. Being a Java program, Solr includes no shortage of technology whose acronyms contain the letters J and X.

This tickles the enterprise pink, because these sorts of developers love nothing more than hanging out around a whiteboard drawing boxes and arrows and, from time to time, writing XML to make it look like they're doing real work. Solr thrives in this environment, being an Apache Foundation project, the Apache Foundation, of course, widely known as a cruel experiment to see what happens when bureaucrats do open source.

Being backed by Lucene, Solr is bound by Lucene. It is easier to tune Solr for relevancy than it is to tune MySQL, but it still doesn't provide the search quality to justify its popularity. That, and Solr has scalability problems. Lucene was originally designed as an embeddable library, so scalability wasn't a top priority. As a network layer on top of Lucene, Solr must accommodate for this. "In my internal benchmarks, Solr routinely performed 2x-4x slower [than Sphinx], both for indexing and searching," Aksyonoff tells The Reg.

Sphinx takes a different approach to search. For example, using MySQL or Solr, each query will be trimmed of its stopwords. This is a method out of the information retrieval books, where you remove low-signal words (like "the" or "and") that can degrade search quality. This was one of Aksyonoff's original problems searching song lyrics, a query like "I feel you" would yield Cuil-like results. By default, Sphinx does not remove stopwords (but it can be configured to) and ranks results based on phrase matching.

Having its roots in database search, Sphinx speaks MySQL, which means you can connect to its search server with a MySQL client and issue queries that look very similar to SQL. It will easily index data out of a MySQL or Postgres databases, but also supports an XML format for arbitrary data. There are also clients for PHP, Python, Ruby, and any other language that's worth using. Scalability is no issue, both in searching and indexing: Sphinx can index 10 megabytes of data per second and can search up to 100 gigabytes of text on a single processor. It also supports multi-machine distributed searching, as in the case of Craigslist.

In the vicious meritocracy of open source software, the only yardstick of success is the distribution of your code. In this regard, Sphinx still has some way to go, but there's still profit to be made. High-volume installations at Craigslist and The Pirate Bay are a validation for Aksyonoff, who now makes his living on Sphinx.

"We aren't exactly bathing in money, but on the other hand, we're paying the salaries from our revenues instead of burning venture capital and have always been," he says. His company, Sphinx Technologies, provides consulting and support services, as well as custom development. Judging by how well Sphinx is performing in real-world installations, for Aksyonoff, the General Public Business Model might actually pay off.

You can download Sphinx here. ®

Ted Dziuba is a co-founder at Milo.com You can read his regular Reg column, Fail and You, every other Monday.