Microsoft, Facebook, Google box clever on really big systems

The enemy of my enemy...

Facebook's decision to release under open source a large-scale data management project similar to - and inspired by - Google's BigTable has received backing from an unusual quarter: Microsoft.

Data center futures architect and distinguished database developer James Hamilton, has complemented the pimply faced social network for releasing what he said "looks like a well-engineered system."

Hamilton spent ten years at IBM working on DB2 before joining Microsoft in 1997 to work on SQL Server, and recently collaborated with fellow database guru Michael Stonebraker on an examination of future database architectures.

Stonebraker put the cat among the pigeons by slagging off Google's MapReduce database tool earlier this year - he called it a step backwards. Stonebraker also noted there are limitations in BigTable and its open-source equivalent Hbase.

Conspiracy theorists might find it interesting that Stonebraker's co-author David de Witt joined Microsoft in April to head up a new research effort into large databases. They should also remember Microsoft is an investor in Facebook.

Adding insult to injury, Facebook has put its project - called Cassandra - up on Google Code. Cassandra is not alone on Google code. Another BigTable clone called Hypertable was set up on Google Code earlier this year.

Like BigTable, Cassandra is designed to get round the limitations of traditional relational databases in large-scale, online applications.

Cassandra is the work of a Facebook team led by Jeff Hammerbacher, an ex-Harvard student who was recruited by Facebook chief Mark Zuckerberg from troubled Wall Street bank Bear Stearns.

Hammerbacher is now reported to have become the latest Faceboook employee to have served his notice. Can we expect this Facebook youf to show up in well-remunerated style at Redmond this Fall?

And, will all this prompt Google to put BigTable - or even MapReduce - into open source? Given it decided to release its Protocol Buffers technology to open source this week - its not beyond the bounds of possibility.

There is a precedent: Facebook released Thrift, its clone of Protocol Buffers, to open source last year.

The games continue.®