Hadoop World Rackspace Hosting has spent the past two years helping craft the OpenStack cloud control freak and getting it running its public and private cloud services. And now, to get more of the IT wallet, Rackspace wants to peddle more services atop OpenStack, whether it is humming in your shop or in its own glass houses. And one of the first services that Rackspace wants to plunk on top of OpenStack with a price tag on it is, not surprisingly, the Hadoop big data muncher.
As a proponent of full open source cloud control software, it is not surprise at all that Rackspace has partnered up with Hortonworks rather than Cloudera, MapR Technologies, or Greenplum, all of which have some proprietary code in their Hadoop distributions.
To be fair, Greenplum, the big data arm of storage giant EMC, has just opened up its Chorus management tool for Hadoop and data warehouse clusters and the only proprietary Hadoop code it ships is a repackaged version of MapR's M5 Edition called Greenplum MR; you can get a supported version of the Apache Hadoop stack called Greenplum HD, which has no proprietary stuff in it.
Every one of the major Hadoop distributors can trace its roots back to Yahoo!, where the MapReduce computing methodology and distributed file system were cloned from ideas based on systems in use by Google several generations ago to run its search engine.
Cloudera has Doug Cutting, Hadoop's creator, on its payroll as well as many other heavy hitters from inside the former Hadoop factory at Yahoo! But Hortonworks is literally a spin out of the Yahoo! tech team that worked on Hadoop after the search engine company open sourced it under an Apache license in 2006 and has the most stringent view of Hadoop being open source, down to the last bit.
"This is our first foray into Hadoop and a good alignment for us," Huw Edwards, manager of corporate development at Rackspace, tells El Reg, adding that while the deal is by no means exclusive and Rackspace is keeping its options open, and the fact that OpenStack and the Hortonworks Data Platform are completely open source makes both Rackspace and its clients feel more comfortable about not getting vendor lock-in.
Of course, the irony is that if there is only one true open source and supported version of Hadoop, then that is a kind of lock in, too. You need two vendors living on a shoestring to really be free, and the odds are the market will create a couple alternatives.
The details are a bit sketchy right now, but the plan is for Rackspace to work with Hortonworks to streamline the operation of HDP 1.0, which just shipped in June, with the Rackspace Cloud and tuned to work well with OpenStack.
One of the issues the companies are working on is moving data from a set of Cloud Servers, the name for virtual servers on Rackspace's cloudy infrastructure, to its Cloud Files object storage. Rackspace will allow customers to puff up and suck down Hadoop clouds on demand so they don't have to invest in infrastructure over the long haul if their jobs don't demand it.
So, for instance, you might collect telemetry data in real-time and plunk it directly into the Cloud Files service, but only chew on it every couple of hours, days, or weeks depending on your business and the nature of the data. Either way, you only fire up the Cloud Servers to do the chewing when you need to, and when you are not chewing, you can turn off the capacity and not spend money.
This Hadoop public cloud service will be followed by a private cloud offering that will have Rackspace techies manage a private cloud built to the same specs inside your own data center, much as it does with OpenStack infrastructure clouds today.
The Hadoop public cloud service will be available in early 2013, Edwards says, with the private Hadoop cloud service coming sometime in late 2013. You can contact Rackspace now if you want to do a custom engagement before the Hadoop big data service is available, or to get in line for the beta program.
By the way, Rackspace is by no means a newbie to Hadoop. The company has a number of customers using its hosted servers to run Hadoop clusters on bare metal, but as Edwards points out, this requires a significant capital investment on the front end. Rackspace has customers using Cloudera, MapR, Greenplum, and Hortonworks distros on these hosted machines.
Now it will be coming up against Amazon Web Services' Elastic MapReduce service, which runs the Apache Hadoop distribution as a service as well as the MapR M5 Edition. It won't be long – okay, on second thought, maybe it will be long – before Oracle, IBM, Hewlett-Packard, and Dell all fire up Hadoop clusters on their clouds.
With a new CEO and ex-Googler Marissa Mayer at the helm and the company flush with $3.6bn, you have to wonder if Mayer is thinking that maybe letting Hortonworks go was a bad idea. If her former employer demonstrated anything, it was that constantly inventing, smashing, and reinventing server, storage, database, file systems, and other technologies is the only way to extract big profits out of the intertubes. ®