IBM stamps its pedal to the metal: Spark flies onto Big Blue's Bluemix

Citizen analysts and Excel junkies please take note

IBM has made its planned Apache Spark service available on its Bluemix cloud platform.

The service was announced in June, when IBM pledged 3,500 staffers to Apache's analytics project, with Spark also becoming the basis of IBM's analytics and commerce platforms and its Watson Health Cloud.

Spark-as-a-service will be available on a pay-as-you-go basis with the ability to reserve instances, but there won't be volume discounts for big firms.

According to IBM, Spark is like MapReduce – an architecture running at near infinite scale that revolutionizes data processing for the masses. Spark is, of course, beginning to replace MapReduce for some.

No more proprietary boxes and chipsets or need to engage rocket scientists building high-performance clusters; this is analytics for the generation who don't like paying enterprise prices and for whom commodity x86 clusters is king.

But a lot's changed since stately IBM anointed Spark, which was created in 2009 and accepted into the Apache hall of open-source fame in 2013.

Amazon has revealed it's working on its own analytics-as-a-service Space Needle, which was revealed earlier this month for AWS. Amazon has been moving into the enterprise for some time, displacing the established infrastructure vendors – vendors like IBM. Now it's going up a level, into applications and cash-rich BI and analytics apps at that.

S3, Redshift, and Aurora have been the building blocks. Last week Amazon told Wall Street Aurora is now its fastest growing service ever – having overtaken Redshift. Aurora is AWS's MySQL-compatible database engine.

Such are the waves AWS has created that SAP was forced into pre-announcing its analytics-as-a-service, Project Orca.

Analytics isn't a friendly world for users. The apps are pricey, there are lots of high-priced consultants, and it's a bit like ERP or CRM in that the final software never quite lives up to original expectations. The second you deploy it, it's out of date.

AWS argues that it's democratizing things – thanks to its low, low prices, ease of adoption, and flexibility of architecture. Amazon has forced IBM to compete in its own back yard, the enterprise, in a game it's long thought its own: data and analytics.

So how does IBM hope to arrest that giant sucking sound that is data going into Jeff Bezos' cloud?

A gamble by Big Blue?

IBM's rolling Spark-as-a-service not just at the suits plugging away at their BI and analytics dashboards, but also at those building tools and partnering. The open-source nature of Spark, therefore, is important because anybody can build code and tools that plug into other Spark-based tools and frameworks.

In this respect, IBM is gambling it can repeat the success of the Eclipse tools framework from the early 2000s. IBM threw its weight into Eclipse, turning it into the framework for its Rational tools and achieving a major coup in the process: It cut the price of building its own Rational IDE and pulled rivals around into supporting the Eclipse framework and, thereby, its own platform and runtimes.

Eclipse raised an army of ISVs building tools to the open-source code that was IBM-ready. But open cuts little ice in the cloud of today, and that's where IBM's theory and past experience of Eclipse breaks down.

OpenStack was intended as the open alternative to AWS, but OpenStack public clouds are folding – Hewlett-Packard was the latest to bring down the shutters.

Proprietary platforms such as AWS are winning, as long as they are clothed in support for open-source operating systems, languages, and middleware.

No one was ever fired for buying IBM open software, right?

Derek Schoettle, IBM Cloud Data Services general manager, countered that open is important: AWS is proprietary and incomplete. Spark, with IBM, means not just choice, but also the opportunity to flesh out a platform more complete, from the user interface through to the processing engine and everything in between lacking in AWS.

However, unlike with Eclipse, IBM in no way can be considered the standard bearer to which all others are forced to rally. Before IBM, Cloudera, Intel, DataBricks, and MapR – to name but a few – were already committed to Spark. Moreover, Spark is available on AWS. So what exactly is IBM's differentiator this time?

"IBM's been around for a long time and has had success working with large organizations, and moving huge organizations and their data to a mixed or a purely private [cloud] estate," Schoettle told The Reg. "We have a full complement of offerings that lets enterprises move some or all or part of that environment to the cloud."

Also, IBM – while working with the traditional enterprise customers – wants to attract new, smaller firms who'd normally not consider working with IBM. "We are focused on attracting new customers. We can now partner with small and fast-moving organizations," Schoettle said. "The interest is from IBM's existing base of customers and also communities who've not worked with IBM in the past; the attractive part is that IBM is open – the Spark community."

Armed with Spark, Schoettle reckons IBM can win data scientists and "citizen analysts" who would have used Excel plus data visualization tools.

Just how much does IBM expect to be able to make from such grassroots? It's not saying. "It's early in this game," Schoettle said. "We feel very good about the next 5-10 years and our role in analytics and the cloud opportunity."

IBM won't lose out on Spark. It's just got a lot of ground to make up convincing the "as-a-service" converts it has something they need as much as IBM does. ®