Revolution speeds stats on Hadoop clusters

R language teaches 'meaningful' math to elephants


Revolution Analytics, the company that is extending R, the open source statistical programming language, with proprietary extensions, is making available a free set of extensions that allow its R engine to run atop Hadoop clusters.

Now statisticians that are familiar with R can do analysis on unstructured data stored in the Hadoop Distributed File System, the data store used for the MapReduce method of chewing on unstructured data pioneered by Google for its search engine and mimicked and open sourced by rival Yahoo! as the Apache Hadoop project.

R can now also run against the HBase non-relational, column-oriented distributed data store, which mimics Google's BigTable and which is essentially a database for Hadoop for holding structured data. Like Hadoop, HBase in an open source project distributed by the Apache Software Foundation.

With MapReduce, unstructured data is broken up and spread across server nodes, where the data is mapped across the nodes (with replication at multiple points for performance as well as fault tolerance) and chewed on in parallel (that's the reduce part) rather than in series like you would have to do on a single machine.

Revolution R Hadoop logo combo

With the marriage of R and Hadoop, explains David Champagne, chief technology officer at Revolution Analytics, the R engine is installed atop each Hadoop node in the cluster. Instead of programming a reduction algorithm in Java, as you do in Hadoop, you set up an R algorithm from an R workstation, and it is parsed out to the Hadoop nodes by Hadoop's mapping function. Statistical analysis is thus done in parallel on the data is stored in HDFS.

You don't do MapReduce and then extract data that comes back to the workstation for the analysis, but you chew on data right where it is in the cluster, and then aggregate it. In essence, R is using Hadoop as a grid controller, managing where specific algorithms run and the data they run against.

"We allow not just sums, means, and averages, which can be done easily in Java or Python, but statistically meaningful analysis," says Champagne. And you don't have to know jack about Java or MapReduce to run an R algorithm against a Hadoop cluster with either HDFS or HBase as its data store. "We want to hide some of the complexity of the MapReduce approach from R programmers and statisticians."

The tool that makes this integration possible is called RevoConnectR for Apache Hadoop, and Champagne tells El Reg that it was tested against Cloudera's CDH3 commercial distribution of Hadoop combined with the Revolution R Enterprise 4.3 stats engine. But you can take the Revolution R Community Edition and plunk it down on an open source Hadoop cluster (from Apache or from Cloudera) and the Hadoop connector for R will also work.

You can download the R connector for Hadoop from GitHub. Both Revolution Analytics and Cloudera want to encourage customers to use their commercial releases – and pay for tech support, of course. But this R-Hadoop connector is supplied for free even if they do. ®


Other stories you might like

  • Alcatel-Lucent Enterprise adds Wi-Fi 6E to 'premium' access points
    Company claims standard will improve performance in dense environments

    Alcatel-Lucent Enterprise is the latest networking outfit to add Wi-Fi 6E capability to its hardware, opening up access to the less congested 6GHz spectrum for business users.

    The France-based company just revealed the OmniAccess Stellar 14xx series of wireless access points, which are set for availability from this September. Alcatel-Lucent Enterprise said its first Wi-Fi 6E device will be a high-end "premium" Access Point and will be followed by a mid-range product by the end of the year.

    Wi-Fi 6E is compatible with the Wi-Fi 6 standard, but adds the ability to use channels in the 6GHz portion of the spectrum, a feature that will be built into the upcoming Wi-Fi 7 standard from the start. This enables users to reduce network contention, or so the argument goes, as the 6GHz portion of the spectrum is less congested with other traffic than the existing 2.4GHz and 5GHz frequencies used for Wi-Fi access.

    Continue reading
  • Will Lenovo ever think beyond hardware?
    Then again, why develop your own software à la HPE GreenLake when you can use someone else's?

    Analysis Lenovo fancies its TruScale anything-as-a-service (XaaS) platform as a more flexible competitor to HPE GreenLake or Dell Apex. Unlike its rivals, Lenovo doesn't believe it needs to mimic all aspects of the cloud to be successful.

    While subscription services are nothing new for Lenovo, the company only recently consolidated its offerings into a unified XaaS service called TruScale.

    On the surface TruScale ticks most of the XaaS boxes — cloud-like consumption model, subscription pricing — and it works just like you'd expect. Sign up for a certain amount of compute capacity and a short time later a rack full of pre-plumbed compute, storage, and network boxes are delivered to your place of choosing, whether that's a private datacenter, colo, or edge location.

    Continue reading
  • Intel is running rings around AMD and Arm at the edge
    What will it take to loosen the x86 giant's edge stranglehold?

    Analysis Supermicro launched a wave of edge appliances using Intel's newly refreshed Xeon-D processors last week. The launch itself was nothing to write home about, but a thought occurred: with all the hype surrounding the outer reaches of computing that we call the edge, you'd think there would be more competition from chipmakers in this arena.

    So where are all the AMD and Arm-based edge appliances?

    A glance through the catalogs of the major OEMs – Dell, HPE, Lenovo, Inspur, Supermicro – returned plenty of results for AMD servers, but few, if any, validated for edge deployments. In fact, Supermicro was the only one of the five vendors that even offered an AMD-based edge appliance – which used an ageing Epyc processor. Hardly a great showing from AMD. Meanwhile, just one appliance from Inspur used an Arm-based chip from Nvidia.

    Continue reading

Biting the hand that feeds IT © 1998–2022