Cloudera, MapR, IBM, and Intel bet on Spark as the new heart of Hadoop

Screeching sound heard amid Spark stampede as Cloudera changes opinion on Hive


Cloudera has rallied four major companies behind a scheme to tie two open source projects together for the benefit of the Hadoop community.

The partnership between Cloudera, IBM, Intel, DataBricks, and MapR to port Apache Hive onto Apache Spark is due to be announced this week at the Spark Summit in San Francisco. El Reg heard rumors of it last week after stumbling across a proposal by Cloudera to lift Hive onto Spark.

For those not familiar with the galaxy of codenames in the Hadoop world: Spark is a general-purpose cluster computing system originally developed at the University of California, Berkeley and based on the Hadoop File System. It can be used as an alternative data processor to Hadoop MapReduce and is billed as being around 100 times faster than MapReduce when running in memory or 10 times faster when running on disk.

Hive, meanwhile, is data warehouse software that uses a SQL-like language to query data stored in Hadoop.

Both projects are important, with Spark seen by many as a potential successor to MapReduce and Hive viewed as a likely candidate for accomplishing SQL-on-Hadoop work.

By lifting Hive up onto Spark, Cloudera & Co are hoping to force some consolidation in the labyrinthine Hadoop ecosystem, and in doing so are placing less emphasis on one of Cloudera's own projects, Impala.

In a chat with El Reg, Justin Erickson, Cloudera's director of product management, said the company has decided to push Hive because it wants to "go and combine the forces of the Spark community with the Hive community to make batch processing [in Hadoop] faster."

"Hive is the standard for doing batch on Hadoop right now," said Matt Brandwein, the company's head of product marketing. "We want to reduce fragmentation in the community. People are getting a bit aware of the fact there are so many options for so many different objects. Spark is the successor."

The move has big ramifications for the Hadoop ecosystem, and for Cloudera. In the past, Cloudera has been skeptical of the value of Hive. In a blog post late last year, Mike Olson, the company's chief strategy officer, wrote, "Decades of experience had taught people to expect real-time responses from their databases. Hive, built on MapReduce, couldn't deliver."

To address the perceived shortcomings of Hive, Cloudera built its own software, Impala. But with the new partnership between Cloudera, MapR, Databricks, and Intel, it seems like Cloudera has warmed to Hive and will use the tech as its main way of dealing with the wider Hadoop community, while continuing to develop Impala as a way to make a buck.

Another little complication in this story is that there already is a Hive-on-Spark project called Shark. But Cloudera feels that Shark has diverged too much from mainstream Hive.

"Shark took an approach of replacing several key components of Hive, including the query planner and other elements of Hive," Cloudera explained. "The result of this was that maintaining compatibility with Hive became very difficult as changes to Hive can not be transparently back-ported to Shark. With the Hive-on-Spark approach, we are making a much more limited change to only the physical query planner, which means that the Hive community can make changes and add new functionality to Hive and have this transparently work with either Spark or MapReduce or Tez. As such, the maintenance burden will be much lower for Hive on Spark and will be more deeply integrated with the core Hive community."

Speaking of Tez, Cloudera's move also puts pressure on Hortonworks, which helped develop the competing data-processing framework. But Cloudera says Spark, like Tez, is merely an option.

As the company explains in an FAQ document, "It is not a goal for the Spark execution backend to replace Tez or MapReduce. It is healthy for the Hive project for multiple backends to coexist. Users have a choice whether to use Tez, Spark or MapReduce. Each has different strengths depending on the use case. And the success of Hive does not completely depend on the success of either Tez or Spark."

When contacted for comment, Hortonworks said the decision to pour development resources into Hive on Spark is broadly a good thing. "It's an admission that the open source community driven model is the right one," Shaun Connolly, the company's VP of strategy, said in a chat with El Reg.

Another way of interpreting all of this is that now that Cloudera has raised $900m in funding – $740m of which came from Intel – it's trying to throw its weight around and take more of a leading role in the Hadoop community.

By taking charge of some of the planets in the the galaxy of open source projects associated with Hadoop, Cloudera can develop a better understanding of the future direction of the software and more carefully hone its business to reap cash off the growing user base. ®

Similar topics


Other stories you might like

  • Prisons transcribe private phone calls with inmates using speech-to-text AI

    Plus: A drug designed by machine learning algorithms to treat liver disease reaches human clinical trials and more

    In brief Prisons around the US are installing AI speech-to-text models to automatically transcribe conversations with inmates during their phone calls.

    A series of contracts and emails from eight different states revealed how Verus, an AI application developed by LEO Technologies and based on a speech-to-text system offered by Amazon, was used to eavesdrop on prisoners’ phone calls.

    In a sales pitch, LEO’s CEO James Sexton told officials working for a jail in Cook County, Illinois, that one of its customers in Calhoun County, Alabama, uses the software to protect prisons from getting sued, according to an investigation by the Thomson Reuters Foundation.

    Continue reading
  • Battlefield 2042: Please don't be the death knell of the franchise, please don't be the death knell of the franchise

    Another terrible launch, but DICE is already working on improvements

    The RPG Greetings, traveller, and welcome back to The Register Plays Games, our monthly gaming column. Since the last edition on New World, we hit level cap and the "endgame". Around this time, item duping exploits became rife and every attempt Amazon Games made to fix it just broke something else. The post-level 60 "watermark" system for gear drops is also infuriating and tedious, but not something we were able to address in the column. So bear these things in mind if you were ever tempted. On that note, it's time to look at another newly released shit show – Battlefield 2042.

    I wanted to love Battlefield 2042, I really did. After the bum note of the first-person shooter (FPS) franchise's return to Second World War theatres with Battlefield V (2018), I stupidly assumed the next entry from EA-owned Swedish developer DICE would be a return to form. I was wrong.

    The multiplayer military FPS market is dominated by two forces: Activision's Call of Duty (COD) series and EA's Battlefield. Fans of each franchise are loyal to the point of zealotry with little crossover between player bases. Here's where I stand: COD jumped the shark with Modern Warfare 2 in 2009. It's flip-flopped from WW2 to present-day combat and back again, tried sci-fi, and even the Battle Royale trend with the free-to-play Call of Duty: Warzone (2020), which has been thoroughly ruined by hackers and developer inaction.

    Continue reading
  • American diplomats' iPhones reportedly compromised by NSO Group intrusion software

    Reuters claims nine State Department employees outside the US had their devices hacked

    The Apple iPhones of at least nine US State Department officials were compromised by an unidentified entity using NSO Group's Pegasus spyware, according to a report published Friday by Reuters.

    NSO Group in an email to The Register said it has blocked an unnamed customers' access to its system upon receiving an inquiry about the incident but has yet to confirm whether its software was involved.

    "Once the inquiry was received, and before any investigation under our compliance policy, we have decided to immediately terminate relevant customers’ access to the system, due to the severity of the allegations," an NSO spokesperson told The Register in an email. "To this point, we haven’t received any information nor the phone numbers, nor any indication that NSO’s tools were used in this case."

    Continue reading

Biting the hand that feeds IT © 1998–2021