Intel just installed itself in Hadoop guru Cloudera – and why their rivals should be worried

Chipzilla's distro shelved, cash injected, exec in place


Analysis Intel is canning its own Hadoop distribution in favor of using Cloudera's flavor of the analytics software. And the chip giant is making a "significant equity investment" in the upstart that's higher than had been reported elsewhere.

This strategic move was announced by Intel and Cloudera on Thursday following years of heavy investment by Chipzilla into its own Intel Distribution for Apache Hadoop, which will now be put to sleep. Some of its code will be merged into Cloudera's tech.

The partnership is a big bet by Intel that the future of data storage and computation lies in the Hadoop data processing framework – an open-source technology originally developed by engineers at Yahoo! in 2005 in response to two Google papers describing software it had built to deal with a deluge of online data.

One of the creators of Hadoop, Doug Cutting, works for Cloudera, which formed in 2009 to commercialize the technology. These days, Cloudera can be found helping companies such as AMD shift workloads out of traditional systems like Oracle Database and into Hadoop.

As a result of the partnership, Intel will keep its engineers assigned to Hadoop development, and put an exec on Cloudera's board of directors, as well as buying a stake in the upstart.

"Hadoop will be the application that is deployed across more servers than any other enterprise application," said Intel's data center group chief Diane Bryant on a webcast discussing the coupling.

Cloudera announced last week that it raised $160m in a round of funding from companies including T Rowe Price and Google Ventures.

Next week, Cloudera will announce the completion of an additional round of fundraising that will include an investment from Intel. The chip maker has already put out a statement about that, noting: "This is Intel's single largest data center technology investment in its history."

A Venturebeat scoop on Wednesday indicated Intel will pour as much as $90m into the company, but we've heard it's far more.

"I believe those people are underestimating the amount of investment," Jason Waxman, chief of Intel's cloud infrastructure group, told El Reg on Thursday. "The rumor's floating are inaccurate," Cloudera chief Tom Riley told us when we asked about the figure.

There, there, Oracle. Dry your tears on these dollar bills

Placing an as-yet-unnamed Intel executive on the board of Cloudera will have huge ramifications for other IT giants keen to exercise influence over the software distribution.

That board-level move will give x86 overlord Intel valuable intelligence and tighten its grip on the steering wheel of Hadoop, perhaps guiding the software away from other processor architectures, such as those designed by Oracle and IBM (SPARC/Power) and ARM.

When we asked Cloudera what would happen, hypothetically, if a non-x86 chip company came up with a briefcase of cash wanting a board seat.

"That hypothetical would be awkward," Riley said. "We're partnering with the industry leader. The industry leader has given us enough money."

Almost all of Cloudera's customers are running on Intel hardware, Riley said.

ClouderaIntel

Hadoop troupe ... Intel's Diane Bryant and Cloudera's Tom Riley

The partnership means Intel and Cloudera will work together to add some of the software the chipmaker wrote for its own distribution to Cloudera's source code base, as well as the main Apache Hadoop project.

"We've taken two great distributions and are combining the best of them into one," said Riley on the webcast. "Intel is bringing engineers, Intel is bringing field resources to this relationship."

There will be "lots and lots of engineers on both sides," Riley added to El Reg afterwards. "We're building a joint strategic partnership office team members assigned to it."

Intel will hook the Hadoop code up to architecture extensions in its chips, such as Trusted Execution Technology, tightening Cloudera's hardware integration. It also plans to do a lot of work tying Hadoop into its "Internet of Things" low-power silicon as well, Waxman told us.

Cloudera has steadily increased its share of the Hadoop market through a combination of expertise and marketing. It competes with rivals like Hortonworks, MapR and VMware's spinoff Pivotal.

Intel's move is significant as it's trying to exercise control over an open-source technology that has the potential to boost data-processing capabilities within data centers and subtly shape the hardware market. Just as Linux shook up how servers and server clusters are built, Hadoop may do the same for processors and Intel wants to be there. ®

Broader topics


Other stories you might like

  • Florida's content-moderation law kept on ice, likely unconstitutional, court says
    So cool you're into free speech because that includes taking down misinformation

    While the US Supreme Court considers an emergency petition to reinstate a preliminary injunction against Texas' social media law HB 20, the US Eleventh Circuit Court of Appeals on Monday partially upheld a similar injunction against Florida's social media law, SB 7072.

    Both Florida and Texas last year passed laws that impose content moderation restrictions, editorial disclosure obligations, and user-data access requirements on large online social networks. The Republican governors of both states justified the laws by claiming that social media sites have been trying to censor conservative voices, an allegation that has not been supported by evidence.

    Multiple studies addressing this issue say right-wing folk aren't being censored. They have found that social media sites try to take down or block misinformation, which researchers say is more common from right-leaning sources.

    Continue reading
  • US-APAC trade deal leaves out Taiwan, military defense not ruled out
    All fun and games until the chip factories are in the crosshairs

    US President Joe Biden has heralded an Indo-Pacific trade deal signed by several nations that do not include Taiwan. At the same time, Biden warned China that America would help defend Taiwan from attack; it is home to a critical slice of the global chip industry, after all. 

    The agreement, known as the Indo-Pacific Economic Framework (IPEF), is still in its infancy, with today's announcement enabling the United States and the other 12 participating countries to begin negotiating "rules of the road that ensure [US businesses] can compete in the Indo-Pacific," the White House said. 

    Along with America, other IPEF signatories are Australia, Brunei, India, Indonesia, Japan, South Korea, Malaysia, New Zealand, the Philippines, Singapore, Thailand and Vietnam. Combined, the White House said, the 13 countries participating in the IPEF make up 40 percent of the global economy. 

    Continue reading
  • 381,000-plus Kubernetes API servers 'exposed to internet'
    Firewall isn't a made-up word from the Hackers movie, people

    A large number of servers running the Kubernetes API have been left exposed to the internet, which is not great: they're potentially vulnerable to abuse.

    Nonprofit security organization The Shadowserver Foundation recently scanned 454,729 systems hosting the popular open-source platform for managing and orchestrating containers, finding that more than 381,645 – or about 84 percent – are accessible via the internet to varying degrees thus providing a cracked door into a corporate network.

    "While this does not mean that these instances are fully open or vulnerable to an attack, it is likely that this level of access was not intended and these instances are an unnecessarily exposed attack surface," Shadowserver's team stressed in a write-up. "They also allow for information leakage on version and build."

    Continue reading
  • A peek into Gigabyte's GPU Arm for AI, HPC shops
    High-performance platform choices are going beyond the ubiquitous x86 standard

    Arm-based servers continue to gain momentum with Gigabyte Technology introducing a system based on Ampere's Altra processors paired with Nvidia A100 GPUs, aimed at demanding workloads such as AI training and high-performance compute (HPC) applications.

    The G492-PD0 runs either an Ampere Altra or Altra Max processor, the latter delivering 128 64-bit cores that are compatible with the Armv8.2 architecture.

    It supports 16 DDR4 DIMM slots, which would be enough space for up to 4TB of memory if all slots were filled with 256GB memory modules. The chassis also has space for no fewer than eight Nvidia A100 GPUs, which would make for a costly but very powerful system for those workloads that benefit from GPU acceleration.

    Continue reading
  • GitLab version 15 goes big on visibility and observability
    GitOps fans can take a spin on the free tier for pull-based deployment

    One-stop DevOps shop GitLab has announced version 15 of its platform, hot on the heels of pull-based GitOps turning up on the platform's free tier.

    Version 15.0 marks the arrival of GitLab's next major iteration and attention this time around has turned to visibility and observability – hardly surprising considering the acquisition of OpsTrace as 2021 drew to a close, as well as workflow automation, security and compliance.

    GitLab puts out monthly releases –  hitting 15.1 on June 22 –  and we spoke to the company's senior director of Product, Kenny Johnston, at the recent Kubecon EU event, about what will be added to version 15 as time goes by. During a chat with the company's senior director of Product, Kenny Johnston, at the recent Kubecon EU event, The Register was told that this was more where dollars were being invested into the product.

    Continue reading
  • To multicloud, or not: Former PayPal head of engineering weighs in
    Not everyone needs it, but those who do need to consider 3 things, says Asim Razzaq

    The push is on to get every enterprise thinking they're missing out on the next big thing if they don't adopt a multicloud strategy.

    That shove in the multicloud direction appears to be working. More than 75 percent of businesses are now using multiple cloud providers, according to Gartner. That includes some big companies, like Boeing, which recently chose to spread its bets across AWS, Google Cloud and Azure as it continues to eliminate old legacy systems. 

    There are plenty of reasons to choose to go with multiple cloud providers, but Asim Razzaq, CEO and founder at cloud cost management company Yotascale, told The Register that choosing whether or not to invest in a multicloud architecture all comes down to three things: How many different compute needs a business has, budget, and the need for redundancy. 

    Continue reading

Biting the hand that feeds IT © 1998–2022