Intel came a-knockin' for Cray super interconnects

'Once a Crayon, always a Crayon'

The news that supercomputer maker Cray was selling off its interconnect hardware business to chip giant Intel was a bit of a shocker yesterday, and the top brass at Cray got on the horn with Wall Street bright and early this morning to explain the deal a bit more.

First of all, Peter Ungaro, president and CEO at Cray, confirmed that it was Intel that approached Cray to acquire the interconnect assets. These included designs and other intellectual property as well as 74 employees who worked on the "SeaStar" XT, "Gemini" XE, and future "Aries" chips, which allow for hundreds of thousands of x86 cores and hundreds of teraflops to dozens of petaflops of floating point oomph to be aggregated into a couple hundred cabinets of servers.

"They came to us looking for our technology and skills," Ungaro said on the call.

After being burned a number of times by shipment delays and bugs in Advanced Micro Devices' Opteron processors over the past decade, Cray took a step back with its impending Aries interconnect and moved a little further away from the processor so it could use Intel Xeon chips, GPU coprocessors, FPGAs, or any other kind of accelerator that might be useful in future HPC systems.

The SeaStar and Gemini interconnects were tied very tightly to the HyperTransport point-to-point interconnect on the Opteron processors, basically extending HyperTransport with a high-bandwidth, low-latency network that allows for data to be exchanged across those hundreds of thousands of cores much faster than is possible with Ethernet or InfiniBand networks. The Aries interconnect doesn't rely on HyperTransport or Intel's analog, the QuickPath Interconnect, but rather on PCI-Express links.

Intel has the leg up as the supplier for processors for the "Cascade" systems that Cray is designing for the US Defense Advanced Projects Research Agency – and which are expected to be commercialized and shipping in the first half of 2013 to other supercomputer shoppers – because it has on-chip PCI-Express 3.0 controllers on its Xeon E5-2600 processors. AMD only supports PCI-Express 2.0, which has half the bandwidth, with its current Opteron 6200s and it is not clear when it will support PCI-Express 3.0 links or move controllers on chip as Intel has done with the Xeon E5s. This is obviously bad news for AMD, which will stop making money selling hundreds of thousands of Opterons to Cray customers as soon as the Cascade machines ship. The Nvidia Tesla GPU coprocessors, which have been paired with Opterons inside of Cray XK6 machines, were already eating into Opteron processor sales.

This, among other reasons, might be why AMD bought SeaMicro for $334m. If AMD was going to lose the high-end supercomputer battle against Intel in the Cascade generation at Cray, then the 3D torus/mesh interconnect created by SeaMicro, which can scale to thousands of nodes, could give AMD something to peddle to OEMs chasing smaller HPC shops as well as hyperscale and dense cloud infrastructure.

To a certain way of thinking, Intel's acquisition of Cray's interconnect business is a knee-jerk reaction, but for all we know, Intel started talking to Cray before AMD started talking to SeaMicro. What is clear is that these two chip-makers are facing off and know that integrated interconnects for server processors is the wave of the future.

Cray knows that, too, and Ungaro said as much on the call. "From the technology trends we saw, the hardware was not going to be as important to us in the future as the software trends," Ungaro said. "Our customers understand where the technology trends are going." And that means more and more integration of components to get the low power consumption and low latencies that exascale-class machines are going to require.

Intel's Data Center and Connected Systems Group, which makes processors and chipsets for servers, storage, and networking gear, has been buying up interconnect assets in the past year, and it is clear that Intel was going to do some sort of interconnect – with or without the help of Cray. Intel bought Ethernet ASIC maker Fulcrum Microsystems last July for its 10GE and 40GE chips for switches and routers, and then acquired the InfiniBand biz from QLogic for $125m back in January.

And now it has paid $140m to get Cray's Gemini and Aries interconnect. The deal includes 34 patents and patent applications (Cray has 227 patents and applications at the moment, which seems like a very small number, but there you go) as well as the actual designs for the interconnect chips themselves. Under that deal, Intel will continue to make Gemini chips for Cray and will also make Aries chips for Cray, very likely with the same foundry that Cray is currently using. Cray retains exclusive access to the Gemini and Aries chips, so Intel can't turn around and sell them to any other system maker.

Intel is also hiring the 74 employees who worked on the Gemini and Aries interconnects, which is about 8 per cent of Cray's 925-strong workforce. "Once a Crayon, always a Crayon," Ungaro said on the call to try to buck up the troops, but strictly speaking, this is not true. Sometimes, once a Crayon, now an Intelon.

Ungaro said that the current roadmap for the Aries interconnect and their Cascade systems remains unchanged. DARPA is still going to pay Cray for its development milestones as the development project completes this year and Cray will sell Cascade machines between now and the end of 2016, which is when a new generation of machines, code-named "Shasta" and being co-developed with Intel, will come to market.

For that Shasta generation, Intel will be doing the interconnect development funding, and it may or may not seek funds from Uncle Sam or other governments as it sees fit. Cray will not have to go begging to the US government for interconnect deals, and can concentrate on helping Intel define the characteristics of the Shasta interconnect and coming up with a differentiated software stack and system packaging. After paying some taxes, it will have around $130m of that $140m left over in which to invest in software and packaging and three-and-a-half years of sales in which it can milk its investments (or rather DARPA's) in the Gemini and Aries interconnects.

It must have been pretty clear to Ungaro and his team that Intel and AMD were going to do supercomputer interconnects, with or without the help of Cray, or Cray would not have done this deal. And so perhaps taking the money and making some more for the next three years was probably the right move, financially speaking. In theory, because Cray is not investing in interconnects between now and 2017, it should be a more profitable company going forward. And that Wall Street no doubt will be happy about. But by 2017, Cray will be on the same footing, more or less, as Dell and Hewlett-Packard in the HPC server racket, with a software stack and some packaging expertise, to be sure, riding above whatever Intel cooks up. But the thing is, it won't be the same Cray any more.

Once a Cray, sometimes an Intel. ®