Oak Ridge changes Jaguar's spots from CPUs to GPUs

Nuke lab's super upgraded to a 20-petaflops Titan

GPU will use one-eighth the juice of a CPU to deliver each flop

Scott, you will recall, is the former CTO at Cray and has been intimately involved with several generations of Cray interconnects and system designs. By the end of 2012, when Titan is finished, the Kepler GPUs will account for around 85 per cent of the aggregate flops in the machine, says Scott. The expectation is that a single Kepler GPU coprocessor will deliver 1 teraflops of double-precision. But the important thing, says Scott, is that a GPU delivers each flop for about one-eighth the juice of a CPU.

"I am not dissing CPUs," says Scott. "But they are designed to run single threads as fast as possible, which will be important for serial work. But we need something that is fundamentally low power for calculations, and that is the GPU."

Oak Ridge is publicly hedging on the flops that Titan can deliver, saying the machine will be between 10 and 20 petaflops of peak theoretical performance. But clearly, with 18,688 Kepler GPUs, you get 18.7 petaflops right there. And there's probably going to be at least 3 petaflops on the Opteron side. That gets you to over 20 petaflops peak. The reason that Cray, Nvidia, and Oak Ridge are hedging on the petaflops is that on Linpack tests done on several large ceepie-geepie systems thus far, more than half of the peak gigaflops in the system have gone up the chimney.

Oak Ridge has secured $97m in funding to upgrade Jaguar to Titan. The upgrade from the XT5 nodes to the XK6 nodes and the swapping out of the old Opteron processors for the new Opteron 6200s will be completed this year and generate $60m of revenues for Cray.

The addition of the Kepler GPUs plus other services to be performed by Cray represent the remainder of the contract, which will be completed in the second half of 2012. (Maybe those Kepler GPUs aren't coming this year, eh?) The contract has upgrade options beyond these two phases, which probably include swapping out Opteron processors or GPUs, which could generate even more money for Cray and/or Nvidia.

ORNL's "Titan" OLCF-3 system is two steps away from exaflops (click to enlarge)

Oak Ridge has plans to get to get well above 20 petaflops. The OLCF-4 system, which is due in 2015, will scale from 100 to 250 petaflops and will be based on the "Cascade" system design that the US Defense Advanced Research Projects Agency is paying Cray to develop. The Cascade system will use the future "Aries" interconnect and will link nodes and coprocessors together through PCI-Express links and will support both AMD Opteron and Intel Xeon processors as compute nodes. It will probably also support a variety of coprocessors, including GPUs and FPGAs, too. Out in 2018, with the OLCF-5 design, it looks as if Cray is moving to a new network structure and hopes to hit an exaflops.

And yes, it will be able to play Crysis. But will it be able to play budget crisis? That is the real question. ®