Cray previews XT6 Opteron nodes

Cash for number clunkers

SC09 The SC09 supercomputing trade show finished up late last week, but El Reg still has a bunch of things to tell you about. One of them is a preview of the Opteron blade server for the upcoming high-end XT6 and midrange XT6m supercomputers from Cray.

Like all of the progeny of the Red Storm supercomputer that Cray built for Sandia National Laboratories in the mid-2000s, the future XT6 and XT6m supers will be based on Opteron processors from Advanced Micro Devices and a high-speed interconnect called Seastar created by Cray that lashes all of the processors together through their HyperTransport links in a 2D or 3D torus mesh.

Like the XT4 and XT5 machines, the XT6 will use the SeaStar2+ interconnect, and as El Reg previously reported, Cray shops will have to wait until the second half of 2010 to get their hands on a lower-latency, higher-bandwidth interconnect (presumably to be called SeaStar3, but Cray has not said). The XT5 machines are based on the current quad-core Shanghai 8300 and six-core Istanbul 8400 Opterons, but the upcoming XT6 blades will use the Magny-Cours Opteron 6100s.

The XT5 and XT6 family of blades are very similar. A blade can have four or eight processor sockets and has a removable module for the SeaStar2+ interconnect so the mesh electronics can be upgraded separately from the processors and memory. In the photo of the XT6 blade below, the interconnect module has been removed from the blade and placed behind it (the chunk of electronics with four brassy squares). This interconnect module plugs into the blade in the green area behind the DDR3 main memory and to the right of the bank of voltage regulators.

The Cray XT6 Opteron 6100 blade node

This particular XT6 blade has eight processor sockets - four are under the heat sinks in the middle of the blade, four more have had their heat sinks removed - and supports up to 64GB of main memory for each part of sockets, which are configured as a compute node. (There are four SeaStar2+ processors, one for each pair of processors.) Using the Opteron 6100s, which will come in eight- and twelve-core flavors, this blade can support 64 or 96 cores across four nodes.

The SeaStar2+ interconnect used with the XT6 allows for a 3D torus mesh to be created between the nodes, and scales to hundreds of cabinets if customers have a big enough checkbook. (This interconnect is, in fact, used in the Jaguar massively parallel supercomputer that just took the pole position in the Top 500 supercomputer ranking last week.) A trimmed-down and cheaper SeaStar2 interconnect used with the XT5m midrange boxes implements a 2D torus mesh and scales from one-half to six full cabinets, which is from 24 to 144 blades and from 768 cores (using Shanghai Opterons in a half-cabinet) to 6,912 cores (using Istanbul Opterons in six cabinets).

All things being equal, and assuming the SeaStar2+ interconnect isn't choking, moving to the Opteron 6100 processors should more or less double the performance of an XT5m or XT5 with an equivalent socket count.

The University of Edinburgh in Scotland is the first HPC center in the world to pony up some cash to buy an XT6 machine, and will be upgrading its Hector system next year. The current machine is an XT4 system that has a scalar and a vector brain and that uses the earlier SeaStar2 interconnect. The scalar portion uses 2.3GHz quad-core Opteron processors (22,656 cores in total) and has a sustained performance on the Linpack Fortran benchmark of 174.1 teraflops. The machine also has 28 vector nodes, each with four vector multistreaming processors (MSPs) designed by Cray. This vector portion of Hector is rated at 2.87 teraflops.

While Cray is cooking up a new interconnect for the second half of 2010, it is promising XT6 and XT6m customers that the blades they buy today will be upgradeable to the future Interlagos twelve- and sixteen-core processors from AMD due in 2011, will be based on a 32 nanometer wafer-baking process, and will use a new core code-named Bulldozer.

Cray also introduced its own play on the US government's "cash for clunkers" program for helping to boost auto sales during the summer as part of the Obama administration's economic stimulus plan. Under Cray's cash for number clunkers (our term, not theirs) program, if you have a workstation or cluster made by Appro, HP, IBM, Silicon Graphics, Sun Microsystems, or Verari Systems that is between two and four years old, Cray is giving trade-in credits on the gear that are good toward the purchase of a Cray CX1 deskside cluster, an XT6m midrange box, or a high-end XT6. No word on what these credits are, and how much they may reflect fair market value of gear. The market for used workstations and supers is not exactly a liquid one. The cash for clunkers program is being managed for Cray by XS International, which is a dealer of secondhand IBM, SGI, Sun, and HP systems based outside of Atlanta, Georgia. ®