US energy lab's pump-happy petaflopper goes green

Shiny new Xeon–Xeon Phi hybrid splashes into water-cooled data center

The US Department of Energy's National Renewable Energy Laboratory has hooked up with HP, Intel, and partners to design a new hybrid supercomputer and an energy-efficient data center wrapper for it that will – among other green gains – use the exhaust heat from the supercomputer to heat adjacent offices.

After all, when you're the DoE, you can't be messing around running supercomputers or the data centers that house them in an inefficient manner. You have to lead by example.

The new machine, which has yet to be nicknamed, will be set up in a data center at the Energy Systems Integration Facility in Golden, Colorado, and will be significant for a number of reasons.

First, NREL will have its own behemoth cluster and won't rely on processing capacity at the "Red Sky" supercomputer, rated at 433.5 teraflops on the Linpack Fortran benchmark test, that's installed at Sandia National Laboratory down in New Mexico. That super was built by Sun Microsystems using quad-core Xeon 5500 processors back in 2009 when Sun was still in the HPC cluster business – Oracle is not interested in traditional, technical HPC.

The new machine is being built by HP in conjunction with Intel, which wants to use the box to showcase future Xeon processors and Xeon Phi x86 coprocessors, formerly known as "Many Integrated Core" or "Knights Corner" before they were officially branded Xeon Phi chips in June.

The plan, according to Marc Hamilton, vice president of high performance computing at HP, calls for NREL to spend $10m to get a petaflops-class machine that is "all Intel inside." The machine will be built in stages between now and the summer of 2013, when it will be completed and up to its full processing capacity.

In the first phase, NREL will take delivery of 1,600 server nodes from HP. These will be comprised of a mixture of ProLiant SL230s and SL250s Gen8 nodes, which slide into the SL6500 chassis and offer the densest computing that HP puts on the market.

The SL230s nodes (that "s" is part of the product name, not a plural) will be equipped with eight-core Xeon E5-2670 processors in the "Sandy Bridge" family that were announced in the spring, with 2GB per core or 32GB of main memory. The SL250 nodes will use a mix of the same processors, which spin at 2.6GHz, and Xeon Phi coprocessors, which plug into PCI-Express slots and are each expected to deliver at least 1 teraflops of double-precision floating point oomph.

The plan calls for an upgrade of the processors to future "Ivy Bridge" Xeon E5 chips and the addition of 600 Xeon Phi coprocessor cards to the cluster. Hamilton said the peak performance of the cluster, which will be lashed together with 56Gb/sec FDR InfiniBand switches and server adapters from Mellanox Technologies, will be "north of one petaflops of performance." The machine will run Linux and be attached to a Lustre-based clustered file system.

The SL230s is a half-width computing node that is 1U high; the SL250s is a half-width node that is 2U high. Both machines have only one PCI-Express 3.0 x8 slot, so the only reason to choose the SL250s is to give the Xeon Phi coprocessor a little more space in the chassis. You might think it's all about airflow, but Hamilton tells El Reg that one of the secret sauces in the new NREL petaflopper is component-level water cooling inside the server. Hamilton says that this cooling method has not yet been formally launched by HP, so he is not giving out names or details; he adds that it will be available commercially in the coming months.

"Pumps trump fans," says Steve Hammond, computational science center director at NREL. "A small pump moving water is more efficient at getting rid of heat than a slew of muffin fans."

Hammond tells El Reg that this water cooling was an integral part of the cluster and data-center design, which is expected to have a power usage effectiveness (PUE) ratio of 1.06. PUE is the ratio of the power consumed by the data center divided by the power consumed by the IT gear when it is running; a PUE of 1.06 and means the NREL design will have nearly all of its power being used by the gear, not for moving cold air around.

NREL looked at using mineral oil and other coolants to get rid of the heat in the cluster, but Hammond said he did not want to have to "get into a pair of coveralls to service a machine."

The DOE's NREL data center warms offices with data center heat (click to enlarge)

Hammond says that the data center will have no mechanical chillers and will only use evaporative cooling, which is a choice you can make in the dry Colorado air. The design that HP and Intel have come up for the SL server nodes is capable of taking away about 90 per cent of the heat generated by the servers, leaving that evaporative cooling to only have to cope with the remaining 10 per cent of the load on the server side.

The water-cooling system is configured to bring 75-degree water to the server components and to run with a 95-degree return temperature into the evaporative coolers that chill the water and the air inside the data center as well.

The NREL data center will have about 10,000 square feet of raised floor space, and the petaflops ceepie-phibie machine will only take up about 1,000 square feet of that – NREL has plenty of room to grow.

The data center and adjacent office design calls for waste heat pulled out of the data center to be used to warm offices in the colder months, and some excess heat may be used in other building throughout the Golden campus.

The new data center opens on October 18 and NREL takes delivery of the first server nodes on November 5.

The one thing that was not clear was where the renewable energy lab is getting its juice from – presumably not from burning dirty ol' coal. ®