Moore's Law puts supercomputers out to pasture because power – not just the cost of electricity, but the availability of juice – is the biggest constraint at the big supercomputing centers. And sometimes the lack of budget helps lock the gate, and HPC cloud computing butchers the cow.
That's the case with the massive 28-rack supercomputer at the University of New Mexico nicknamed "Encanto," which was supposed to be a foundation for high-tech investment and research in the Southwest state with the motto "Land of Enchantment."
The idea with the Encanto project was to rent out capacity on the box, as El Reg reported when the Xeon-based cluster, built by Silicon Graphics using 3,500 of Intel's four-core Xeon processors and actually running in an Intel data center, was first fired up.
The Encanto project was allocated $14m by the New Mexico state government in 2007, with $11m of that going for the system itself and $3m going for gateways – presumably a mix of networking gear and telecom services – to link into the system from around the state. At the time Encanto was first turned on, it cost $2m a year to keep it running. So over five years, it cost as much to run the supercomputer as to buy it.
The Encanto Altix ICE cluster currently has 14,336 cores, a peak theoretical performance of 172 teraflops, and a sustained performance or 133.2 teraflops on the Linpack Fortran benchmark. That gives it a ranking of 185 on the most recent Top500 supercomputers list from November 2012. The largest machines have between 50 and 100 times more oomph than Encanto, and also big budgets from the larger national or transnational supercomputer organizations that can afford to pay and run these monstrous machines.
According to a report in the Albuquerque Journal, New Mexico state information technology secretary Darryl Ackley is going to chop up Encanto into bits and put it out to pasture in various research universities in the state. No one wants to buy Encanto as-is because of the high cost of running the machine.
The irony is that six years ago, when Encanto was being conceived, the Altix ICE 8200 machines were reasonably power-efficient. Since then, however, Intel, AMD, and IBM have cranked up the core count and lowered the thermal envelopes of their floppiest processors, helping machines burn significantly less juice.
Encanto burns 861 kilowatts, which works out to 154.7 megaflops per watt sustained performance. The 17.6 petaflops hybrid CPU-GPU "Titan" supercomputer at Oak Ridge National Laboratory burns 8.21 megawatts, delivering 2,142 megaflops per watt. That's performance per watt improvement by a factor of about 13.8.
The irony is that we are going to need a much larger leap in efficiency to get to exascale computing in a 25 megawatt power budget, as the major HPC players are working together to accomplish by 2018 to 2020. That's a factor of 50 times the performance increase in only three times the power budget to get around 40 gigaflops per watt in something on the order of 200 to 300 racks.
New Mexico's Encanto supercomputer
The Encanto machine was supposed to have rented out enough capacity to New Mexico's universities, industrial companies, and consultants to be self-sufficient in five years, and as El Reg pointed out five years ago, this all had a pie-in-the-sky feel to it, and Encanto would be a rather run-of-the-mill supercomputer by 2013. And so it was, and so it is.
Ackley told the Albuquerque Journal that a few tire-kickers came looking at Encanto, seeing if they might want to buy it, but it was too expensive to operate. The University of Mexico is slated to take ten racks of the box, with New Mexico State University taking four and the New Mexico Institute of Mining and Technology taking two.
If the machines are free or have a modest price, then this makes sense. Otherwise, those schools should simply buy a modern Xeon or Opteron cluster – maybe with Tesla GPU or Xeon Phi coprocessors – and stop living in the past.
The lesson here is that if you want to build a supercomputer processing and application development center, your budget has to be sufficient to not only run a supercomputer but also to upgrade it and make it appealing for modern applications.
As usual, it takes money to make money. And for a lot of companies, buying some capacity on Amazon makes a lot more sense and is probably easier – and cheaper – than whatever New Mexico could come up with for Encanto. Moore's Law might have killed Encanto, but AWS is nailing the coffin shut. ®