IBM has pulled the plug on the "Blue Waters" petaflops-class, Power7-based supercomputer that it was contracted to build for the National Center for Supercomputing Applications at the University of Illinois.
In a statement released today by IBM and NCSA, the two parties said that Big Blue terminated the Blue Waters contract because the Power7-based behemoth was more complex and expensive than they had both bargained for.
"The University of Illinois and NCSA selected IBM in 2007 as the supercomputer vendor for the Blue Waters project based on projections of future technology development," the statement reads. "The innovative technology that IBM ultimately developed was more complex and required significantly increased financial and technical support by IBM beyond its original expectations. NCSA and IBM worked closely on various proposals to retain IBM's participation in the project but could not come to a mutually agreed-on plan concerning the path forward."
Vacancy: NCSA's National Petascale Computing Facility
The withdrawal of the contract leaves NCSA with an 88,000-square-foot data center that opened in June 2010 and that was to house the Blue Waters super. For all you Cray, Silicon Graphics, Hewlett-Packard, and Dell sales reps: the National Petascale Computing Facility is located in Champaign-Urbana, Illinois, on the south-west side of the University of Illinois campus. Look for the oil storage containers and the parking garage, and if you find yourself down by the Imported Swine Research Laboratory, you've gone too far east.
John Melchi, who led the NPCF project and is head of the administration directorate of the NCSA, was bragging in April 2010 that the data center for Blue Waters was completed ahead of time and under budget, which has to be some sort of record in the IT business.
Speaking to El Reg this morning just after a trip to the dentist and ahead of a slew of phone calls from HPC vendors, Melchi said that the total value of the Blue Waters contract, which was being funded mostly by the National Science Foundation with some application code work being paid for by the university. To date, IBM had shipped three racks of the Blue Waters supers to NCSA, and these will be returned. IBM has to give back $30m to NCSA.
The plan was to have a full Blue Waters system with over 1 petaflops of sustained performance up and running by the end of 2012, Melchi explained. "It is really cool technology, and my colleagues and IBM are saddened by this," he said. "I think in the end, the price tag was even too high for IBM."
This is precisely the reason why in May 2009 NEC and Hitachi pulled out of the $1.2bn K super project to build a 10-petaflopper for the Japanese government. During the Great Recession, both companies were losing money and said they would lose even more money manufacturing the K machine. Fujitsu, which was also a partner on the project, salvaged the deal and the K super, built using the Sparc64-VIIIfx processor and the Tofu interconnect, was ranked the top-performing super on the Top 500 supercomputer list in June. It is not clear how much the Fujitsu-only version of the K super cost, but it was surely a lot less than the planned $1.2bn.
Like K, Blue Waters, which is being commercialized by IBM as the Power 775 server, is a complex machine with very high manufacturing costs. The machine has 2U server drawers that fit into custom racks that are 30 inches wide and 6 feet deep. Each node has 256 cores, 2TB of memory, and a series of home-grown hub/switch chips that can lash together up to 2,048 Power 775 drawers (a total of 524,288 Power7 cores) with a peak performance of over 16 petaflops.
NCSA was never going to have a fully loaded system and was only seeking to get sustained performance of over 1 petaflops. The trouble is that – based on IBM's list pricing – a machine with 1 petaflops of peak performance would run about $150m, and you might need to double the raw capacity of the machine to get sustained performance of 1 petaflops, for maybe $300m. Presumably, the manufacturing costs for this custom-made machine were too high.
The real question people want to know is if the Blue Waters super actually worked. "We have no reason to believe that it didn't," said Melchi, adding that this was really a financial decision on the part of IBM's upper management.
And now, NCSA is looking for a new supercomputer. "We've been coordinating with the NSF to ensure the project remains a success and that we deliver a sustained petaflops system to the scientific community," Melchi said.
By the way, IBM is still selling the Power 775 systems and sources at the company tell El Reg that it already has customers lined up for the machines. ®