Startup takes Reg's coveted 'Top FLOP' award

SiCortex's supercomputing sauce

SC06 SiCortex has bucked one of the more disturbing supercomputing trends - the disconnect between form and function.

The Massachusetts server start-up last week unveiled a system that would moisten the eyes of both Seymour Cray and John De Lorean. Its SC5832 crams 5,832 processor cores into a "Pimp My Cluster" chassis with lighted cabinet doors that rise like wings. The elegant SiCortex box stood out with ease among the mishmash of "industry standard" cluster jobbies at the Supercomputing '06 conference.

As mentioned, however, the SC5832 garnered attention as much for its function as its form.

"There is room in this industry for nice design, but the systems won't make it if they cost more just to look good," SiCortex director of software Larry Stewart told us.

The high performance computing (HPC) world has gone through a dramatic change over the past few years. It has shifted from pricey, proprietary servers tightly glued together to cheaper systems daisy-chained to each other by the thousands. As a result, HPC customers have enjoyed a sharp fall in supercomputer prices, although that dip has come with tradeoffs.

Most of the cheap systems that make up HPC clusters run on chips from Intel and AMD. So, the clusters tend to show ever-increasing processing performance - especially with regard to floating point operations - as Intel and AMD produce ever-speedier chips.

Unfortunately, a number of issues have cropped up as vendors and customers charged after the faster chips at all costs.

For example, the software running on today's clusters spends an awful lot of time waiting for data from memory, since memory technology has not improved at the same rate as chips. In addition, clusters often waste a lot of processor cycles on communications between the thousands of chips and also suffer from myriad hardware failures as a result of both the quantity of servers in a cluster and the hot chips inside of them.

The SiCortex team has tackled all of these major issues affecting high performance computing clusters (HPC) at once by building an unorthodox box from scratch.

The most basic component of the SiCortex systems is a six-core chip. The TSMC-manufactured 64-bit MIPS cores run at 500MHz and can handle two instructions per cycle, while chewing up just .6 watts.

SiCortex then packs 27 of these chips - with two DDR DIMMs each - on a single server board. Its large SC5832 holds - wait for it - 36 of the server boards to give you the 5,832 cores. The half-rack SC648 can hold 108 chips or 2916 cores. (You can see the node layout here.)

Before getting to more of the technical details on the hardware, we'll note that the SiCortex boxes run Linux (a modified version of Gentoo), SiCortex MPI and the Lustre file system. So, despite the eccentricities of the hardware, the software remains standard for HPC customers.

Like us, many of you are probably wondering how well those MIPS cores stack up against much more powerful x86 chips. Sure, they save on power, and there are more of them, but at what cost to overall horsepower?

The SiCortex engineers - and we talked to just about all of them - argue that their system design creates a type of virtuous performance circle. By using low power cores, they can stick the cores closer together, which improves interprocessor communications. Meanwhile, each core is surrounded by plenty of memory and plentiful I/O. In addition, the chips produce less heat and consequently less system failures.

Now, there's plenty of detail on exactly how all this happens, but we've plucked a few of the choice descriptions from SiCortex's literature to give you a flavor.