Calxeda hurls EnergyCore ARM at server chip Goliaths

Another David takes aim at Xeon, Opteron

Weaving a fabric from on-chip switches

The EnergyCore Fabric Switch is an 8x8 crossbar switch with 80Gb/sec of bandwidth. The switch links out to five 10Gb/sec XAUI ports and six 1Gb/sec SGMII ports that are multiplexed with the XAUI ports. The fabric switch can provide three 10 Gigabit Ethernet MACs and has IEEE 802.1Q VLAN support on those ports.

The EnergyCore Fabric Switch doesn't just link the chip to the outside network through virtual ports, but is used to link EXC-1000 chips to each other in a cluster. It is, in effect, a distributed Layer-2 switch.

The fabric switch on each ECX-1000 chip has three internal 10Gb/sec channels for linking to three adjacent chips on a single system board, plus five external 10Gb/sec channels for linking out to adjacent nodes in a complete system.

With those five external channels, the fabric switch can implement mesh, fat tree, butterfly tree, and 2D torus interconnections of system nodes. With those five lengths, the switch fabric can do a hop from node to node through the fabric in under 200 nanoseconds, according to Freund, and those five external 10Gb/sec virtual ports coming out of each switch can lash together up to 4,096 ECX-1000 chips into a cluster.

Calxeda is not trying to do cache coherency over one to four ECX-1000 sockets on a system board or across the 1,024 possible system boards that the integrated fabric switch scales to. And it is not particularly worried about latencies as parallel workloads pass data around this switch fabric.

"If you look at the workloads we are aiming at, they are not latency sensitive," says Freund. This includes offline analytics like MapReduce big data chewing, Web applications, middleware and Memcached, and storage and file serving. It would be interesting to see how a network of these puppies runs a shared-nothing database cluster.

The topology of the EnergyCore Fabric Switch can be changed on the fly from one style to another and the settings are stored on flash memory on the chip package. Bandwidth can be dynamically allocated in 1Gb/sec, 2.5Gb/sec, 5Gb/sec and 10Gb/sec virtual pipe sizes by the fabric switch and presents two Ethernet ports to the operating system.

The idea is to eliminate the top-of-rack Layer 2 switch that is typically used in a cluster these days with the on-chip switch. While it is possible to build a cluster with 4,096 ECX-1000 chips by using 10Gb/sec XAUI cables and the four ports coming off the Calxeda board to cross link them all, Freund says that most companies will put two real 10GE switches in a rack and use these like end-of-row switches and only lash together 72 four-socket nodes (about a half rack of servers) with the integrated fabric.

The other important thing about that EnergyCore Fabric Switch is that is has dynamic routing, which means you can get around congestion in a network of nodes and also, in conjunction with that management controller, optimize operations for latency or reduced power consumption – or boosted power consumption if you have some work that needs to run faster.

Aiming at microservers as well as clusters

Calxeda plans to offer a two-core chip with most of those switching features turned off running at 1.1GHz as well as a 1.1GHz version with four cores and everything activated. The plan also calls for a clock speed ramp to 1.4GHz on the four-core version in the ECX-1000 line. The two-core version will be aimed at microservers, with their modest I/O and small scalability requirements per node, and Freund says that the two-core variant of the ECX-1000 chip will burn only 1.5 watts per socket, and a stick of low-voltage DDR3 memory will add another 1.26 watts.

The full-on four-core version running at 1.4GHz with all the features turned on and its memory will consume under 5 watts. That four-core version of the chip, by the way, burns 3.8 watts at the full 1.4GHz speed and can idle at under a half watt. That's 2 watts for the four cores and 4MB of L2 cache, 1.5 watts for the fabric switch, management engine, I/O and network controllers, and one 1Gb/sec Ethernet link running, plus 1.26 watts for that single low-volt DDR3 4GB memory stick.

The ARM cores have 14 power states and can come out of a very low power state in microseconds, which is useful as workloads change.

To get server makers started – and to get them excited about the possibilities of building servers with the ECX-1000 chips – Calxeda has put together a four-node server reference card, as shown below:

A four-server Calxeda EnergyCore system board (click to enlarge)

This board has one memory slot per processor socket, and the 32-bit machine tops out as 4GB of memory. The board has extra SATA ports, but only the four on the right are activated at this time. The boards have two PCI-Express-style connectors running along the bottom, which is how the servers draw their power and implement their fabric interconnect.

In this implementation, the reference design, called the EnergyCard, has four 10Gb/sec links running into the to the system board. You can do four complete servers for 20 watts on a board that is 10 inches long.

It's hard to imagine server makers won’t be lining up to get their hands on these EnergyCards. And if they want to start selling them right away, Calxeda is good with that, too. The chips will be able to run Canonical's Ubuntu and Red Hat's Fedora Linuxes to start; Windows Server 8 could eventually get there if Microsoft gets interested. (So far, it has made no commitments, even with Windows 8 for clients and mobile phones getting an ARM port.)

The ECX-1000 chips will sample in late 2011, right on time, and volume shipments of the chips will start in the middle of 2012. That is about when Applied Micro will begin sampling its 64-bit X-Gene chip, which has its own crossbar switch but one that implements up to 128-way symmetric multiprocessing across 64 of its two-core chips.

It will be interesting to see how these two ARM server chips compete against each other as well as against other RISC chips and Intel Xeon and AMD Opteron processors. And remember, graphics chip maker Nvidia, which sells ARM-based SoCs for smartphones and tablets, has also promised ARM chips of its own designs for PCs and servers. Thank heavens for a little competition to keep Intel and AMD honest. Or whatever you might call it. ®