HPC Blog IBM gave the audience a deeper dive into the OpenCAPI initiative and hardware at the recently concluded HPC Advisory Council annual meeting in Lugano, Switzerland.
OpenCAPI is a new connection type that gives a high bandwidth, low latency, connection for memory, accelerators, network, storage, and other devices like ASICs.
One of the design goals for CAPI (Coherent Accelerator Processor Interface) was to minimise latency – shooting for "equivalent effective latency of DDR standards". They've done a good job of reaching that goal, with very low 10 nanosecond latency for devices like accelerators and even achieving 5ns latency for advanced memory.
The latency numbers are a pretty big deal – POWER8 round trip latency is about 10ns, and that's very sporty for a processor. Even more important is that the 10ns latency of OpenCAPI beats PCIe like a drum, since a typical PCIe round trip incurs a 100ns latency penalty.
When it comes to bandwidth, OpenCAPI today offers 25GBps while PCIe gen 3 currently offers 32GBps. The speaker, IBM's Jeff Stuecheli, explained that succeeding instances of OpenCAPI will run at 32GBps then 56GBps and higher.
He also pointed out that OpenCAPI is hardware agnostic, meaning it doesn't have any design lockouts, like, for example, some FPGAs, which only connect into the QPI channel of the motherboard through the processor socket. This limits customer choice in FPGAs and thus increases cost. Plus there's a potential reliability concern, since a failure in the FPGA would bring down the entire system.
Both a video of his talk and his slides are available here. In the rest of his discussion, he talks about how OpenCAPI devices operate in the virtual address spaces of the applications they support and how the virtual-to-physical address translation occurs in the CPU. He also goes into details about the protocol stack and the programming model.
This slide deck and discussion is much more detailed than other presentations I've seen about CAPI and OpenCAPI. It gives us more context and provides a fuller explanation of why other companies (Dell, HPE, Micron, AMD, plus others) have joined the OpenCAPI consortium. One notable absentee is Intel, which is understandable, since CAPI is a competitor to Intel's OmniPath product set and the venerable PCIe standard.