Canadian storage startup Diablo has pulled out what it hopes will be a PCIe-flash killer: its new tech provides fast flash data access by interfacing a NAND partner's solid-state drives to a host server's memory bus.
Diablo, founded in 2003, has invented what it calls Memory Channel Storage (MCS) in which NAND flash is packaged in DIMM (Direct Inline Memory Modules) units called TeraDIMMs, and slotted into a server's DIMM sockets. A partner provides the NAND flash and will be the sales channel for the TeraDIMM (or whatever the ultimate brand name is).
The TeraDIMMs have ultra low latency compared to SSDS and PCIe flash cards.
Described by Daiblo's patent application:
a TeraDIMM is a new class of DIMM memory that enables the creation of physical memory systems that can match the virtual memory capabilities of CPUs by integrating non-volatile memory (eg, NAND flash, phase-change, etc) into, for instance, a DDR-3 or DDR-4 memory channel.
Diablo was given $28m by investors in November last year to help develop its MCS chipset and its ASIC hardware and firmware technology and produce TeraDIMMs.
This uses the DDR3 protocol and the flash is addressed as part of the existing memory address space. Diablo provides the DDR3 to SSD controller and firmware but needs a flash product vendor to provide the NAND, like SMART.
We already know Diablo and SMART Storage are working together with an exclusive relationship.
In April, John Scaramuzzo, president of SMART Storage Systems, said: "The convergence of Diablo’s groundbreaking MCS technology and our award-winning Guardian Technology Platform is not just about re-architecting storage, but re-architecting the integration of servers, storage and Flash acceleration. We are setting a new standard for ULL (ultra low latency) storage in the enterprise, improving performance over currently available SSDs, including PCIe-based drives.”
TeraDIMMs fitting into a server's DIMM slots
By slotting MCS access into a partner's flash chips, Diablo claims it has made data access much faster than with PCIe-connected flash. The firm says its TeraDIMMs, which have 200GB or 400GB of 19nm MLC capacity in a chart we saw, have latencies approaching that of DRAM - let's say 5-10µs write latency - and can do 150,000 read IOPS and 65,000 write IOPS. The things can run at 10 full drive writes a day for five years, the firm tells us.
IO performance scales as more TeraDIMMs are added, but the latency stays constant. Diablo claims the latency is half that of existing PCIe flash products and provides a chart to show this.
TeraDIMM latency (red) versus PCIe flash latency (blue) in high-frequency trading scenario.
The case being made is that, in memory subsystem terms, PCIe flash is quite distant. With TERADIMMs data is in the memory subsystem and so access to it is faster.
The TeraDIMMs can be used for in-memory compute with an app's entire dataset in the combination of DRAM and flash. Diablo compares a 25,000 144GB DRAM servers using MEMCACHED with a 5,000 servers kitted out with 16GB of DRAM and 800GB of TeraDIMM flash, saying the TeraDIMM servers provide RAM disk performance and $31m/year in total cost of ownership savings.
Servers can have, say, 16 to 24 DIMM slots. Fill 16 of those with 400GB TeraDIMMs and we have 6.4TB of flash. What we have here is a way of providing terabytes of NAND to servers in a way that has so far been done with masses of PCIe flash such as Fusion-io's ioDrive 2 with up to 10TB using 8 x 1.2TB modules, and its ioScale with 3.2TB per card. LSI's WarpDrive goes up to 3.2TB and Micron is promising a 16TB P520 PCIe flash card in the future.
Performance comparison with PCIe flash
Jim Handy, Director at Objective Analysis, said: "Flash SSDs boost system performance, and flash over PCIe is even faster, but the best performance will come from flash on the memory channel." Diablo says its MCS architecture reduces latencies by more than 85 per cent over PCI-Express based SSDs and 96 per cent over SATA/SAS based SSDs.
Performance comparisons are tricky as a generic PCIe flash card does not generally equal a 200/400GB TeraDIMM.
Let's take a 1.2TB Micron P420m MLC PCIe flash card, which runs at 750,000 random read IOPS. A server fitted with 3 x 400GB TeraDIMMs, the same NAND capacity, would receive 3 x 150,000 IOPS — 450,000 IOPS in total — from the flash, with a read latency of, say, 5μs versus the P420m's ≤75μs.
With equivalent NAND capacity levels, the TeraDIMM reacts faster than the P420m but doesn't deliver more IOPS. That might be because the IOPS are not being counted in a consistent fashion with, for example, 4K data blocks being used in both cases.
This is just The Reg storage desk's back-of-the-envelope guesstimating. We'd need a real benchmark to see what the deliverable performance is in each case.
Diablo claims that its technology provides radically lower $/IOPS numbers than PCIe flash.
How about application software? What does it need to do to make use of TeraDIMMs?
Diablo CEO Riccardo Badalone says app software doesn't have to change at all. The Diablo-accessed NAND can be used as a cache by caching software in which case the application doesn't see it directly, or it can be a block storage device that happens to be in the host's memory address space. A driver can be hooked into it to turn it into a swap engine.
Software environments that can benefit from TeraDIMMs are said to include:
- Database and the Cloud
- Big Data Analytics
- High Frequency Trading and other financial applications
- Server and Storage Virtualization and Consolidation
- Virtual Desktop Infrastructure
SanDisk acquiring SMART Storage
Diablo and SMART have been working on this technology for two years and it appears that SanDisk's purchase of SMART Storage came as a happy surprise to Diablo. Badalone said: "It's definitely a very positive event for us."
Happily SMART is a user of Toshiba NAND, and Toshiba and SanDisk are partners in the NAND foundry business.
The acquisition gives SMART surety of NAND supply and adds SanDisk's distribution channels to SMART's existing channels.
SanDisk has its own PCIe flash product line: the Lightning brand of 200 and 400GB SLC flash products with a gen 2.0 PCIe interface. They clearly need refreshing to PCIe gen 3.0 and, probably, to be given NVMe support. It will be fascinating to see how SanDisk positions its PCIe products with what we presume is the fruit of the exclusive Diablo/SMART Storage relationship.
Badalone says there are lots of proof-of-concept trials taking place with tier 1 end users.
At this point Fusion-io, Virident and Diablo/SMART (presumably) have flash architectures for servers that can provide multi-terabyte flash capacities for applications and enable in-memory computing. Customers are encouraged to use Fusion-io software to get maximum use from Fusion's flash hardware.
Diablo's way of providing flash-enhanced performance with no app software changes may very well appeal as an instant fix for servers in performance-intensive environments that are suffering from data access latency limitations, even where the data is coming from SATA and/or SAS-connected SSDs, or even PCIe flash cards. ®