Out of nowhere, Nimbus, an all-flash array supplier, has beaten incumbents NetApp and 3PAR at eBay, with 12 arrays and 100TB of flash installed to accelerate eBay's server virtualisation work.
Nimbus CEO Thomas Isakovich said: "It's one of the largest single deployments of flash anywhere... This is a seven-figure account for us... It's something I've personally been working on for six months."
Open the Pod eBay doors Hal
Before Nimbus came onto the scene, eBay had VMware ESXi v4.1 with thousands of hosts, and NetApp FAS iSCSI and 3PAR SAN storage arrays, with about 3PB of installed storage capacity in total. It wanted to roll out standard "Pods" for virtualisation and these had to serve VMs on demand to thousands of hosts.
The Pods are are standard combinations of servers, vSphere, storage and networking: converged IT stacks in effect. The eBay guideline was that the smaller and more power-efficient a Pod was, the better. It chose Dell blade servers and Cisco networking. Who would supply the Pod storage?
NetApp and 3PAR both bid for the project, with Nimbus bidding its S-class array in 10TB capacity form. NetApp bid a system using its FlashCache (PAM) controller caching technology.
HDD and flash array price equivalence
Michael Craft, eBay's manager of QA systems admin, said: "Nimbus delivered a solution based entirely on flash memory, not a hybrid approach, at a price point comparable to primary disk arrays and with no software licensing fees to boot."
According to Nimbus figures, its S-class cost less to buy and much less to run than a NetApp array configured with the same amount of usable capacity; Isakovich saying: "NetApp has about 20 per cent file system overhead, so you need 18TB of raw capacity."
Nimbus CEO: "We can justify flash right now because the acquisition cost is less."
The acquisition cost for a 10TB Nimbus S-class with a year's support was $129,536 while the 10TB usable NetApp system was $135,168, based on 450GB, 15,000rpm drives. The purchase of years 2-5 support was $90,112 from NetApp and $45,056 from Nimbus, giving a five-year fixed cost for NetApp of $227,908 and $174,920 for Nimbus, $52,988 less.
Nimbus then added in expected power usage and cooling needs, giving out these numbers:
- NetApp: 13,140 kWh power usage, 44,834 BTU for cooling.
- Nimbus: 2,190 kWh power usage, 7,472 BTU for cooling.
The five-year total cost of ownership was much lower with Nimbus, even without taking performance into account. Here, according to Nimbus, it creamed NetApp, producing 800,000 4K IOPS versus the FAS array's 20,000. The cost/IOPS was $0.22 for the S-class and $11.39 for the NetApp array.
Isakovich says 3PAR and NetApp bid caching and tiering-based products whereas Nimbus bid a primary storage array, and was able to do so because it has a Halo O/S feature set including deduplication; thin-provisioning; security through encryption; snapshots; replication; and high-availability through synchronous mirroring.
He said that the S-class (below) delivers consistent performance, whereas the others depend on flash cache hits for performance, and if the data isn't in cache, performance slows.
He added: "Our products took about half the floorspace of the equivalent NetApp product and we use less power... We can justify flash right now because the acquisition cost is less."
Also: "I'm not even considering deduplication. We're doing it pound for pound of usable capacity. The dedupe is money in the customer's pocket."
The Nimbus S-class now has 8Gbit/s Fibre Channel (to 8 ports) and 40Gbit/s InfiniBand (to 8 ports) added to the existing 1 and 10GbitE host connectivity.
The system uses 24 flash modules, not standard off-the-shelf solid state drives (SSDs), with different flash densities per module to provide the 2.5, 5 and 10TB capacity points.
Isakovich said: "We have developed new flash modules which deliver about two and a half times the performance of our first generation modules."
The S-class also has a new 144Gbit/s mid-plane with a fully non-blocking architecture so that all 24 flash modules get full bandwidth and there's no over-subscription. There are also 12 processor cores instead of the previous 8.
These changes boosts overall S-class gen 2 performance about three times over the gen 1 S-class. It can deliver 80,000 4K IOPS with a 50:50 mixed read and write workload, and 8GB/sec throughput with the same workload mix.
Up to 25 S-class enclosures can be stacked in a rack to get to 250TB of capacity, inside a single file system.
The new S-class is now shipping with sales and support in North America, Europe and Japan. The pricing is straightforward: $24,995 for a 2.5TB S-class; $49,995 for a 5TB one; and, you guessed it already, $99,995 for a $10TB one.
Thin end of a great big fat wedge
In the overall eBay scheme of things, 100TB of flash compared to 3PB of spinning disk isn't much to write home about, but write home you should. This is the thin end of a mighty, mighty wedge.
Up until now flash arrays have basically competed on the idea that they blow spinning disk arrays away on performance, although they do cost a lot more on a $/GB basis. Nimbus's numbers show it can blow NetApp and 3PAR away on the IOPS and latency fronts, and compete on the cost of the usable storage capacity as well. That's a significant marketing shift, with Nimbus claiming: "15k disk is dead; flash takes over now."
Flash is changing everything.
Three-bit multi-level cell flash is coming, which will lower $/GB flash costs and make all-flash arrays more affordable.
If customers like eBay, McAfee, Lockheed Martin and the US DoD, are satisfied that a relatively unknown and small vendor's flash array beats NetApp and 3PAR on performance, endurance, five-year cost of ownership, and has satisfactory availability, data protection and support features – right in the virtualisation sweet spot of the enterprise market – then every spinning disk vendor is at risk.
When Huawei-Symantec, Nimbus, Violin, TMS, Solid Air, SolidFire, WhipTail, or others with all-flash array product, come calling on HDD array customer accounts for new virtualisation projects the incumbents can no longer, if they ever did, take business for granted. Flash is changing everything. ®