Storage bunfight: Hazelcast says Gridgain faked benchmark result

In-memory startup denies Hazelcast's alleged porkies


In-memory software startup GridGain has denied an allegation by Hazelcast that it committed benchmark fraud.

This comes after GridGain published benchmark data showing it is faster than Hazelcast, which is one of its competitors.

Hazelcast CEO Greg Luck believes that the GridGain "tests were faked, illegitimate and intentionally misrepresentative" and has gone into details about it on his blog which says:

GridGain/Apache Ignite have released a benchmark which purports to show that they are faster than Hazelcast. This follows on from our Hazelcast/Ignite Benchmark using Yardstick, their own benchmark, which shows Hazelcast 3.6-EA was much faster than Ignite 1.4.1.

In looking into it we discovered that they had faked the test.

Greg Luck, CTO at Hazelcast

Hazelcast CEO Greg Luck

He wrote: "Instead of configuring the tests the same, they were using "map.put() instead of map.set() in Hazelcast, while using put() is equivalent to our set() operation because their put() does not have a return value." In addition, Luck alleged that GridGain ran "sync backups in Hazelcast for replicas while configuring their own backups as async."

"We ran a comparison using our own benchmark tool correcting for these issues," wrote Luck. "And we were way faster. Just as we were against Apache Ignite 1.4.1."

And then he clearly says:

We believe their benchmark is an intentional misrepresentation.

The blog then details test results and gives directions for anyone wishing to reproduce the tests.

GridGain response

Dmitriy Setrakyan, GridGain co-founder and EVP of engineering, said: "We were very surprised to see the Hazelcast blog. Specifically, with the accusations they are making. One would think that they would at least make an effort and reach out to the Ignite community or the GridGain team.

"I should also mention that we have made an effort to reach out to Hazelcast on their user group in the past, publishing our benchmark results and asking Hazelcast to review. Greg Luck has removed our post and provided no further commentary. I guess this shows the difference between a commercial company and an Apache project."

GridGain has re-run its original benchmark result, with Setrakyan telling us: "We just reran all the benchmarks, and reconfirmed all the results on Amazon AWS with Ignite being over 50 per cent to 100 per cent faster in many tests. The results are published on the Ignite website here."

He specifically commented on the Hazelcast CEO's points about the Map calls. Regarding Map.put() > Map.set(), "We have updated the benchmark, but this change made no noticeable difference and did not affect the previous outcome."

Also, concerning synchronous vs asynchronous backups: "The configuration we ran the benchmark with was 100 per cent identical between Apache Ignite and Hazelcast, so Hazelcast statements are simply false. On top of that, from their blog it seems that they ran the benchmarks on their own hardware and for some reason also changed the test scenarios. Our benchmarks were run on Amazon AWS, the code is published, and anyone can retry them."

Setrakyan emphasized that GridGain is committed to benchmarking: "We take benchmarks very seriously and regularly run benchmarks to compare Ignite and GridGain performance to other products. In our runs, we have over 20 different configuration scenarios and use cases that we test. We reran all the benchmarks today and came out ahead on all of them."

Funding round

GridGain has also gained $15m in B-round funding, led by Sberbank of Russia and MoneyTime Ventures, with the participation of Almaz Capital and RTP Ventures.

Victor Orlovski, a general partner at MoneyTime Ventures, will join GridGain's board. The cash will pay for expansion in sales, marketing and R&D and help drive GridGain's international growth.

Total funding is now $27.2m, quite modest in terms of last year's stratospheric storage funding rounds. ®


Other stories you might like

  • Microsoft unveils Android apps for Windows 11 (for US users only)

    Windows Insiders get their hands on the Windows Subsystem for Android

    Microsoft has further teased the arrival of the Windows Subsystem for Android by detailing how the platform will work via a newly published document for Windows Insiders.

    The document, spotted by inveterate Microsoft prodder "WalkingCat" makes for interesting reading for developers keen to make their applications work in the Windows Subsystem for Android (WSA).

    WSA itself comprises the Android OS based on the Android Open Source Project 1.1 and, like the Windows Subsystem for Linux, runs in a virtual machine.

    Continue reading
  • Software Freedom Conservancy sues TV maker Vizio for GPL infringement

    Companies using GPL software should meet their obligations, lawsuit says

    The Software Freedom Conservancy (SFC), a non-profit which supports and defends free software, has taken legal action against Californian TV manufacturer Vizio Inc, claiming "repeated failures to fulfill even the basic requirements of the General Public License (GPL)."

    Member projects of the SFC include the Debian Copyright Aggregation Project, BusyBox, Git, GPL Compliance Project for Linux Developers, Homebrew, Mercurial, OpenWrt, phpMyAdmin, QEMU, Samba, Selenium, Wine, and many more.

    The GPL Compliance Project is described as "comprised of copyright holders in the kernel, Linux, who have contributed to Linux under its license, the GPLv2. These copyright holders have formally asked Conservancy to engage in compliance efforts for their copyrights in the Linux kernel."

    Continue reading
  • DRAM, it stacks up: SK hynix rolls out 819GB/s HBM3 tech

    Kit using the chips to appear next year at the earliest

    Korean DRAM fabber SK hynix has developed an HBM3 DRAM chip operating at 819GB/sec.

    HBM3 (High Bandwidth Memory 3) is a third generation of the HBM architecture which stacks DRAM chips one above another, connects them by vertical current-carrying holes called Through Silicon Vias (TSVs) to a base interposer board, via connecting micro-bumps, upon which is fastened a processor that accesses the data in the DRAM chip faster than it would through the traditional CPU socket interface.

    Seon-yong Cha, SK hynix's senior vice president for DRAM development, said: "Since its launch of the world's first HBM DRAM, SK hynix has succeeded in developing the industry's first HBM3 after leading the HBM2E market. We will continue our efforts to solidify our leadership in the premium memory market."

    Continue reading

Biting the hand that feeds IT © 1998–2021