NSS Labs sues antivirus toolmakers, claims they quietly conspire to evade performance tests

Alleges CrowdStrike, Symantec, ESET, Anti-Malware Testing Standards Org collusion

NSS Labs has thrown a hand grenade into the always fractious but slightly obscure world of security product testing – by suing multiple vendors as well as an industry standards organisation.

Its lawsuit, filed in California this week against CrowdStrike, Symantec, ESET, and the Anti-Malware Testing Standards Organization (AMTSO), has alleged no less than a conspiracy to cover up deficiencies in security tools.

These vendors not only knew of bugs in their code and failed to act, but they were “actively conspiring to prevent independent testing that uncovers those product deficiencies," NSS Labs claimed. The lawsuit hopes to illuminate bad practices that harm consumers, Vikram Phatak, chief exec of NSS Labs, claimed in a statement.

At the heart of the matter, NSS labs has accused the named security vendors of forging a pact to collectively boycott NSS – an independent test lab. Why? Well, if one of them avoided a test all others participated in then it looks bad, but if there’s a collective “no thanks,” then any opprobrium is avoided.

The charge is serious: vendors have come up with a scheme to avoid tests that may expose vulnerabilities they’d rather not have to invest in repairing, never mind the negative PR backlash from poor results. AMTSO – which aims to establish standards for fair testing – is allegedly “actively preventing unbiased testing” and facilitating this bad practice. In addition, Crowdstrike and other unnamed vendors have clauses in their user contracts that prohibit testing without permission, NSS Lab alleged.

"If it is good enough to sell, it is good enough to test,” Phatak argued.

This isn't the first time NSS Labs and Crowdstrike have locked horns: last year CrowdStrike filed an injunction against NSS Labs to prevent the release of test results during the RSA Conference. The injunction failed but legal arguments continue..

In a statement, Crowdstrike dismissed NSS’s legal offensive as baseless:

NSS is a for-profit, pay-to-play testing organization that obtains products through fraudulent means and is desperate to defend its business model from open and transparent testing. We believe their lawsuit is baseless.

CrowdStrike supports independent and standards-based testing — including public testing — for our products and for the industry. We have undergone independent testing with AV-Comparatives, SE Labs, and MITRE and you can find information on that testing here. We applaud AMTSO’s efforts to promote clear, consistent, and transparent testing standards.

El Reg also asked the other named parties in the lawsuit to comment. We’ll update this story as more information comes to hand. ®

Updated to add at 0812 UTC on 21 September

ESET told The Reg: “We are aware of the allegations made by NSS Labs. However, as legal proceedings have just been initiated, we are unable to say more at this time beyond the statement that we categorically deny the allegations. Our customers should be reassured that ESET’s products have been rigorously tested by many independent third-party reviewers around the world, received numerous awards for its level of protection of end users over many years and are widely praised by industry-leading specialists.”

NSS Labs’ chief exec Vikram Phatak responded to Crowdstrike’s pay-to-play allegations: "We are where we are because we refused to be pay-to-play and CrowdStrike knows it."

Updated to add at 1920 UTC on 21 September

"The Anti-Malware Testing Standards Organization is disappointed by the antitrust lawsuit raised by a member organization (NSS), and we categorically deny all claims made against us," AMTSO's CEO Dennis Batchelder told us in a statement.

"AMTSO’s testing standard has requirements for transparency and ethical engagements for both security vendors and testers, and it was developed by testers and vendors for the benefit of the customers of tests. The testing standard is voluntary. It holds both testers and vendors accountable to ethical and fair practices, including ensuring that tests are fair to all participants. It does not tolerate backroom deals, 'fitted' results, or offering private, pay-to-play, undisclosed advantages to some vendors but not others."

Batchelder added that he hopes NSS will work out its differences with its fellow AMTSO members, "rather than trying to use the legal system to tear down what we all built together."

Updated to add on 0707 UTC 28 September

Symantec told The Reg: "Symantec is committed to the highest levels of integrity and security on behalf of our customers, employees, and partners. We rely upon testing from third-party organizations for an unbiased view of the effectiveness of our products. We also believe that ethical, fair and transparent testing methodologies across all vendors is fundamental to provide consumers with accurate and unbiased evaluations. Organizations such as AMTSO help provide consistency and a standard of excellence for testing within the industry for the benefit of end users."

Bootnote

Other security testing labs are available with other examples including AV-Comparatives, AV-TEST, and SE Labs, among others. For what it's worth: the anti-malware market is split between consumer and corporate sales with enterprise revenues forming the largest part of the market, even for the likes of Symantec.