twitter facebook rss

Anti-malware testing issues

Posted by on December 4, 2016.

There is something wrong with this picture. Carl Gottlieb has founded a new online service called TestMyAV. It’s purpose is to allow you to test your own anti-virus product: “Giving you the malware, testing guides and tools you need to test antivirus products for yourself. Trust yourself, not the ‘experts’.”

The site provides the malware and advice on how to handle malicious files. One page gives a list of the tools that it suggests will be helpful: “Virtualising your test environment is ideal – bordering on necessary – and provides several advantages.” The tools it lists come under the headings: Process / System Monitoring Tools; Network Monitoring Tools; Automation Tools; Compression Tools; Mutation Tools; Lab Components; and CLI Commands. At the bottom of the page it says, “Want a few pointers? Fancy a chat? Just want an antivirus product recommendation?”

Another page provides testing guides, and ends with “Just want an antivirus product recommendation?”

The malware page lists the malware packages you can download, and ends with, yes, “Just want an antivirus product recommendation?” I won’t bore you with any more – let’s just accept that every page ends with “Just want an antivirus product recommendation?” We’ll come back to that.

Elsewhere in the anti-malware ecosphere there has been a not-so-subtle shift in the marketing battle-cry of one of the leading next-gen machine-learning (ML) products, Cylance. For a long time the Cylance approach was to say that legacy anti-virus doesn’t and cannot work because it is purely signature-based malware detection. Cylance wasn’t alone in this approach; but it’s a gross oversimplification that is frankly misleading.

It resulted in, shall we just call it an antipathy, between the legacy companies and the newcomer machine learning companies. I’ve written about the outcome here: VirusTotal Policy Change Rocks Anti-Malware Industry. But good came of it. A number of next-gen anti-malware companies have already joined AMTSO (an industry organization dedicated to improving testing standards), and have had their products tested to AMTSO standards by third-party test laboratories. These include Carbon Black, Crowdstrike, FireEye, Invincea, Palo Alto Networks and SentinelOne – with more in the pipeline – but not Cylance.

Concomitant with joining AMTSO is an agreement not to use VirusTotal results as a form of product comparison – and without that comparison it is no longer so easy to suggest that legacy anti-virus is signature detection only and doesn’t work. Cylance has never made that commitment to AMTSO; but now suffers from a different marketing problem. Unable to beat the machine-learning buzz word, legacy AV is rapidly unveiling it’s own ML capabilities, bringing out their own ML-based front-ends. They have used ML behind the scenes for around a decade.

The problem is that machine learning can only be as good as the data from which it learns – and the legacy companies have been collecting malware samples and analyzing their behavior for decades. Using the term ‘machine learning’ to differentiate its own product from its competitors is no longer realistic. Which brings us to the not-so-subtle shift: Cylance is now attacking anti-malware test results.

Disrupting a major market segment by displacing archaic and ineffectual approaches to a pervasive security problem, and publicly calling out flawed performance testing methodologies that perpetuate consumer misunderstandings, is the equivalent to pouring a bottle of steak sauce on your jugular and unleashing the neighbor’s pit bull – you must be prepared for a long and potentially damaging fight.
From Finland to Israel to Silicon Valley, Who’s Afraid of Next-Gen AV?

I asked Dennis Batchelder, President and CEO of AMTSO, what he thought about this statement. He said simply, “I applaud the notion of challenging the status quo. I can’t wait until Cylance is ready to join and contribute their ideas of how our industry can improve its ability to protect the world.”

Cylance is resolutely declining to join the rest of the anti-malware ecosphere. But it is not alone in behaving badly. Symantec recently published its own blog with the headline: AV Comparatives / MRG Effitas: Symantec Endpoint Protection 14 once again decimates Cylance Protect 1.2 – and a more ridiculous headline you’ll be pushed to find. Decimates? Really? This sort of thing does no-one any good.

And it is exactly the approach that allows Cylance to say:

Bottom line: don’t take our word for it. If you think Symantec really is 100% effective, test it for yourself and see what you find.

Go to the Test for Yourself site and get the tools, and then get the truth.
The Hidden Cost of ‘Pay-to-Play’ AV Testing

Now here’s the irony. Cylance is saying, don’t trust the independent experts – do it yourself and we’ll help you (which sort of implies that they are neither independent – true – nor expert – not true).

At least, you might say (going back to the start of this post) TestMyAV is independent, allowing you to do your own independent testing. Well it isn’t. And, here’s what is wrong with this picture. TestMyAV was founded and is run by Carl Gottlieb, and funded by Cognition. Carl Gottlieb is CTO at Cognition. Cognition is a major reseller of Cylance. Funded and run by Cognition, who else would you expect TestMyAV to recommend? Full circle.

The danger is that people will be taken in by this ‘don’t trust the independent experts, just trust me’ proposal.

So what is the solution? Should companies heed Cylance and do it themselves? Well, yes and no – but in neither case should independent expert advice be ignored.

The ideal solution is indeed for companies to test AV products themselves; but in situ and configured by the vendor. The buyer should insist on a three-month trial. It doesn’t have to be free, only that it is not binding. This should be done with two different products, so the buyer can see exactly how each product works in his own environment. That short-list of two products will probably come from word of mouth recommendations and an analysis of third party test results.

You might be surprised at how many vendors will accept this, if you insist hard enough. The implication is that if they don’t, they rather fear they would lose in the real life comparison.

Of course, small companies will have less success in getting a trial period. But I would then suggest that small companies simply do not have the resources to do their own testing. Never, ever would I recommend downloading live malware over the internet. Small companies should analyze test results, do a paper analysis of features, and talk to peers already using those products.

Finally, let me summarize everything by saying that Carl Gottlieb is a great guy, and Cylance is a great product. I just wish that both would have the confidence of their convictions and use independent third-party testing to demonstrate their excellence rather than resorting to slagging off the competition. And, yes, that last bit also applies to Symantec.


Brief flame on Twitter following this post. Start here and see the full thread:

One thought on “Anti-malware testing issues

  1. Great write-up Kevin!

    I just stumbled across TestMyAV last weekend and the first thing I noticed was their association with Cylance.

    I believe that without playing by the same rules, Cylance won’t be able to prove they’re in fact better.

    Furthermore, IMO it’s incredibly irresponsible to let users test malware.

    And the lack of transparency makes it worse.

    Marketing-wise it’s an interesting strategy, but I do feel that they could be more transparent in their relation to Cylance.

Leave a Reply

Your email address will not be published. Required fields are marked *

Submitted in: Expert Views, Kevin Townsend's opinions | Tags: , , ,