How well does your vendor do in cybersecurity tests?

Be sure to examine the results

Choosing an anti-malware solution can be tough. IT teams have dozens of products to choose from in their battle against infection from millions of exploits, both new and old. The cybersecurity industry often uses testing agencies to help it define the best products, but it also harbours a dirty secret: not all of them are reliable, or even credible.

Testing agencies run a battery of tests and scores are designed to evaluate anti-malware products’ suitability in specific areas such as reliability and performance. The results are often distilled down to simple percentage scores. Occasionally, testing agencies release results that enable vendors to claim their products are 100 per cent secure. That’s bogus.

Nothing is ever 100 per cent secure in the IT world because the space is so dynamic. With 390,000 new malicious programs emerging every day, it’s impossible to guarantee that you will be able to protect against all of them – so a vendor’s 100 per cent score on a security test should immediately ring alarm bells.

Even executives at Symantec, one of the heavy hitters in the traditional antivirus space, have admitted that the old model is flawed, adding that they can only detect around 45% of new malware.

How is it possible that executives at AV companies openly admit the limitations of their products and even quantify the results, but testing agencies still award scores nearing or reaching 100%?

IT teams choosing anti-malware solutions for their users have a duty to check these results and ensure that they are getting the truth.

Here are some things to think about when assessing anti-malware test reports, and some questions that you can present to the testing agencies involved.

Follow the money

Some test reports are commissioned by vendors. Others invite vendors to contribute financially to a group test of different products. Ask the testing agency whether it takes financial contributions from anti-malware vendors. If it does, that warrants further investigation.

One of the antivirus industry’s secrets is that many products use the same core engines to scan for malicious software. Products that use the same basic code and techniques to find viruses, worms and spyware leave themselves with fewer options for differentiation. So where are the different results in the tests coming from?

Product configuration and virus samples can be one way to differentiate anti-malware tools. Does the testing agency treat products differently based on the vendor’s contribution? Does the agency spend more time configuring and optimising some products than it does on others? Or are there strict standards in place to ensure that each product is tested in the same way, using the same setup and parameters?

What samples are used?

With so many malware strains emerging daily, testing agencies understandably have to use a sample of the available malicious software, rather than attempt to census test all of it.

Just as with statistical surveys, the way the sample is chosen can have a significant effect on the results. The testing agency should test a significant number of malware strains – as many as possible – rather than just, say, the top couple of hundred.

It’s important for testing agencies to test samples that companies are likely to encounter in the real world. Tests on samples of malicious software shown to be prevalent in the wild are less likely to skew results than a selection of samples that do not reflect real-world systems.

So ask some polite questions. How many test samples did the agency use? Were the same samples used for each vendor? Did any vendors get to choose the samples that were tested?

Made to measure malware

Even if a testing agency is honest in its sample selection, it faces another challenge that can stop its results being accurate: custom malware. This is becoming more prevalent, as malware authors tweak their code for individual attacks specifically to avoid being caught by anti-virus products.

In its 2016 Data Breach Investigations Report, Verizon even put some numbers to this. It revealed that 99 per cent of hashes are seen for less than a minute on the network before disappearing. Most malware is seen only once. The upshot: attackers are changing their malware strains fast.

How do anti-malware testing agencies cope with that? Ask them. The answer had better be good because the security of your business depends on it.

Have results been edited?

Another question to ask the author of that anti-malware testing report is do malware vendors get to challenge the results of the test before they are published? Do they even perhaps get editorial approval? Is that approval based on any financial contribution?

These are good things to know when you’re assessing the validity and independence of a report – especially a report that guarantees a 100 per cent test score.

Do-it-yourself testing

So what are the alternatives? Testing anti-malware products yourself is one way forward, but it is not something to be undertaken lightly. Testing needs to be done safely and with careful planning. You wouldn’t test vaccines against killer pathogens in an uncontrolled environment, and you shouldn’t do it with digital ones either.

IT testing teams should create a step-by-step plan for obtaining a representative sample of malware that will give them results relevant to the real world. They should have a strategy for housing those malware samples in a secure environment, and then use the appropriate tools to test those products.

Find out more about how to do that and empower yourself to test anti-malware products to satisfy your own criteria.

Not all testing agencies have hidden agendas. There are some competent, well-managed and honest product testers out there committed to delivering reliable results to IT professionals evaluating cybersecurity tools. But not all testers are created equal.

Knowing what to ask – and what to do if the answers aren’t convincing – puts the power back in your hands.