San Francisco, California – February 18, 2017 – Testing products in a fair and balanced way is very difficult. Product developers routinely make bold claims about the capabilities of their products. AMTSO supports the right of testers to put these claims to the test, to provide independent validation of their accuracy (or otherwise).
Since our foundation, AMTSO has strongly advocated realistic testing methodologies, exercising all features and layers of all participating products and giving genuine insight into how well they perform at stopping the types of attacks affecting real users in the real world.
We have been asked to explain AMTSO’s opinion on recent privately-commissioned anti-malware tests. We believe that we will be in a much better position to provide useful feedback on specific tests after adoption of the AMTSO standards, but we do want to respond to these inquiries by providing some general statements about problems we’ve seen in recent tests.
- We reject turning off product capabilities while comparing the capabilities of products in real-world use, as we believe that this introduces bias in the results.
- We believe that any claims about what the results of tests show must be valid and accurate, and they must provide both data and evidence that the scenarios tested and the methodologies used do in fact match the resulting claims. In our opinion, test reports without this data and evidence should be rejected.
- We believe that tests that don’t give the tested product vendors an opportunity to engage and to comment on the approach or to validate their configuration are unfair.
- We believe that all comparative tests should follow our draft standards.
- We support the rights of a tester to run any test it wants to, and to test any available product without limitation, consistent with the AMTSO draft standards.
AMTSO is currently running an open pilot to trial the draft standards, and welcomes participation from both member and non-member testers. For more information contact [email protected].
About the Anti-Malware Testing Standards Organization (AMTSO)
AMTSO is a California-based non-profit mutual benefit corporation founded in 2008. AMTSO’s mission is to improve business conditions related to the development, use, testing and rating of anti-malware solutions. AMTSO membership is open to industry-wide academics, reviewers, testers and vendors. Additional information regarding the organization, including charter documents, membership and educational materials are available on the AMTSO website.
MEDIA CONTACTS
You can find our current press contact information on our Press page.