AppEsteem UwS Certification 2019 (series)

Item “Adherence to the test plan” marked “Acceptable”, comment: “In a certain case one sample which was not outside the described 3-day curation window was used in the test. But on dispute this issue was resolved. In a certain other case one sample for which certification status had expired at the time of the test was used. This was also resolved on dispute.”

Response from tester: “Thanks for this feedback. We automatically curate the samples and these get auto-excluded in our results feeds. However, the dispute user experience was still showing these files. We’ll work on adjusting the dispute process so these auto-excluded samples are more clearly marked as auto-excluded.”

Item “Feedback and Dispute” marked “Problematic”, comment: “Dispute information was automatically submitted without providing us the option of verifying our response beforehand. This issue has been reported to AppEsteem with a request to be able to review our response before submitting.”

Response from tester: “Thanks for this feedback. We’ll find a way to make it more clear that this data is auto-saved.”

Item “Execution of Test” comment: “We would request ongoing stats on the absolute numbers and percentages of detected/missed Deceptors, detected Certified apps. This would allow us to understand how we’re progressing in real time. Thank you.”

Response from tester: “Thanks for this feedback. This capability exists in the user experience, and we have recently added a results feed for each AV that can be checked via automation for updates.”

Item “Feedback and Dispute” marked “Problematic”, comment: “The process is functional but it’s still a little rough around the edges. If possible it would be nice to either have individual videos of each test case, or some sort of tagging/labeling/skipping to the proper time in the video rather than having to manually scrub through an hour of footage to find it. It would also be nice to receive email notifications for any FN/FPs or at least a monthly summary rollup email since testing seems to be on random days with no notification at all of any problems.”

Response from tester: “Thanks for this feedback. We have added the timestamp for each sample underneath the video to help you find these. We’ve also added a results feed for the AVs that can be checked via automation for updates.”