The comprehensive framework is designed to provide standardized, transparent, and use-case-driven evaluations, allowing testers to weigh results based on their specific security needs
San Francisco, California – March 26, 2025 – AMTSO, the cybersecurity industry’s testing standard community, has published its Sandbox Evaluation Framework, the first standardized methodology for assessing the effectiveness of sandbox-based malware analysis solutions. This framework, created by AMTSO’s Sandbox Evaluation Working Group, provides security professionals, researchers, and vendors with a structured approach to evaluating and comparing sandbox technologies based on key performance indicators, including detection capability, anti-evasion technology, speed, and reporting accuracy—while also allowing testers to weight results according to their unique requirements, ensuring a tailored assessment for diverse security environments.
“With the Sandbox Evaluation Framework and other frameworks AMTSO has been developing, we are providing the cybersecurity community with a practical blueprint that enables organizations and testing professionals to establish a testing environment and integrate these methodologies into their evaluations,” said Vlad Iliushin, President and CEO of AMTSO. “It’s a valuable resource that can be leveraged to establish or enhance existing testing methodologies and assure consistent, transparent assessments.”
In an era where cyber threats are growing in complexity, sandboxes have become a vital defense mechanism, analyzing potentially malicious files, URLs, and phishing attempts in controlled environments. However, the absence of a standardized testing methodology has led to fragmented evaluations, making it difficult to compare solutions fairly. AMTSO’s Sandbox Evaluation Framework addresses this gap by defining a comprehensive and transparent scoring system that aligns with real-world cybersecurity use cases.
“This framework marks a significant milestone for the industry, shifting the focus from generic benchmarks to a use-case-driven evaluation of sandbox performance,” said Jan Miller, Lead Author of the Sandbox Evaluation Framework, and CTO of Threat Analysis at OPSWAT. “By aligning testing methodologies with real-world security challenges, it provides an unbiased, data-driven approach that helps organizations select the right solution for their needs.”
Key Features of the AMTSO Sandbox Evaluation Framework:
- Holistic Evaluation Criteria – The framework assesses sandbox solutions based on six key performance indicators (KPIs), including analysis capability, anti-evasion techniques, scalability, reporting, automation, and security compliance.
- Use-Case-Driven Scoring – It introduces customized weight profiles for different security needs, such as large-scale malware processing, phishing triage, zero-day detection, and threat intelligence generation.
- Standardized Benchmarking – The framework incorporates open-source benchmark tools and a scoring formula to enable transparent and repeatable evaluations.
Developed by the AMTSO Sandbox Evaluation Working Group, consisting of leading cybersecurity experts from OPSWAT, VMRay, Venak Security, Malwation, and others, the framework reflects industry-wide expertise and best practices.
“At VMRay, we are proud to participate in developing the AMTSO Sandbox Testing Methodology, which sets a new industry benchmark for evaluating sandbox solutions,” said Ralf Hund, CTO at VMRay. “This methodology ensures organizations can accurately test a sandbox’s ability to detect unknown malware and phishing threats—reflecting the real-world challenges every company faces today.”
The AMTSO Sandbox Evaluation Framework is now available to cybersecurity professionals, vendors, and researchers. AMTSO encourages broad industry adoption and welcomes feedback for future updates.
For more information and to access the full framework, visit www.amtso.org/documents.
About AMTSO:
AMTSO is the cybersecurity industry’s testing standard community, consisting of over 50 security and testing member companies from around the world. The organization offers a platform for knowledge-sharing and collaboration on objective standards and best practices for anti-malware testing and assessment of other cybersecurity products. The AMTSO standard raises the bar for cybersecurity tests, contributing to more fairness in the industry, and creating transparency for consumers and businesses looking for the best digital protection. www.amtso.org