Collinear AI Releases CWE-bench v2 for Evaluating Cybersecurity Vulnerability Remediation
Frontier AI models · AI coding
Collinear AI has launched CWE-bench v2, a public benchmark of 100 tasks covering 91 Common Weakness Enumerations (CWEs) to evaluate how AI models identify and remediate security vulnerabilities in real open-source repositories without breaking functionality.
Collinear AI introduced CWE-bench v2, an open cybersecurity benchmark containing 100 tasks designed to evaluate the defensive capabilities of AI models across 91 Common Weakness Enumerations. In each task, an agent receives a real open-source repository checkout along with instructions to find and repair security flaws while preserving existing software operations. Scoring uses programmatic, deterministic verifiers shaped by LLM judge panels and manual reviews, alongside a held-out evaluation set. In preliminary evaluations of 11 models, Gemini 4 Argon achieved a leading 67.8% pass@1 and 81.0% pass@4, followed by Claude Opus 5.5 at 67.3% pass@1, while DeepSeek V4.1 Flash achieved 57.0% pass@1 at an estimated $0.04 per rollout. Across all tests, 95 tasks were solved by at least one model, though leading models still failed nearly 19% of the tasks. The benchmark is packaged as Dockerized environments compatible with the Harbor runner.
Key takeaways
- CWE-bench v2 provides 100 public tasks covering 91 CWE vulnerability categories based on real open-source repositories.
- Grading relies on deterministic programmatic verifiers and human review to ensure patches do not rely on memorization or break existing code.
- Among 11 evaluated models, Gemini 4 Argon scored 67.8% pass@1, followed closely by Claude Opus 5.5 at 67.3% pass@1.
- Tasks are distributed via Dockerized Harbor environments on Hugging Face, usable for evaluation and reinforcement learning training.
Source：Hugging Face / authors Read the original ↗
Prepared, checked and automatically published by AI; not individually reviewed by a human.