Collinear AI Releases CWE-bench v2 to Benchmark LLM Code Security Auditing and Patching
Frontier AI models · AI coding
Collinear AI introduced CWE-bench v2, a benchmark evaluating frontier AI models on real-world cybersecurity tasks. Across 250 tasks covering 185 distinct CWEs, models must inspect repositories, identify undisclosed flaws, and apply fixes without breaking functionality. Evaluations as of November 2026 showed Claude Opus 5.5 leading with a 67.1% Pass@1 rate, while DeepSeek-V4.1-Flash achieved 57.6% at an average rollout cost of $0.04.
Collinear AI has published CWE-bench v2, an expanded cybersecurity evaluation framework designed to measure LLM agent capabilities in software auditing and vulnerability remediation. Moving beyond standard classification or exploit generation, the benchmark tasks models with inspecting open-source repositories, detecting vulnerabilities without explicit hints, and writing functional patches verified by deterministic graders. The v2 release features 250 tasks spanning six programming languages (C/C++, JavaScript/TypeScript, Go, Java, Python, and Rust) across 185 Common Weakness Enumerations (CWEs), categorized into ten weakness classes. The benchmark divides tasks into an open-source public set of 100 tasks across 91 CWEs and a held-out set of 150 tasks across 126 CWEs. Graders rely on programmatic checks ensuring both that exploits are blocked and existing functionality is preserved without relying on memorized upstream fixes. Leaderboard figures reported as of November 2, 2026, show Claude Opus 5.5 achieving top performance with 67.1% Pass@1 (79.2% Pass@4) at an average rollout cost of $1.85, followed by Muse Wonder at 66.1% and Gemini 4 Argon at 65.0%. DeepSeek-V4.1-Flash scored 57.6% Pass@1 at $0.04 per rollout. Across all evaluated models, 20 out of the 250 tasks remained completely unsolved, with failure modes frequently stemming from localization, missed detection, incorrect invariant enforcement, or incomplete remediation.
Key takeaways
- CWE-bench v2 expands coverage to 250 tasks and 185 distinct CWEs across C/C++, JS/TS, Go, Java, Python, and Rust, divided into 100 public and 150 held-out tasks.
- Evaluation simulates realistic security workflows: agents inspect raw code without vulnerability names or counts, requiring safe patching verified by deterministic checkers without breaking existing features.
- November 2026 leaderboard data shows Claude Opus 5.5 leading at 67.1% Pass@1 ($1.85/rollout), with DeepSeek-V4.1-Flash reaching 57.6% Pass@1 at a low cost of $0.04 per rollout.
- Twenty out of 250 tasks remained unsolved by all tested models, with common failure points occurring in trust boundaries, authentication, and error handling.
Source：cwe-bench.com Read the original ↗
Prepared, checked and automatically published by AI; not individually reviewed by a human.