Skip to content
Evals · Nov 3, 2026

CWE-bench v2 expands public cybersecurity evaluation suite to 100 tasks across 91 CWEs

Collinear AI’s updated benchmark adds 100 programmatic tasks spanning 91 vulnerability classes, with deterministic scoring and a held-out set for progress tracking. Results are tracked on the Hugging Face Hub.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • CWE-bench v2 introduces 100 public tasks to evaluate cybersecurity capabilities of AI models.
  • Tasks span 91 CWEs and use deterministic, programmatic scoring for reproducibility.
  • Leaderboard tracks open-weight and API-only models; Gemini 4 Argon leads with 67.8% pass@1.
  • Benchmark is open-sourced and Dockerized, with a gated dataset on Hugging Face.
  • Cost-performance trade-offs show DeepSeek V4.1 Flash, Claude Opus 5.5, and Gemini 4 Argon on the Pareto frontier.

Collinear AI has released CWE-bench v2, an updated benchmark designed to test AI models on cybersecurity tasks. The benchmark consists of 100 programmatic tasks where an agent receives a real open-source repository and a short instruction, with the goal of identifying and fixing a security weakness without breaking existing functionality. Scoring is fully deterministic, enabling anyone to reproduce results without additional cost.

The benchmark covers 91 different CWEs (Common Weakness Enumerations), spanning a wide range of vulnerability classes. This breadth ensures that models must understand multiple weakness surfaces to perform well, rather than relying on recognizing familiar patterns alone. To maintain quality, each task was reviewed to ensure instructions appear realistic, verifiers check for correct behavior, and reference fixes pass while preserving original functionality. Tasks were also manually reviewed and are designed to fail memorized solutions.

CWE-bench v2 includes a held-out set of tasks that models have not seen, allowing for progress tracking on unseen challenges. Access to the held-out set requires contacting cyber@collinear.ai. The benchmark is open-sourced, with the dataset, grader, and harness publicly available. Users can run evaluations directly from the Hugging Face Hub using the Harbor tool, which Dockerizes each task and provides a deterministic verifier.

Results are tracked on two leaderboards hosted on the Hugging Face Hub: one for open-weight models and another for API-only models such as Gemini, Claude, and GPT. As of the latest update, 11 models have been evaluated. Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, followed closely by Claude Opus 5.5 (67.3% pass@1, 78.0% pass@4). Ninety-five tasks are solved by at least one model, indicating the tasks are solvable, though the leader still fails 19% of the set on every attempt.

The benchmark also provides cost-performance insights. DeepSeek V4.1 Flash (1017), Claude Opus 5.5, and Gemini 4 Argon form the Pareto frontier in terms of efficiency. DeepSeek achieves 57.0% pass@1 at $0.04 per rollout, while Claude Opus adds ten percentage points at $1.83, and Gemini edges ahead at $3.92. The dataset is gated on Hugging Face and requires accepting access conditions before use.

The benchmark’s tasks are tagged as reinforcement learning environments, enabling their use not only for evaluation but also for training defensive agents. The dataset ships with a registry.json file, and Harbor can run the entire benchmark directly from the Hugging Face repository without requiring local cloning. Users must authenticate with their Hugging Face token to fetch tasks via git.

Sources
  1. 01Hugging Face — CWE-bench v2: More coverage, quality, and transparency
Also on Evals

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.