← CVE-bench on GitHub
CVE-bench · fix real CVEs

CVE-bench · The Cost–Performance Pareto Frontier

A security benchmark: AI systems must fix real CVEs in open-source software. A fix counts only when the security regression test passes after the agent's patch is applied — the vulnerability is actually closed. Ranked by resolve-per-dollar, not raw score. One tunable Value Score blends capability and price; drag the slider to set your cost-vs-capability tradeoff. New here? Hit “How it works”.

● resolve % measured by the official harness ● conformance firewall: no gold patch/test during solving ● cost/instance self-reported by submitters

Security work, measured like a benchmark

~$0.003–$0.03 per instance. Each CVE-bench problem is a real disclosed vulnerability plus a security regression test (a FAIL_TO_PASS check that goes green only when the vuln is fixed). The agent gets the buggy repo and the issue; it never sees the gold patch or the gold test while solving — that's the conformance firewall.

The economics are the same lever as code-repair benchmarks: the moat is the harness (a stateful loop gated by the project's own security test), not brute-force compute. Capability (resolve %) is measured under the firewall; price ($/instance) is self-reported. The dataset here is a small validated seed — the machinery is complete and scales as more CVEs are validated. (resolve % measured; cost self-reported — see honesty notes below.)

The frontier

measured, conformant pilot (full set pending) official (reported) cost-Pareto frontier

Leaderboard — sorted by Value Score

ValueSystemScaffoldResolve % $ / instRun totalResolve / $Evidence
Welcome

CVE-bench · The Cost–Performance Pareto Frontier

cheaper · cost / instance → resolve % →

A security benchmark: AI systems must fix real CVEs. Most leaderboards rank by score alone — this one ranks by resolve-per-dollar, the cheapest system that closes each level of vulnerabilities.

Each dot is a system; the green line is the frontier — nothing beats it on cost and capability. Resolve % is measured by the official harness under a conformance firewall. Drag the Value Score slider to set your own cost-vs-capability tradeoff.