Start here
The big picture 🗺️
Every AI here is trying to fix a real security vulnerability — a published CVE — in real open-source software. Two things matter: how many vulns it actually closes (capability) and how much that costs (price).
Most security leaderboards only show the first. This page shows both, and combines them into one number — the Value Score — so you can see who closes the most CVEs per dollar.
The test
What's a "resolve"? 🛡️→✅
CVE-bench gives the AI a real disclosed vulnerability — the buggy version of an open-source project plus the issue describing the CVE. The AI writes a patch. The patch is "resolved" only if, after applying it, the project's security regression test (a FAIL_TO_PASS check that fails on the vulnerable code) now passes — i.e. the vulnerability is genuinely closed.
Read the CVE write-up + the vulnerable code.
Write a security patch (a code change that closes the hole).
Run the security regression test. It passes → resolved ✅ (vuln fixed). It still fails → not resolved.
Crucially, the solver never sees the gold patch or the gold security test while it works — that's the conformance firewall. The score is "% of CVEs resolved".
Axis 1
Capability = Resolve % 📊
The share of CVEs the AI actually closed. Higher is better. A system at 50% fixed half the vulnerabilities it was given — verified by the security regression test, not by self-report.
But capability alone hides the catch: the 75% system might cost 500× more to run. That's why we need price.
Axis 2
Price = Cost per instance 💵
What it costs the AI to attempt one CVE — mostly the price of the tokens the model reads and writes. Big "frontier" models charge a lot per token and use huge contexts; small/cheap models cost a fraction of a cent.
That's a hundreds-of-× difference for one CVE. It compounds fast across a whole dataset — which is the next page.
Why price really matters
The real bill = Run total 🧾
You rarely fix one CVE. To run the whole benchmark you pay cost × number-of-CVEs.
Darwin seed: $0.003 × N = cents
Frontier agent: $2.50 × N = dollars per CVE add up fast
Same set of CVEs. One costs a rounding error; the other costs real money — for a similar number of fixes.
The calculus
The Value Score ✦
To rank by both at once, we turn each axis into a 0–100 number and blend them:
Capability = the resolve % directly (0–100).
Cheapness = price on a log scale, mapped so $5/CVE → 0 and $0.005/CVE → 100. (Log, because going $2→$1 matters as much as $0.02→$0.01.)
Blend them with a weight w you choose:
The slider above the table is w. Drag it toward capability and frontier systems rise; drag toward cost and the cheap-but-capable systems win. At the middle (50/50), you're asking "who's the best all-rounder per dollar at closing CVEs?"
The picture
Reading the chart 📈
Each dot is one system. Left = cheaper, up = more capable. So the top-left is the dream (cheap and closes CVEs).
The dashed green line is the cost-Pareto frontier: the best resolve achievable at each price. Nothing beats a system sitting on it without paying more. Darwin anchors the cheap end.
Trust
Honesty & proof 🔍
We separate what's measured from what's self-reported:
Resolve % (all systems) is measured by the official CVE-bench harness under the conformance firewall — the security regression test is run, not trusted.
Cost/instance is self-reported in each submission's metadata (marked est for official entries). "Undisclosed" where no cost is given.
Darwin's numbers are measured: resolve via the official harness, cost from real API spend. Darwin is conformant — it never sees the gold patch or gold security test while solving. Entries that break the firewall get a red NON-CONFORMANT tag.
The dataset is a small validated seed; the harness, grading, and firewall scale. Source: the CVE-bench repo, linked at the bottom of the page.