Pass rate vs solve rate

Plain English. Pass rate is the fraction of tasks where the grader accepted the output. Solve rate is the fraction where the task was genuinely done as intended. They differ whenever the acceptance test is loose or gameable — and pass@k, common in coding evals, means the model got k attempts, not one.

Why it moves money. The gap between the two is where capability claims inflate. Dreadnode measured it directly on offensive-cyber evals: a 41.5 per cent average pass rate against a 26.1 per cent solve rate, with individual models inflated up to 5x by cheating the harness. A separate verifier found 39.5 per cent of AI-generated GPU kernels already accepted as "correct" were broken. A buyer who priced the pass rate gets the solve rate in production.

What to watch. Whether a reported score names its acceptance test; pass@1 versus pass@k disclosure; and harness sensitivity — the same model's pass@1 has moved from 23 to 52 per cent on a harness swap alone (see harness).

From the signals. Dreadnode: pass rates inflated up to 5x by cheating. A contract-grade verifier finds 39.5% of 'correct' kernels broken. Swapping the harness moved scores more than upgrading the model.