Measuring on planted bugs Diagram

The short answer

• The sysmobench replay reviews specifications a model checker has verified, and copies with one bug planted
• Opus is shown the planted diff and says whether a finding names it
• The verified specs count the false alarms

Not shown here

• The hooks and delivery, which the replay bypasses by calling the worker directly
• Connected but out of scope: Hook entry, Turn log and jobs, Settings, Router, DeepSeek Flash, Project tests

What it guarantees

• On sysmobench, 104 of 106 planted bugs the checker catches were confirmed and ranked first: the answer key is the planted diff; on small systems 1 of 158 correct specs got a confirmed finding, on etcd 4 of 15 with the request

Measuring on planted bugs

How do we check polywatch where the right answer is exact?

Git history cannot say which commits are really correct. Specifications checked against traces of the real system can, and a planted bug says exactly what a good reviewer should find.

The long read

The sysmobench replay reads the price-of-trust study's ledgers: which specifications passed the checker, and which verified ones had a bug planted in them by another model.

Each specification is reviewed as a new file named spec.js, with the study's task prompt (which contains the real source code) as the request. Nothing in the file or its name says whether a bug was planted.

After the run, Opus is shown the diff between each mutant and its source and says whether any finding names the planted change. The verified originals give a clean count of false alarms, because the checker has already said they are correct. On the spinlock and lock service, 90 of 91 planted bugs were confirmed and ranked first, and 1 of 158 verified specs got a confirmed finding.

Terms used here

turn: One round of Claude Code work: the user's prompt and everything Claude does in answer, up to when it stops.
hook: A command Claude Code runs at a fixed event, such as after a file edit, when a turn stops, or when the user submits a prompt.
claim: One specific, checkable statement a reviewer makes about a defect: which file, where, and what is wrong.
finding: A claim that survived confirmation (it was not refuted), with a score. Confirmed findings carry Opus's evidence.
confirmed: Opus checked the claim against the code and said it holds.
ledger: An append-only file of every review and every outcome, used to calibrate the ranking.
mutant: A copy of a correct specification with one small behavioral bug introduced on purpose, so the answer is known.
sysmobench: A benchmark of executable specifications of real systems, checked by replaying traces recorded from the real implementation.