How your outcomes change the ranking Diagram

The short answer

• The outcome goes into the ledger
• The ranker reads the ledger on every review, so the next findings of that kind score higher or lower, and a kind that keeps being wrong drops below the threshold

Not shown here

• The model providers, which the learning loop never touches
• Connected but out of scope: Hook entry, Review worker, Review results

Doctrine

• Recorded outcomes set the ranking: each kind of finding starts from a prior precision worth four outcomes; every real or false the user records moves it

How your outcomes change the ranking

How does telling polywatch a finding was real or false change what it shows next time?

The starting weights are guesses, and every codebase is different. polywatch gets better at a project only through the user saying which findings were worth their time.

The long read

The loop starts with the user. After seeing a finding, they tell Claude it was a real bug or a false alarm. The findings skill tells Claude to record it by running the CLI's outcome command, with the review id and the finding's number.

The CLI appends one line to the ledger. Nothing else changes at that moment.

The next time a review runs, the ranker reads the whole ledger. For each kind of finding (confirmed or not, high, medium or low severity) it counts the outcomes and computes a precision: the starting guess counts as four outcomes, and each recorded real or false moves the estimate. Findings are sorted by that precision, and only the top two above 0.25 are shown. A kind that has been wrong often enough falls below the line and is hidden; a kind that has been right rises.

Terms used here

turn: One round of Claude Code work: the user's prompt and everything Claude does in answer, up to when it stops.
hook: A command Claude Code runs at a fixed event, such as after a file edit, when a turn stops, or when the user submits a prompt.
tier: polywatch's label for how hard a change is to get right, from its size and whether it touches concurrency, state or protocol logic.
finding: A claim that survived confirmation (it was not refuted), with a score. Confirmed findings carry Opus's evidence.
confirmed: Opus checked the claim against the code and said it holds.
precision: The share of findings of one kind that turned out to be real bugs, as estimated from recorded outcomes.
ledger: An append-only file of every review and every outcome, used to calibrate the ranking.
outcome: The user's verdict on one finding after the fact: real bug or false alarm.