Twinned from https://github.com/cfsh/ninefold/issues/481 by tools/beads/import_github.py's reconcile pass.
Filed unclaimed and unlabelled from ninefold-a during design-001 pieces 2–5. Backlog, for a future triage round.
Sibling to #450, and deliberately not folded into it. #450 is *"a test that asserts nothing passes like a test that asserts everything"* — about test code. This is about what we tell the reviewer afterwards, which is a different artifact with a different failure mode and a different fix.
merge_pr.sh's feedback ledger enforces that every non-advisory finding has a later reply. It cannot check whether the reply is true, and it says so:
> Timestamps can't verify itemization *within* a bundled comment — that part stays on you.
So the machinery guarantees a reply exists. Nothing guarantees the reply describes the tree the PR is actually at. And the specific way it goes wrong is not lying — it is a claim that was true of the tree it was measured on, restated about a later tree.
Every one was caught by ninefold-reviewer, and each cost a full verdict round to re-establish.
| # | The claim | What was true |
|---|---|---|
| 1 | *"All three addressed — 25cbad3"*, posted on #466 | The commit was on #472's branch. #466's head was unmoved, so H1/H2/H3 were genuinely unfixed on the PR whose thread said they were done. (#472 F4) |
| 2 | *"The recipe is deleted from the module docstring, the PR body, and — permanently — the shim"* | Two of three. The PR body still carried it, as a section heading addressed to whoever merges the PR. (#470 F5) |
| 3 | *"Mutation-checked, 8/8 caught"* | True at 0ff1b3c. The next commit added and not args.dry_run, which made one of the eight mutants unreachable — invalidating a row without a re-run. (#466 J1) |
| 4 | *"H2 addressed"* | Fixed in one of the two locations the finding named by line. (#466 J2) |
Instances 1 and 3 are the interesting pair: both were honest and verified when made, and both became false one commit later. That is what makes this systemic rather than a discipline problem — the claim has no binding to a sha, so nothing can notice it aged.
Cost per instance is roughly one reviewer round: a verdict written, read, replied to, and re-verified — on a tier=2 PR that is otherwise converged. Four of those is a meaningful fraction of a chunk, and the failure repeats *across* sessions, because the next session inherits the habit and not the scar.
It also degrades the ledger's own value. The reviewer has now had to verify claims rather than review code often enough to say so explicitly (*"Verified in the tree at head, not from the replies"*). A ledger the reviewer has learned to distrust is the #389 shape again: a gate that runs and cannot be believed.
0ff1b3c"* rather than *"8/8 caught"*. Costs nothing, and turns instance 3 from a false statement into a true-but-stale one a reader can check. It does not catch instances 1, 2 or 4.merge_pr.py --check: for each finding the reply claims done, require the reply to cite a sha, and warn when that sha is not the PR's current head. Mechanical, no judgement needed.rebuild_f5.sh returns to the branch it started on and the session committed without re-checking. A git branch --show-current assertion in the commit path, or having rebuild_f5.sh restore the *pre-invocation* branch explicitly, closes that one narrow case — and it has now happened twice (the other was an nflib/proc.py fix landing one PR too high in the stack).(3) is the only one that addresses a *cause* rather than a symptom, and it is three lines. (1) is nearly free and would have made two of the four self-evident on reading.
Not a proposal to slow down replies, and not an argument for shorter ones. The four instances above are all in replies that were detailed and substantively correct about the *work*; the defect is purely that the work and the claim drifted apart in time.
Related: #450 (tests that cannot fail), #446 (a PR's own follow-ups going stale), #478 / #471 (the same "can this report be trusted" question, one layer down in the probe gate).
None.
No comments.