the mutation harness is rebuilt from scratch every time, and two rebuilds were themselves broken nf-g7us ← Beads

Twinned from https://github.com/cfsh/ninefold/issues/482 by tools/beads/import_github.py's reconcile pass.

Filed unclaimed and unlabelled from ninefold-a during design-001 pieces 2–5. Backlog, for a future triage round.

Extends #450 in the direction #450 explicitly does not go. That issue argues for the discipline — *"every regression test must be demonstrated to FAIL against the unfixed code. Not argued — run."* This is about the tooling for it, which does not exist, and about what that costs.

It is also #414's complaint one level up: *"none of that scaffolding survives the session — the next person re-derives it."*

Measured, this session

I ran the demonstrate-it-fails discipline six times across pieces 2–5. Every time I wrote the harness from scratch: read the file, pick mutations, patch, run unittest, diff, restore. Roughly 40–60 lines each, all of it thrown away.

Two of the six were themselves defective, and both failed toward looking correct — the exact class #450 is about, reproduced inside the tool built to detect it:

1. A no-op harness reported 8/8 survivors. str.replace(old, new, 0) replaces *zero* occurrences in Python; I meant -1. So no mutation was ever applied and every mutant "survived", which reads identically to "the tests are worthless". I only caught it because the result was implausible — had it said 8/8 *caught*, I would have believed it.
2. A crash mid-loop stranded a mutation in the working tree. The harness hit the cp1252 blocker (design 001 §2.4) printing a ✓, died between mutate and restore, and left if False: in poll.py. I found it by accident on the next grep. A tool that edits source and can die between mutate and restore is a tool that can silently corrupt the tree it is verifying.

Cost is not only the rebuilding. Both defects consumed a debugging cycle each, and defect 1 briefly convinced me a fix was untested when it was fine.

Why the value is real rather than tidiness

When the harness worked, it was the highest-yield check in the session — it caught things reading did not:

• five tests that passed with the idle +2 bug fully restored, because they re-implemented the rule instead of calling it;
• a test that passed with os.chdir(root) wrapped in if False:;
• two deviations with no test at all, found only by mutating them out and seeing nothing go red;
• a CHARTER.md contract test that survived rewriting the rule it claimed to pin.

Every one of those was in code I had just written and believed covered. That is the argument for making it cheap: it works, and it is currently priced like a research project.

Shape, uncosted

A small tools/mutate.py taking a mutation list and reporting caught/survived would cover every use above. The properties that matter, all learned the hard way:

• restore in a finally, and verify the restore — defect 2 is unacceptable in a tool that edits tracked files;
• refuse to run on a dirty tree, or stash first, for the same reason;
• assert the mutation actually changed the file — defect 1's guard, one line;
• import nflib first — otherwise it prints ✓/✗ into cp1252 and dies mid-loop, which is how defect 2 happened;
• require a green baseline before and after, so a harness that leaves the tree broken says so.

Whether it belongs in presubmit.py as a lane is a separate and harder question — mutation runs are slow (the suite runs N+1 times), so it is plausibly a --mutate mode invoked deliberately rather than on every merge. Worth deciding with real numbers rather than by argument; at 328 tests and ~5s a run, an 11-mutation sweep is about a minute, which is not obviously too slow to gate.

⚠️ One caution, from aedanpope on #450's thread: a mutation lane *"would be designed to only mutate code changed within the PR. But it's still 🙃"*. That scoping is right and the reservation is fair — mutation testing rewards writing tests that kill mutants, which is not identical to writing tests that catch bugs. This issue proposes a tool a human invokes, not a metric anything is scored against, and the difference is the whole point.

Related: #450 (the discipline), #481 (claims that outlive the tree they were measured on), #414 (throwaway scaffolding).