Twinned from https://github.com/cfsh/ninefold/issues/450 by tools/beads/import_github.py's reconcile pass.
Filed unclaimed and unlabelled from ninefold-a during design-001 piece 1/1b work. Backlog, for a future triage round.
presubmit.py now gates the tooling tier, and pieces 2–5 will grow that suite from 76 cases to several hundred. Nothing gates the test code itself. A test that asserts nothing passes exactly like a test that asserts everything, and the tier is about to bet its correctness on tests written by a session that will be context-reset tomorrow.
This is #389's shape one level up. #389 was *"nothing ran probe_all, so main's UI gate rotted unnoticed"*. Here the gate runs — it just cannot tell you whether what it ran was worth running.
Twelve reviewer findings across five rounds. Four were in my own test harnesses, which are the one artifact nobody reviews:
sed extractions returned empty strings — a backreference eaten during generation — so it compared "" to "" and passed. *Empty inputs agree with each other about everything.* It sat green while the thing it tested was broken.sed range through a nested heredoc extracted nothing, so dispatch_health was defined as a no-op. Three assertions expecting silence passed for the wrong reason.gh stub always exited 0, so "auth failure" and "empty queue" were literally the same input — the two cases that most needed to differ. The defect (a failed listing stamping the reviewer *healthy*) survived 24 green assertions.1. Every regression test must be demonstrated to FAIL against the unfixed code. Not argued — run. I did this exactly once, deliberately, for the --list regression:
``` # fix reverted # fix restored TypeError: 'NoneType' object is not subscriptable FAILED (errors=1) OK ```
That took ninety seconds and converted "I believe this test covers it" into "this test covers it". The three vacuous cases above are all things this would have caught immediately, and none of them were caught by reading.
2. A harness needs a *dimension matrix*, not a case list. Design 001 §5.5 says to run "every branch that matters" but not how to know what they are. Empirically the branches that mattered were never the ones enumerated:
| round | dimension the oracle never varied | defect it hid |
|---|---|---|
| 1 | subprocess exit status | failed gh pr list stamped ok=true |
| 2 | invocation mode (--check vs full) | cadence hold applied on one path of two |
| 3 | extraction extent | the gate invocation was outside the extracted block entirely |
The blind spot moved each time; it never closed. The reviewer's framing, and it is the transferable lesson: for each script, enumerate *(inputs × invocation modes × failure modes)* before writing assertions, and record the matrix next to the tests. A dimension nobody wrote down is a dimension nobody varied.
Stated so the tradeoff is visible. Both are process conventions, and conventions rot exactly like the code they govern — a checklist item everyone skips is worse than nothing, because it looks like coverage. Convention 1 is genuinely mechanical and could be enforced (a presubmit.py lane that re-runs new tests against HEAD~1 and expects failures, which is mutation testing narrowed to the one mutation that matters). Convention 2 cannot be enforced and is only ever a habit.
The honest counter is that this tier's whole argument for Python was *testability*, and testability that nobody audits is the same trust-without-verification the migration exists to end.
Related: #414 (the diagnosis this extends), #433 / design 001 §5.5 (the migration method), #389 (a gate nothing ran).
None.
No comments.