Twinned from https://github.com/cfsh/ninefold/issues/414 by tools/beads/import_github.py's reconcile pass.
Strategic uplift, filed unclaimed. Not urgent; the evidence just got concrete enough to be worth writing down.
run_tests.sh gates the C# tier and probe_all.sh gates the UI tier (#389 is wiring the latter to actually run). The shell tier has neither, and it is not a small tier any more:
| script | what breaks if it silently regresses |
|---|---|
| tools/pr/merge_pr.sh | the feedback ledger stops enforcing; merges land with unanswered review asks |
| tools/pr/probe_gate.sh | the UI gate records a poisoned baseline, or silently stops recording |
| tools/pr/rebuild_f5.sh | f5/integration goes stale or skips a branch — Aedan test-drives the wrong tree |
| tools/playtest/publish.sh | a sitting's journal + captures are never shipped |
| tools/playtest/manifest.sh | /playtest routes dictation to the wrong PR |
| f5.sh | the session dir and the game's derived dir stop matching |
These are the scripts the whole async workflow rests on. Every one of them is currently verified by a human reading the diff.
Three PRs in one session (#403, #405, #407) each needed a synthetic git repo built by hand to verify shell behaviour — and I built essentially the same harness three times:
manifest.sh — a repo with a remote, a branch merged into an f5 branch, then advanced, to prove the under-report window.publish.sh — a repo with session dirs, .published markers and mtime ordering, to prove the churn narrowing and the exit contract.probe_gate.sh — a repo with an orphan screenshots branch and a stubbed probe_all.sh, to drive 7 scenarios.Two of those runs produced wrong results that looked right. A git checkout -qf main reverted the tracked script under test back to its committed version mid-run, so I was measuring the OLD script while believing I was measuring the new one. I only caught it because a count was off by one. That is exactly the class of thing a harness prevents by construction, and exactly the class of thing a careful diff read does not.
The reviewer flagged the same gap independently on both #403 (*"the three shell scripts have no automated coverage; this repo has no shell harness"*) and #407 (*"ships with no regression guard of its own… scenario 7 is the one that would be costly to re-break"*).
None of that scaffolding survives the session. The next person re-derives it.
Deliberately not a proposal to adopt bats or any framework — the repo's taste is hacky-first, and a dependency here would be its own argument.
The reusable part is small and already written three times: make a throwaway repo with a remote, optionally an orphan branch, optionally stub scripts, run the thing, assert on output. Something like tools/tests/shlib.sh with mkrepo, stub, assert_contains, assert_unchanged <ref> <path>, plus tools/tests/*.sh per script and a run_shell_tests.sh runner alongside run_tests.sh.
The cases worth pinning first are the ones already discovered and currently guarded by nothing but prose:
probe_gate.sh — a build failure must not report every standing red as ✓ FIXED; must not record off main; must honour "recording nothing".publish.sh — exit 0 for nothing-to-do vs 1 for tried-and-failed; a manifest-only rewrite must not republish.manifest.sh — a branch that advanced after the f5 rebuild must still be reported, at the sha in the build.merge_pr.sh --check — the ledger must go red on unanswered feedback (this one needs API stubbing, so probably last).Stated so the tradeoff is visible rather than rediscovered: these scripts change rarely, and a harness is itself code that rots. The honest counter is that they change rarely and are load-bearing and fail silently — the combination that makes a regression sit undetected, which is the same argument #389 makes about probe_all not running.
Related: #389 / #407 (the UI gate), #411 and #412 (reviewer follow-ups that both touch shell behaviour with no way to pin it).
None.
No comments.