Twinned from https://github.com/cfsh/ninefold/issues/389 by tools/beads/import_github.py's reconcile pass.
probe_all.sh exists to be the UI regression gate — #181's words: *"run this before merging UI PRs the way run_tests.sh gates logic PRs"*. Nothing runs it. rebuild_f5.sh verifies dotnet build, run_tests.sh and the headless import on every rebuild; it does not run probes. So the gate only fires when a human remembers.
#387. #359 merged and regressed three probes on main (focus-cujs, focus-ring, grid-scroll). Nobody noticed. I found it by accident — I was measuring a clean baseline before quoting numbers on an unrelated PR, and only then bisected it.
The two branches had silently diverged, both freshly built when measured:
| | passed | failed |
|---|---|---|
| main | 12 | 8 |
| f5/integration | 24 | 0 |
That's the shape of the problem in one table. f5 is verified constantly — every rebuild, several times an hour — and main is verified never, even though main is where merges land and what every new branch is cut from. The branch we check is the disposable one.
It also interacts with #381: probe_all already had 5 standing failures from the #371 decode bug, so "8 failed" didn't read as *new*. A gate with permanent reds stops carrying information, and by then nobody is reading it anyway.
Post-merge on main, because that is the moment a regression enters and the moment someone is present to act on it. merge_pr.sh already syncs main after merging, so it is one call away.
merge_pr.sh runs probe_all after the post-merge sync, and reports new failures against the previously recorded set rather than the raw count. The delta is what makes it readable while standing reds exist — and it degrades gracefully: once the queue is green, "new failures" and "any failures" coincide. Costs a few minutes per merge, and merges are infrequent.main probe line. Cheaper to read, but a poll tick is not where anyone is positioned to fix a fresh regression, and running the suite per tick is far more expensive than per merge.rebuild_f5.sh's verify step. Wrong target — it would gate f5, which is already green. The failure mode this issue is about happens on main.I'd take A, with the new-vs-known delta as the part that actually matters. Recording the known-failing set somewhere (a checked-in file, or derived from the last run) is the only real design question.
probe.sh building first. Already in flight, and it removes the stale-DLL trap that produced four bad measurements from me today, including one where I misattributed two failures to another session's work.main's current failures (verified: main + #335 passes all three of #387's, and #382 handles the decode five). Worth doing before wiring any gate, so it starts from green.None.
No comments.