probe_gate: report the SPAN it swept, instead of claiming one merge caused it nf-gymf ← Beads

open priority 2 task unassigned

Twinned from https://github.com/cfsh/ninefold/issues/478 by tools/beads/import_github.py's reconcile pass.

Unclaimed and unlabelled — inert backlog, per CLAUDE.md.

Split out of #412 (comment 5) so that issue can close cleanly with #472. The cheap half of the same ask — *"don't sweep a sha that's already recorded"* — landed in #472; this is the structural half.

The ask, verbatim

> Back-to-back merges collapse into one sweep, so the gate loses per-merge attribution — and its "this merge regressed main" claim can misattribute. > > Report the span, don't claim the merge.

Why the claim can be wrong

probe_gate runs post-merge on main, compares against the previous recorded sweep, and prints:

``` ████ NEW FAILURES — this merge regressed main ████ ```

That sentence is only true when exactly one merge separates the baseline's sha from the swept sha. Two things break it:

The since field already records the sha a failure *first appeared at*, so the data to do better is present — what's missing is that the headline asserts a cause the tool cannot know.

Shape of a fix, uncosted

The baseline stores the sha it measured. git rev-list --count <baseline-sha>..<swept-sha> gives the number of merges in the span, which is enough to pick honest wording:

The paste-ready gh issue create should carry the same span, since that text becomes a public issue asserting a cause.

Why it is worth doing at all

The gate's value is that a human is present at the exact moment and knows which merge did it. A confident-but-wrong attribution spends that value: the named merge gets investigated, comes back clean, and the next ████ banner is read as noise. That is the same erosion #461 caused for STALE-DISPATCH, and the same argument #389 makes about a gate with permanent reds.

Related: #471 (a flaky probe makes any attribution unreliable regardless of span) — worth reading together, since a span fix and a flake fix address the same "can this banner be trusted" question from different sides.

Dependencies

None.

Comments

No comments.

Add a comment