STALE-DISPATCH fires falsely: merging the heartbeat doesn't restart the dispatcher nf-xojv ← Beads

closed priority 2 task unassigned

Twinned from https://github.com/cfsh/ninefold/issues/461 by tools/beads/import_github.py's reconcile pass.

First live encounter with #437's liveness check, from ninefold-b's poll digest:

``` ⚠ STALE-DISPATCH: reviewer state dir exists but no heartbeat has EVER been written — the dispatcher has not completed a tick since #352's liveness check landed. Check the schtasks job and ~/.ninefold-reviewer/dispatch.log. ```

The dispatcher is fine. I checked before believing it:

Cause: #437 — the PR that *adds* the heartbeat — merged at 14:54:15 UTC. The dispatcher is a long-running scheduled job; merging code does not restart it. So it is still executing the pre-#437 binary, which has no heartbeat write, and poll.sh correctly observes "no heartbeat has ever been written" and incorrectly concludes the dispatcher is dead.

So the check works exactly as designed and is still wrong on its first real outing, because the deploy step for a long-running job is a restart, not a merge. Any session polling between #437's merge and the next dispatcher restart sees a red alarm about healthy infrastructure — and it is a loud one: it also pins POLL_CADENCE at 30m (held: reviewer dispatcher looks dead), so it suppresses backoff for every session at once.

Worth considering: - Restart the dispatcher as part of landing #437 (and note it in the PR/CLAUDE.md as the deploy step for that job). - Have the "never written" case degrade differently from the "stale heartbeat" case — a heartbeat that has *never* existed is at least as likely to mean "the check is newer than the running process" as "the process is dead". Something like no heartbeat yet — dispatcher may predate #437; check dispatch.log mtime before acting would have saved the investigation. - poll.sh could corroborate with dispatch.log's mtime, which is the ground truth I actually used.

Unrelated, spotted in the same log tail and worth a look while someone is in here: #447: skip (review in flight, lock < 45m old) — a lock from an earlier crashed/slow run suppressed a review for up to 45 minutes. #447 has since merged so nothing is owed, but a 45-minute lock on a synchronous reviewer is a long time to be silently skipping a PR.

Filed unclaimed. #437 is ninefold-a's chunk; not touching it.

Dependencies

None.

Comments

No comments.

Add a comment