A probe whose assertions all passed can still be recorded FAIL, when Godot crashes during shutdown nf-dnlc ← Beads

Twinned from https://github.com/cfsh/ninefold/issues/691 by tools/beads/import_github.py's reconcile pass.

Found while verifying the post-merge gate on #685 (a docs-only merge, so nothing could have regressed).

What happened

The gate reported 33 passed, 1 failed with split-notes-editor as FAILING HERE, PASSING IN MAIN'S BASELINE — i.e. a paste-ready "this merge regressed a probe" report, on a diff of two markdown files.

Re-run standalone 3× on an idle-ish box (loadavg ~5): PASS, FAIL, PASS.

The FAIL run is the interesting one. Its log ends:

``` [probe] expect_prop PlayheadShown 'lit Loop' OK [probe] expect_prop PlayheadAdvancing 'yes' OK [probe] shot probe_out/split807-02-looping.png (1906x807) [probe] done ... ERROR: Leaked unsafe reference to object: ():<StyleBoxFlat#...> at: finalize (modules/mono/csharp_script.cpp:179) ERROR: FATAL: Index p_index = 1 is out of bounds (size() = 0). at: get (./core/templates/cowdata.h:197) handle_crash: Program crashed with signal 4 ```

Every assertion passed. [probe] done — not done WITH ERRORS — means _failed was false and VisualProbe called GetTree().Quit(0). Godot then crashed *during teardown*, after the probe's work was complete, and the crash set the process exit code.

Why it reports as a probe failure

probe.py:173 returns result.code — the raw process exit code — so "all assertions passed, engine crashed on shutdown" and "an assertion failed" are the same signal. probe_all.py sees FAIL, probe_gate.py diffs it against the baseline and reports a regression against whatever merge happened to be running.

Why this is worth separating from the known flake work

#615 resolved three "flaky" probes into three causes wanting opposite fixes (a real shipped game bug, an over-specified pixel assertion, host contention). #668 measured the frame-counted-wait exposure corpus-wide. This is a fourth cause and none of those fixes touch it — the probe's timing was fine, its assertions were right, and converting its 10 frame waits to wait_ms would not have prevented it.

It also has a distinct signature that is cheap to detect: [probe] done on stdout together with a nonzero exit code. That is unambiguous — the harness is stating it finished cleanly, and the process disagrees.

Suggested direction (not claimed)

1. probe.py distinguishes the two outcomes rather than passing the exit code through: assertions-failed vs completed-then-crashed. The second is a host/engine event, not a probe verdict.
2. probe_all.py gets a result state for it so probe_gate.py can decline to attribute it to a merge — the same reasoning as #662's wipeout handling, which already refuses to blame a merge for an all-red host failure. This is the single-probe version of that.
3. Worth checking whether the shutdown crash is deterministic to a probe that plays audio to completion (PlayheadAdvancing), since the dummy audio driver is in the log too.

Context

Observed on nf-dev-sg, Godot 4.7.1 stable mono, xvfb-run -a, main @ d88de8e. Related: #615 (three causes), #668 (frame-counted waits), #662 (wipeout attribution), #514 (a flake absorbed into the baseline). Background on why the failure set moves between sweeps: design_docs/005-probe-selection-and-build-config.md §2.2 — an earlier sweep the same night on a near-identical tree failed a disjoint set (butterbar-gate, focus-ring, playhead-song), all three of which passed in this one.