where would CI run? hosted Windows runner vs a self-hosted runner on the sandbox box nf-uz62 ← Beads

Twinned from https://github.com/cfsh/ninefold/issues/417 by tools/beads/import_github.py's reconcile pass.

Follow-up to #389 / PR #407, filed at Aedan's request — he asked specifically whether a presubmit *"requires cloudifying or can be done locally, and can we run CI locally on the dedicated sandbox machine somehow."*

Companion to the merge-result presubmit issue: that one is the what, this is the where. This is the actual blocker — the design is easy, the execution environment is not.

Everything below is reasoning, not measurement. Nothing here has been tried. The whole point of the issue is that one afternoon of spiking replaces all of it with facts.

Starting position

.github/workflows/ does not exist. The repo has zero CI of any kind today. So this isn't "extend CI to cover probes", it's "introduce CI".

The probes need a real Godot editor: probe.sh launches the console exe, drives input, and captures 1920×1080 screenshots. run_tests.sh (xUnit, no Godot dep) would run anywhere — probes are the hard part.

Option A — GitHub-hosted windows-latest

"We'd need a Windows VM" isn't quite the obstacle: GitHub rents one per job. Real considerations:

• Cost. The repo is private, so minutes are billed, and Windows bills at 2× the Linux rate. At 3m28s a sweep plus Godot download/cache, and presubmit firing on every push across ~7 active PRs, this is the thing to price before building.
• Rendering — the genuinely open question. project.godot sets renderer/rendering_method="gl_compatibility" with rendering_device/driver.windows="d3d12". Hosted runners have no GPU. Whether Godot falls back to software rendering reliably enough is unknown, and pixel-stable screenshots on a *different renderer than any developer runs* would be the flakiest thing in the repo.

The encouraging split, counted across tools/probes/*.json on 2026-08-02:

| step kind | count | renderer-sensitive? | |---|---|---| | expect_* assertions (focus, zone, pattern, captures) | 143 | no | | shot (screenshot) | 93 | yes |

The regression-catching value is overwhelmingly in the assertions — #387 was caught by expect_focus, not by eyeballing a PNG. So a plausible split is assertions in CI, screenshots local. That also matches CLAUDE.md's existing rule that PRs changing behaviour must add probe *assertions*, not just shots.

Option B — self-hosted runner on the sandbox box

The box already has the exact working environment: Godot 4.7.1 mono, .NET 8 + 10, a real GPU, and a proven-green suite. A GitHub self-hosted runner is a service registered to the repo that picks up jobs.

• Zero rendering risk — identical to how probes run today, which is the whole appeal.
• No per-minute billing.
• Costs: the box must be on and online for CI to work, so a presubmit becomes a hard dependency on one machine nobody is obliged to keep running. CI silently stops when it's off or busy.
• Contention is the sharper issue: that box is already running several Claude sessions, and probe.sh launches real Godot windows and grabs input. A CI probe run firing while a session runs its own sweep — or while Aedan is F5-ing — would fight over the desktop. Probe runs are not currently safe to run concurrently, and nothing enforces serialisation.
• Security: self-hosted runners on a private repo are the ordinary case, but they do execute PR code on a machine with real credentials. Worth a thought before enabling, not a blocker for a private repo with two committers.

Option C — keep it local, drop GitHub

Not really CI, but honest to name: the current merge_pr.sh gate already runs on the sandbox box. The gap isn't the machine — it's that it can't report a status check, so it's a convention (SKIP_PROBE_GATE=1, or merging from the GitHub UI, bypasses it silently). If we don't want to run infrastructure, the realistic ceiling is what we already have.

The spike that answers this

Roughly an hour, and it collapses most of the above:

1. Minimal workflow on windows-latest: install Godot mono, dotnet build, ./run_tests.sh. Proves the boring half works and prices a run.
2. Add ONE probe (tools/probes/cold-start.json) and see whether Godot renders at all without a GPU.
3. If yes: does an assertion-only subset pass consistently across, say, 5 runs? Flakiness is the deciding number, not pass/fail once.
4. Compare against registering a self-hosted runner and doing the same.

Outcome is a real recommendation instead of this issue's guesswork.

Related: the merge-result presubmit issue, #389, #407, #414 (shell-tier harness — same "no automated coverage" theme, different tier).