Twinned from https://github.com/cfsh/ninefold/issues/682 by tools/beads/import_github.py's reconcile pass.
Our probe testing infra has systemic issues
* probes were required to run for completely unrelated md changes in different folders, or code changes to tooling that doesn't affect UI (some progress made to patch these in recent issues and PRs) * probes are a single bundle, changes to explore shouldn't require re-running tonapse probes and vice versa. * Changes to shared components (e.g. soon we should re-use modal popup across tonapse and explore for game menu), need to trigger the right probes in both, but probably not all gameplay probes * as a technical sub-point, individual probes are stored under /tools, instead they should be stored next to the relevant code / games. * some probes are flaky in some systems, currently tracked through ad-hoc state files, instead flaky probes should be marked as such in the repo * Probes are an expensive part of the presubmit queue. * There is an issue with running probes in parallel, but this is lower priority as _active_ development is usually restricted to 2-3 workstreams. * probes run as a post-merge, instead is it feasible to make it gate merge if they run faster and less flaky (or build in handling that inside the merge script, rather than being handled by a full re-run of merge script). Ideally they can run as presubmit logic.
Design fixes to these problems, that will scale in the mono repo as we add 5+ more games, and for each game add many scenes and gameplay within. (e.g. probes will need heirarchy within games too, changes to one scene don't require probes on another scene)
My past experience inclines me to bring in a build configuration language to allow specifying and maintaining the dependency relationships between probes (and other tests), and code files. We can make a simplifying assumption that probes only run on the Linux VM, so we don't need windows support (if it doesn't make the solution any more complicated, having probe infra & presubmit work on WSL would also be nice).
Consider if build configuration is the right solution, or if other superior practices exist in the non-google world.
See as a technical detail https://github.com/cfsh/ninefold/pull/485 - fold this PR into your design work (since your research will cover similar/same topics), and ratify or adjust the conventions its proposing too (and again build config is related here e.g. "foo.py" living alongside "foo_test.py"). I haven't finished this PR, but the full conventions I'm thinking are:
See also for testing, we prefer real code > fakes > stubs > mocks ("real → fake → stub → mock. Martin Fowler ("Mocks Aren't Stubs") independently" from design doc 001 - this should get up-levelled to style guide).
None.
No comments.