Repository navigation
feat: centralize fleet model selection in a role-based index - #13
Merged
Merged
Conversation
MrGTV-love
force-pushed
the
fm/fm-model-index
branch
from
October 6, 2026 18:20
20f190b to
58fbd98
Compare
MrGTV-love
force-pushed
the
fm/fm-model-index
branch
from
October 7, 2026 01:31
381f36d to
1442638
Compare
…drop scratch test
… withhold dispatch pair
…yphenated test id
…fixed both. The other two did not, so I changed nothing for them. 1. Lint 1 (fixed). ShellCheck SC2034: `tests/fm-control-relaunch.test.sh` line 766 declared a variable, `catalogs`, that nothing used. I removed it from that line. The earlier local lint missed it because local changed-file mode skips SC2034. A full lint run on all changed shell files, with the changed files named so SC2034 runs, was still running when I sent this result, so it is not confirmed. 2. Behavior portable serial 5 (fixed). The bootstrap test "array profile with malformed model is flagged" failed. The cause was `retired_guard` in `bin/fm-model-index.sh`. It called `retired()` on any `model` value. When `model` was the number 5, jq crashed with "number (5) cannot be matched". That crash message replaced the expected "must be non-empty strings" error. The fix runs the retired check only when `.model` is a string. A non-string model still reaches the normal checks, which reject it. Results after the fix: - `tests/fm-bootstrap.test.sh`: rc=0. - `tests/fm-model-index.test.sh`: rc=0. - `tests/fm-dispatch-resolve.test.sh`: rc=0. - `tests/fm-spawn-dispatch-profile.test.sh`: had 54 passes and 0 failures, but had not finished when I sent this result. 3. Behavior portable serial 2 ("rendered export DOM violated the Calm conversation boundary") did not come from this PR. This PR changes no Pi or Calm files. The same test fails in the same shard on PR 12 (run 36781504338) and PR 11 (run 36778047014). This looks like a change in the external Pi version, but I did not confirm that. 4. Behavior portable serial 4 ("Pi outcomes rendering consumers must preserve stock behavior") also did not come from this PR. It fails on PR 12 and PR 11 in the same way, with the same likely cause
…e serial 8 failed because the secondmate test fixture allowed crew-dispatch.json inheritance but not its coupled model-index.json destination. Reproduced the exact CI assertion locally, then corrected all three positive selective-ignore fixtures while preserving intentional refusal cases and production guards. Fixed Lint 1 by quoting the literal model="pane-only" without changing behavior. Updated the existing verification documentation. Verification passed: focused secondmate-harness, bootstrap, model-index, and worker-account suites (total=4, failed=0); canonical full-analysis lint for all four changed test scripts; two-home production inheritance smoke covering dispatch copy and absence convergence; documentation audience check. One optional Cursor case skipped because its executable was unavailable. The full CI shard and live Herdr suite were not rerun
…and registry checks
…heritance boundaries
…res isolated login
…outing inheritance documentation
…nd retirement semantics
…ispatch envelopes
…ate stale guidance
…relaunch fixtures
MrGTV-love
force-pushed
the
fm/fm-model-index
branch
from
October 7, 2026 08:40
493ad77 to
c635550
Compare
…l six named suites without changing production behavior, CI timeouts, skips, or shard placement. Remote spawn now uses five coverage-preserving selector/boundary launches instead of eight and avoids unrelated inherited-config transfers. Model-index fixtures retain all distinct guard, coherence, source-safety, mutation, schema, account-context, and no-jq cases while narrowing inheritance fan-out, removing 72 redundant receiver calls, 20 redundant remote recovery pushes, and 12 redundant local recovery calls. Bootstrap shares immutable setup while retaining all 14 caller/state executions. Dispatch removes two duplicate resolver calls and zero-duration sleep processes. Relaunch retains every refusal’s survivor notification proof, removes six repeated post-clear bootstrap runs, shares selection fixtures, and reduces the cwd-race fixture from ten fixed delays to one finite first-read delay. Contributor guidance now documents focused inheritance fixtures. Verification: all six suites passed through `bash bin/fm-test-run.sh --jobs 1` with zero failed suites and zero gate-skipped suites. Local durations: model-index 222.4s, remote compact-adviser spawn 349.8s, bootstrap 175.8s, dispatch-resolve 55.9s, control-relaunch 524.1s, remote-secondmate-relaunch 86.7s. The same-host model-index baseline was 436.2s, giving an observed reduction of approximately 49%. Dispatch’s existing EPOCHREALTIME capability case skipped under stock Bash. Full pinned ShellCheck 0.11.0 passed for all six changed roots, including the corrected race fixture; documentation ownership checks passed. Darwin timings do not establish Linux shard headroom. No pipeline control, push, or CI rerun was invoked; the outer executor owns the provider rerun
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Question: "If were able to standardize, would it be better to have a standard model index so we did not need to make multiople updates for every model change?"
Context given with the decision page: on 2026-09-29 switching from gpt-6-sol to gpt-6.1-sol needed hand edits in config/crew-dispatch.json (2 model lines and 4 rule texts naming the id), the 7 lane-home copies of that file, memory and rule prose, and Vernant's separate product registry (scripts/model_registry.json, handled in Vernant GRE-1872). The captain's same-day rulings: "use 6.1 sol instead of 6.0 sol. 6.1 released today and it is smarter and more efficient than its predecessor" and "for the purposes of model alignment, gpt-6.1-sol should be considered a suitable alternative to opus 5.5". Standing rule: a retired or prior-generation model id is a hallucination alarm.
Answer chosen on the decision page: "Model index: A-fleet-index". Option A as presented: one fleet model index maps roles to current ids (for example opus-grade: Claude opus[1m] or Codex gpt-6.1-sol; sonnet-grade: claude-sonnet-5-5[1m], stand-in GLM 5.3; easy: gpt-6-luna, stand-in GLM 5.3 Flash); dispatch profiles name the role, never the id, and the spawn step looks the id up at launch; rules and memory say the role; a change to the index is checked against each harness's live catalog and refused if an id is absent; the index propagates to lane homes like other inheritable config; Vernant keeps its own product registry, checked against the index's retired list. Next model release: one edit to the index.
What Changed
config/model-index.jsonandfm-model-index.shfor per-harness model roles, explicit stand-ins, retired-model rejection, and read-only registry checks against the retired list.Risk Assessment
Testing
Public CLI, inheritance, receiver, and private-tmux launch scenarios passed, as did focused regressions after correcting disposable socket/config/root setup; evidence captures emitted commands, diagnostics, persisted configuration, metadata, and mutation timing. Workers remained at trust dialogs, so no worker model turn is claimed; default-secondmate replacement was not live-exercisable within the workspace boundary. Remote receiver checks were direct, not SSH lifecycle checks. All fixtures were removed; no full suite, lint, formatter, or static analysis ran.
Evidence: Model selection, native catalog, retirement, and envelope evidence
Source: Model selection, native catalog, retirement, and envelope evidence
Evidence: Typed intake malformed-dispatch guards
Source: Typed intake malformed-dispatch guards
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 2 issues found → auto-fixed (5) ✅
tests/fm-bootstrap.test.sh:1298- The new remote-notification fixture checks python3 on the original PATH at line 1269, then invokes it through a restricted PATH that excludes Homebrew, virtualenv, and /usr/local/bin installations. If Python is available only there, prerequisite detection succeeds but fake SSH cannot execute it, so the cases fail before exercising inheritance or notification. Both caller paths are affected: tests/fm-bootstrap.test.sh:1343 runs guarded fm-config-push/fm-bootstrap scenarios; :1380 runs ordinary convergence. Resolve Python once and expose that exact executable in each fixture fakebin, following the existing jq dependency pattern at :1291.tests/fm-dispatch-resolve.test.sh:1518- The new native-account and unknown-context cases inherit the operator's documented FM_MODEL_CATALOG_DIR override despite requiring native discovery or no export. For example, with an exported missing catalog directory, this pinned-Claude case never invokes its fake native catalog and leaves claude-catalog-roots unwritten; a readable unrelated export can instead reject synthetic IDs in the unknown-context cases. The checker intentionally prioritizes exports at bin/fm-model-index.sh:129 and permits them despite unknown worker context at :174. Clear the ambient override for these cases while preserving explicitly supplied fixture exports. Other affected sites: tests/fm-dispatch-resolve.test.sh:1547 — unknown Codex/Pi intake; tests/fm-control-relaunch.test.sh:1025 — unpinned indexed relaunch; tests/fm-agy-harness.test.sh:913,943 — indexed role and literal launches; tests/fm-worker-account.test.sh:410,432 — pinned Claude/Pi catalogs; :470 — unpinned Codex/Pi/Pi-signed contexts; :501 — ordinary Claude context; :554 — indexed Cursor/omp selections.🔧 Fix applied.
✅ Re-checked - no issues remain.
bin/fm-secondmate-restart.sh:285- The prior role-resolution fix (dafa7b4) leaves remote restart using two different index generations. Concrete sequence: a remote mate retains an index mapping its role to model-old; the parent changes that role to model-new, then runs the documented update/restart flow before config convergence. This code resolves model-new from the parent index but forwards only that literal, without propagating the index. Even when the remote mate has a valid pinned Claude/Pi account whose readable catalog excludes model-new, its pre-stop check finds no matching index entry and therefore performs no catalog check; replacement spawn likewise treats model-new as a non-entry literal. The working agent is stopped for a replacement that a converged index would reject beforehand. Affected changed siblings: bin/fm-control.sh:954-956,974-976 — resolution and pre-stop check against remote config; bin/fm-spawn.sh:2436-2442,2577-2582 — membership classification and conditional catalog gate. The unchanged transport path is bin/fm-remote-secondmate-relaunch.sh:51-56 → bin/fm-remote-secondmate-control.sh:277-281, which explicitly disables inheritance. Reuse the existing guarded routing-pair propagation before remote relaunch, ensuring resolution and remote pre-stop validation use the same selected pair; if propagation refuses, leave the running mate intact. No new durable state is needed.🔧 Fix applied.
1 warning still open:
bin/fm-remote-secondmate-relaunch.sh:72- The R9-1 fix round (c7fb4f4) introduces live inheritance without preserving reread intent when relaunch refuses. Concrete sequence: the parent changes a role from model-old to model-new; this wrapper publishes the new routing pair; the destination's pinned catalog excludes model-new, so pre-stop validation refuses and the old secondmate remains running. No reread or pending marker is recorded. On the next bootstrap, tracked files are already current and inherited files are unchanged, so bootstrap removes its newly created marker without notifying the surviving worker. Configuration changes can therefore remain unannounced indefinitely. The same invariant applies at bin/fm-remote-secondmate-relaunch.sh:75-79 (partial inheritance failure after unrelated writes), :86-90 (relaunch refusal), and :99-105 (missing route confirmation). Reuse the existing remote nudge marker before transfer, retain it across unsuccessful or unconfirmed replacement, and clear it only after confirmed replacement or successful reread delivery. Existing sibling notification patterns are bin/fm-config-push.sh:199-230 and bin/fm-bootstrap.sh:614-649. Extend the refusal scenarios at tests/fm-remote-secondmate-relaunch.test.sh:357-373 to cover subsequent notification of the surviving worker; no new persistence machinery is needed.🔧 Fix applied.
1 warning still open:
bin/fm-spawn.sh:1071- The round-6 frozen-pair fix and R9-1 remote-relaunch fix leave launch/control siblings selecting and validating different index generations. Concrete remote sequence: role:fast resolves to model-A; during the readiness/sync hops, the operator changes the coherent routing pair to fast=model-B with retired=[]; inheritance publishes that later pair, but launch forwards model-A. Destination spawn now classifies A as a non-entry literal and skips its selected-entry catalog check, allowing a launch inconsistent with the inherited role—even when the destination's authoritative catalog excludes A. Remaining sites: bin/fm-spawn.sh:1145,1170 — later inheritance and earlier-model forwarding; :2437,2438,2582 — independent resolution, membership, and catalog snapshots; :3185 — local secondmate inheritance after selection. The pre-stop sibling at bin/fm-control.sh:956,976,1168 has the same failure: changing the role while the pinned account's external authentication check runs makes the newer index omit the previously selected id, so validation succeeds and control stops the working agent for an unchecked replacement. bin/fm-model-index.sh:205 is the shared exact-model filter that skips these formerly indexed selections. Both role and stand-in selectors, supplied explicitly or through secondmate-harness, are affected. Freeze one routing pair at the launch/control selection boundary and reuse that generation for resolution, membership, catalog validation, inheritance, and replacement launch, using the existing staging mechanism without changing account pins. The staged remote-relaunch wrapper already preserves this invariant.🔧 Fix applied.
3 issues (1 error, 2 warnings) still open:
bin/fm-spawn.sh:1071- The round-6 frozen-pair fix and R9-1 remote-relaunch fix leave launch/control siblings selecting and validating different index generations. Concrete remote sequence: role:fast resolves to model-A; during the readiness/sync hops, the operator changes the coherent routing pair to fast=model-B with retired=[]; inheritance publishes that later pair, but launch forwards model-A. Destination spawn now classifies A as a non-entry literal and skips its selected-entry catalog check, allowing a launch inconsistent with the inherited role—even when the destination's authoritative catalog excludes A. Remaining sites: bin/fm-spawn.sh:1145,1170 — later inheritance and earlier-model forwarding; :2437,2438,2582 — independent resolution, membership, and catalog snapshots; :3185 — local secondmate inheritance after selection. The pre-stop sibling at bin/fm-control.sh:956,976,1168 has the same failure: changing the role while the pinned account's external authentication check runs makes the newer index omit the previously selected id, so validation succeeds and control stops the working agent for an unchecked replacement. bin/fm-model-index.sh:205 is the shared exact-model filter that skips these formerly indexed selections. Both role and stand-in selectors, supplied explicitly or through secondmate-harness, are affected. Freeze one routing pair at the launch/control selection boundary and reuse that generation for resolution, membership, catalog validation, inheritance, and replacement launch, using the existing staging mechanism without changing account pins. The staged remote-relaunch wrapper already preserves this invariant.bin/fm-spawn.sh:2440- The R11-1 fix (c104dea) unnecessarily extends mandatory preselection staging to local secondmates with no selected model. Concrete sequence: a working Claude secondmate has a bareclaudeharness pin and a dangling routing-source symlink or nonregular routing source. Default-model control skips staging, stops the agent, then replacement spawn exits at line 2443; rollback leaves no agent running. Fresh launches also fail before unrelated inheritance can continue. This contradicts the existing warning-only local propagation contract at .agents/skills/secondmate-provisioning/SKILL.md:111-115. No model selection requires this extra branch. Revert this portion of the round to the narrower fix: freeze at model selection, while default-model local launches retain the existing guarded, warning-only inheritance boundary. Related sites: bin/fm-control.sh:939,956 — default selection skips pre-stop staging; :1163,1173-1180 — stop and replacement; :873-874 — stopped worker remains stopped; bin/fm-secondmate-restart.sh:174-175 — update/restart caller; bin/fm-spawn.sh:3204-3206 — intended warning-only propagation.bin/fm-model-index.sh:255- The round-6 routing-coherence fix still approves malformed present dispatch envelopes. Unlike the round-7 index/catalog validators,profilesaccepts an empty file with success and transforms concatenated objects independently. With a valid index and a zero-byte crew-dispatch.json, config-push approves and publishes the empty dispatch over a working destination configuration; typed intake subsequently rejects that same file. Concatenated dispatch objects likewise propagate, while typed resolution consumes only the first object, potentially ignoring an appended approval gate. Require exactly one object at the shared present-dispatch boundary, preserving index-only validation by representing dispatch absence explicitly rather than treating a present empty file like the /dev/null sentinel. Related sites: bin/fm-model-index.sh:258-259 — separate streaming warning pass; bin/fm-config-inherit-lib.sh:605-620,636 — shared coherence validation and local publication gate; bin/fm-config-push.sh:139,160-162 — fleet-wide approval; bin/fm-remote-inherit-push.sh:53 — remote approval; bin/fm-spawn.sh:1018 and bin/fm-remote-secondmate-relaunch.sh:63 — launch/relaunch approval; bin/fm-bootstrap.sh:1046 — diagnostic consumer; bin/fm-dispatch-resolve.sh:163-168,422,470 — stream acceptance followed by first-document consumption.🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Test** - 1 issue found ✅
bash tests/fm-model-index.test.sh— completed with exit 0.FM_MODEL_CATALOG_DIR=<missing disposable export> bash tests/fm-worker-account.test.sh— completed with exit 0 after correcting the runner deadline and restoring the suite's supported temporary-fixture placement.bin/fm-model-index.sh model,profiles, andcheck-registryagainst disposable indexes, dispatch profiles, and registries.bin/fm-model-index.sh checkagainst real Claude initialization and realomp models --jsonlistings, accepting listed IDs and refusing deliberately absent IDs.Public CLI checks for concatenated index documents, concatenated catalog exports, missing catalog evidence, retired identifiers, and explicit stand-ins.Realbin/fm-config-push.shpublication into product-seeded lane homes, followed by the destination's realprofilesconsumer.Real config-push refusals for a catalog-absent model and a removed dispatch role; verified preservation of both routing files and convergence of unrelated configuration.Paused only the disposable config-push process after native validation, mutated its source pair, resumed it, and verified publication of the previously validated bytes.Ran real config-push with both routing sources absent and a dependency PATH containing no jq; verified removal of both destination routing files.Drovebin/fm-remote-inherit.sh put,check, andabsentagainst an isolated receiver, including linked-sibling and oversized-payload refusals.Launched and relaunched a real omp worker throughfm-spawn.shandfm-control.shon private tmux sockets; inspected native panes, persisted metadata, and actual process arguments.Resolved a Claude role and passed its selector to the real machine-login Claude CLI in print mode; receivedFM_MODEL_INDEX_LIVE_OKwithout tool access or session persistence.Stopped all owned private tmux servers, tore down their helper-owned sockets, and removed disposable workspace materials.✅ No issues found.
bin/fm-model-index.sh model,profiles,check, andcheck-registryagainst disposable indices, dispatch files, catalog exports, and registries.Real Claude SDK initialization withclaude -p --input-format stream-json --output-format stream-json --verbose --no-session-persistence --setting-sources '', followed by native-catalog acceptance and absence checks.bin/fm-dispatch-resolve.sh <brief>with empty, concatenated, nonobject, and valid single-object dispatch inputs; network proxies restricted to loopback.propagate_secondmate_inheritancethrough the actual shared executable API, checking persisted destination configuration, pair retention, filesystem guards, link compatibility, and absence without jq.bin/fm-config-push.shagainst isolated source and destination homes for removed-role, missing-index, malformed-dispatch, tracked-file, and nonregular-destination cases.bin/fm-remote-inherit.sh check|put|absentdirectly against disposable homes, including payload integrity, generation conflicts, byte bounds, and linked or nonregular destinations; no live SSH transport claimed.Real private-tmux Claude primary andbin/fm-spawn.sh live-role|live-edit|live-standin <project> --scout --harness claude --model role:strong|stand-in:strong, with concrete launch commands and metadata captured.Synchronized source-index mutation after actual spawn staging, followed by inspection of the launched model and a catalog-refusedbin/fm-control.sh live-race relaunch --harness claude --model role:strong --note 'Remain idle.'.Selected executable regressions fromtests/fm-control-relaunch.test.sh: model-index pre-stop refusal, frozen generations across authentication and replacement, unknown account context, and default-secondmate unsafe routing sources.bash tests/fm-remote-secondmate-relaunch.test.shand containedbash -x tests/fm-model-index.test.shas supplemental non-live regression checks.bin/fm-teardown.sh <lab-task> --force, private-socket server shutdown, and removal of disposable homes, projects, pools, drivers, and transient fixtures.✅ **Document** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.