Repository navigation
fix(bin): refuse Pi worker launches whose --model lacks an exact credentialed match - #4
Merged
Merged
Conversation
…hropic Pi's alias resolver can silently route a bare model like 'sonnet' to an uncredentialed provider (Amazon Bedrock, 2026-09-25), and a Claude Code style suffix like 'claude-opus-5[1m]' fails at launch too. Add pi_model_validate to bin/fm-spawn.sh: it resolves --model against the selected Pi executable's own '--list-models' catalog and refuses unless the model matches exactly one listed row whose provider is anthropic, naming what it resolved to (or that it matched nothing/several things) and the Anthropic ids that would work. An unreadable or empty catalog is refused rather than treated as an all-clear. codex-native/<id> stays exempt: it is the pi-codex-native extension's own runtime-registered provider, never a --list-models row, and is already gated by fm-harness.sh's validate_native_effort plus --effort ultra. The guard sits at fm-spawn.sh's single call site that already resolves PI_BIN and the final --model for every Pi/pi-signed launch path (crewmate, scout, secondmate, and relaunch), before any worktree, endpoint, or metadata is created. Empirically verified (recorded in the pi harness reference): --list-models lists only providers with credentials on this machine, so it can prove a model IS available but never that an unlisted or aliased model is safe. Two existing dispatch-profile tests used a made-up 'openai-codex/...' model string for pi/pi-signed purely to exercise --model/--effort threading; switched those to a real Anthropic id and taught the tests' fake pi to answer --list-models so the new guard does not block them. Adds a colocated regression (tests/fm-pi-model-guard.test.sh) pinning the five required cases with a stubbed catalog. Also manually verified against the real installed Pi catalog on this machine: 'fm-spawn.sh --harness pi --model sonnet' refuses with the concrete Anthropic ids, before creating any worktree or state, and without launching a worker. Out of scope, left for follow-up: other harnesses' model validation is unchanged; nothing here checks that a launched worker is actually processing; config/crew-dispatch.json is untouched.
…/fm-pi-model-guard.test.sh, a portable-serial script, without a duration hint in portable_serial_weight_hints (bin/fm-test-run.sh). That raised the unhinted count to 27 of 176, over the 15% limit, so the coverage guard failed. The guard fails directly in "Test coverage guard" and again inside tests/fm-test-run.test.sh, which is what failed in "Behavior portable parallel 1". Fix: added the line `tests/fm-pi-model-guard.test.sh 17097` to portable_serial_weight_hints, keeping alphabetical order. 17097 ms is that script's measured duration from this PR's own "Behavior portable serial 1" CI job (exit=0). Verified locally: `bin/fm-test-run.sh --check-coverage` passes (serial_unhinted=26); tests/fm-test-run.test.sh exits 0 with all 39 cases ok; tests/fm-pi-model-guard.test.sh passes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Guard it in the spawn tool: refuse to launch a Pi worker unless the model resolves to Anthropic, and say why.
Context for that ask: on 2026-09-25 three Pi workers were launched with --model sonnet. Pi resolved that bare alias to Amazon Bedrock's us.anthropic.claude-sonnet-5, which has no credentials on this machine, so each worker printed Error: No API key found for amazon-bedrock and sat idle at its prompt doing nothing, while its supervisor reported the work under way for about 40 minutes. Earlier the same day a deploy worker launched with --model 'claude-opus-5[1m]' (Claude Code syntax) exited at once with Error: Model "claude-opus-5[1m]" not found. The exact ids claude-sonnet-5 and claude-opus-5 resolve to the Anthropic subscription provider and work. Pi's defaultProvider: anthropic setting was already set and did not prevent the Bedrock resolution. The spawn tool launched all of these without objection.
What Changed
bin/fm-spawn.shaddspi_model_validate, which now runs before everypiorpi-signedlaunch. It reads<pi-bin> --list-models, which lists only providers that have usable credentials. The model is accepted only if it matches exactly one listed row, either by bare id or by exactprovider/id. The launch is refused, with a stated reason, when:sonnetclaude-opus-5[1m]Each refusal lists the unambiguous
provider/idselectors that would work. An empty ordefaultmodel is not checked.codex-native/<id>is also exempt becausefm-harness.shvalidates it separately.The Pi harness reference (
harness-adapters/references/harness/pi.md) now explains the difference between Pi's two model-selection surfaces.--list-modelsshows only credentialed providers, while--modelfuzzy-matches with no such filter. That gap is howsonnetresolved to Bedrock'sus.anthropic.claude-sonnet-5on 2026-09-25.Adds
tests/fm-pi-model-guard.test.shand registers it inbin/fm-test-run.sh. The fake Pi intests/fm-spawn-dispatch-profile.test.shnow answers--list-modelswith a small catalog, so existing spawn tests still pass the new guard.🤖 Generated with Claude Code
Risk Assessment
✅ Low: The fix-round change is a small awk filter. It keeps a suggested selector only when that selector matches exactly one row under the guard's own match rule (bare id or provider/id). A regression test covers a colliding selector: it fails on the old code, and it also confirms that the unambiguous selector the refusal suggests is accepted on retry.
Testing
First I reproduced both 2026-09-25 failures with the real Pi 0.87.1. Then I drove the real fm-spawn.sh with the real Pi catalog (only the tmux pane launch is stubbed), once on the target commit and once on the base commit. On the target commit, the incident models (bare
sonnet,[1m]suffix, unlisted Bedrock id) exit 1 with an explanation and a list of exact ids, and no worker or metadata is created. Exact ids and the gated codex-native pathway still launch. On the base commit, every one of these models launched. I also ran a suggested selector through real Pi, and it answered 'ok'. All live scenarios passed. The two targeted suites, fm-pi-model-guard and fm-spawn-dispatch-profile, pass. They cover the ambiguous, colliding-selector, unreadable-catalog and non-Anthropic cases, which this machine's single-provider live catalog cannot produce, so those cases remain untested live.--model sonnet: spawn refuses (exit 1), explains that Pi would fuzzy-match to a possibly uncredentialed provider, lists exact ids, and launches no workerclaude-opus-5[1m]: refused with reason, no worker launchedamazon-bedrock/us.anthropic.claude-sonnet-5): refused because it is absent from the real credentialed catalogclaude-opus-5(pi) andclaude-sonnet-5(pi-signed): launch proceeds with the model on the launch line and in metaprovider/idselector: spawn accepts it and real Pi runs itcodex-native/<id> --effort ultrapathway still launchesEvidence: Real Pi reproduces the incident (sonnet→Bedrock no key; [1m] not found)
Source: Real Pi reproduces the incident (sonnet→Bedrock no key; [1m] not found)
Evidence: fm-spawn with guard, real Pi catalog: refusals + accepted launches
Source: fm-spawn with guard, real Pi catalog: refusals + accepted launches
Evidence: fm-spawn at base commit: launches sonnet, [1m], Bedrock without objection
Source: fm-spawn at base commit: launches sonnet, [1m], Bedrock without objection
Evidence: Suggested exact ids really run in Pi
Source: Suggested exact ids really run in Pi
Evidence: Live driver script
Source: Live driver script
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 4 issues found → auto-fixed (2) ✅
bin/fm-spawn.sh:2082- The new guard refuses every Pi or pi-signed model whose catalog provider is notanthropic, including models that are credentialed and listed. The intent asks for exactly that ("refuse to launch a Pi worker unless the model resolves to Anthropic"), but the rest of the repo still treats non-Anthropic Pi profiles as supported.tests/fm-dispatch-resolve.test.sh:377-379resolves to--harness 'pi' --model 'openai-codex/gpt-5.6-sol'.tests/fm-bootstrap.test.sh:1142-1165accepts crew-dispatch profilespi:openai-codex/...andpi:kimi-code/k3.docs/verification/dispatch-auth.md:128-168documentspi:openai-codex,pi:xaiandpi:kimi-codingcandidates. After this change,fm-dispatch-resolvecan still recommend a Pi profile thatfm-spawnalways refuses. The commit also changed the two dispatch-profile tests fromopenai-codex/gpt-5.6-soltoclaude-opus-5instead of keeping that path covered. Please confirm that dropping Pi-hosted OpenAI-Codex, Kimi and xAI workers is intended. If it is, the dispatch resolver and config validation should stop offering those profiles. If it is not, the guard should refuse only unlisted, aliased or ambiguous models.bin/fm-spawn.sh:1606- The refusal message suggests bare Anthropic ids (anthropic_idsis built from column 2 only), but the guard itself refuses a bare id that matches more than one row. Example: a machine where another credentialed provider (such as github-copilot) also listsclaude-sonnet-5.--model sonnetis refused with 'use one of these Anthropic ids: ... claude-sonnet-5'. Retrying with--model claude-sonnet-5is then refused as ambiguous with the same suggestion, so the operator never gets a working id. Fix: build the suggestions asanthropic/<id>, which the guard's own exactprovider/idmatch accepts, at least when the bare id is ambiguous.bin/fm-spawn.sh:1611- The guard accepts a bare id such asclaude-sonnet-5when it matches a single listedanthropicrow. The new pi.md text and the guard's own comment say the listing hides uncredentialed providers and can never prove an aliased or unlisted id is safe. A bare id Pi resolves over its full catalog could hit the same blind spot if an uncredentialed provider has the same bare id. The intent reports that these exact ids work today, so this is not a current failure. A narrower option would be to passanthropic/<id>to Pi after the check, which pins the provider; that would change launch arguments, so it needs the author's decision.tests/fm-pi-model-guard.test.sh:17- The header comment says 'tests/fm-pi-model-guard-live-e2e.test.sh is the default-on live guard that proves the same refusal against the REAL installed Pi catalog', but the change does not add that file and it does not exist in the tree. The only live check is the manual one mentioned in the commit message. Remove the sentence or add the test.🔧 Fix applied.
2 issues (1 warning, 1 info) still open:
bin/fm-spawn.sh:1616- The refusal message now lists every catalog row asprovider/id, but the guard's own match treats$2 == mand($1 "/" $2) == mthe same way. So a suggested selector can match a second row whose bare id contains a slash, and the guard then refuses it as ambiguous. Example: an OpenRouter key is configured and Pi listsopenrouter anthropic/claude-sonnet-5next toanthropic claude-sonnet-5.--model sonnetis refused with a suggestion that includesanthropic/claude-sonnet-5. Retrying with that id matches both rows (the provider/id of the anthropic row and the bare id of the openrouter row), so it is refused as ambiguous. This is the same dead-end the previous round'sambiguous-suggestion-recommends-refused-idfix was meant to close. The new test only covers ids that contain no slash (github-copilot). Fix: buildselectorsonly from entries that match exactly one row under the same rule, e.g. drop anyprovider/idthat also equals some row's bare id.bin/fm-spawn.sh:1603- The stated intent says "refuse to launch a Pi worker unless the model resolves to Anthropic". The guard now accepts any provider with exactly one listed match (for exampleopenai-codex/gpt-5.6-sol). This departure is on purpose: the user's fix-round instructions forpi-guard-blocks-documented-non-anthropic-pi-profilesremoved the Anthropic-only check and treat an exact match against--list-modelsas the credential check. It still closes the incident paths (sonnet,claude-opus-5[1m], and unlisted Bedrock ids are all refused), so no action is needed.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
--model sonnet: spawn refuses (exit 1), explains that Pi would fuzzy-match to a possibly uncredentialed provider, lists exact ids, and launches no workerclaude-opus-5[1m]: refused with reason, no worker launchedamazon-bedrock/us.anthropic.claude-sonnet-5): refused because it is absent from the real credentialed catalogclaude-opus-5(pi) andclaude-sonnet-5(pi-signed): launch proceeds with the model on the launch line and in metaprovider/idselector: spawn accepts it and real Pi runs itcodex-native/<id> --effort ultrapathway still launchespi --model sonnet -p ...andpi --model 'claude-opus-5[1m]' -p ...against real Pi 0.87.1 (reproduces the incident: 'No API key found for amazon-bedrock' and 'Model "claude-opus-5[1m]" not found')bash drive-live.sh <worktree>: runs the real bin/fm-spawn.sh with the real Pi binary and real HOME answering --help/--list-models; only tmux/treehouse are stubbed. Models: sonnet, claude-opus-5[1m] (pi-signed), amazon-bedrock/us.anthropic.claude-sonnet-5, claude-opus-5, claude-sonnet-5 (pi-signed), anthropic/claude-opus-5-5, codex-native/gpt-6-astra --effort ultrabash drive-live.sh <base 1e8058e archive>: same drive against the pre-change fm-spawn, for comparisonpi --model claude-sonnet-5 -p ...andpi --model anthropic/claude-sonnet-5 -p ...(a suggested selector really runs in Pi)bash tests/fm-pi-model-guard.test.shbash tests/fm-spawn-dispatch-profile.test.sh✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.