fix(secrets): the Claude coding-plan token had no slot to arrive through - #2700
Conversation
archon boots healthy, passes its health check, and dies on the first workflow with `billing_error: Credit balance is too low`. The cause is not a missing value, it is a missing slot. env.tier-agent.example has declared CLAUDE_CODE_OAUTH_TOKEN since PR #2359, and both compose files wire `${CLAUDE_CODE_OAUTH_TOKEN:-}` into the archon service. But the label was never in the CHIT registry, so the funnel had no route for it and the live env.tier-agent carries none of the five Claude auth vars the example promises. Meanwhile ANTHROPIC_API_KEY is a registered slot, is set fleet-wide from env.shared, and the Claude CLI PREFERS it over the OAuth token. archon says so in its own source: "the CLI prefers ANTHROPIC_API_KEY over the OAuth token, so injecting the install key alongside it would silently rebill the run". So the metered key wins and the coding plan never gets a chance. MODEL_FABRIC_CONTRACT.md:44 already lists `Claude Code Max` in the approved coding-plan inventory. This is a missing slot, not a new policy decision. Registered via the sanctioned path only -- registry entry, then chit-manifest-register, then chit-manifest-sync. No manifest was hand-edited. NECESSARY BUT NOT SUFFICIENT. This routes the label; it does not conjure a value. secrets_sync.py skips an entry entirely when no source value exists (`if source_key is None: continue`), so until a token is provisioned into the CGP source the key is emitted NOT AT ALL -- not emitted empty. That is the same blank-is-not-absent rule #2688 established: an empty secret is not a secret, so absent stays absent. archon therefore behaves exactly as it does today until the value lands. No regression, and no fix yet either. SECOND FIX, unavoidable and deliberate: the same sync emits `min_length: 64` onto secret_key_base in the v1 manifest. PR #2688 added that constraint to the registry, to v2, and to the sync logic, but never re-ran the sync -- and secrets_sync.py reads v1 (tools/secrets_sync.py:336). So the min-length gate shipped in #2688 has been INERT on main: the constraint lived in v2 and in the tooling while the file the enforcer reads never carried it. It could not be excluded here without hand-editing a machine-emitted file, which the pipeline forbids, so it ships with this change rather than silently. Gates: - chit-manifest-register --check: manifest complete, no constraint drift - negative-tested: an unregistered label fails the check with Error 1 - secrets-funnel run end to end: 0 errors, env.tier-agent regenerated Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X9mpK1ZFjexjthZbvcb94f
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔏 CHIT-aware change — control-body routingThis PR touches a CHIT-aware service (ports 8086/8087/8103/8106/8113/9224 surface). Before a Control-Body ACK:
Advisory routing only — the blocking contract check is CHIT Contract. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb3db3f2b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review on #2700 caught that the manifest slot alone does not reach production or self-hosted nodes. sync-secrets-local.yml builds the CGP bundle solely from its explicit env: map, and the v1 sync discards github_secret targets -- so a registered label with no row in that map funnels nothing on any node hydrated from the bundle. Archon would keep the metered-key behaviour this change set exists to fix. The workflow says so in its own comment, two lines above where the key was missing: "Absent from this map = never delivered." Adds CLAUDE_CODE_OAUTH_TOKEN beside the other prod-environment-scoped coding-plan keys (ALIBABA_PRO_CODING_PLAN, KIMI_CODING_API, KILOCODE_API_KEY), which is where it belongs -- it is the same class of credential and resolves the same way under environment: prod. This is the third routing layer the token has to cross: the registry entry so the manifest emits it, this map so the bundle carries it, and the GitHub prod secret so there is a value at all. Declaring a value and routing a value are separate acts, and only the last one is load-bearing. YAML re-parsed clean after the edit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X9mpK1ZFjexjthZbvcb94f
…ds it Codex, correctly: registering the manifest slot does not deliver the value. sync-secrets-local.yml builds its CGP bundle SOLELY from the workflow's explicit env map, and the v1 sync discards github_secret targets -- so a registered label with no row in that map funnels nothing on any node hydrated from the bundle. #2700 hit this exact trap and wrote the warning inline five lines above where this row now goes. I read that PR and still shipped the half that does not reach anyone. It matters more here than it did there, because the nodes hydrated from the bundle are precisely the ones this slot exists for: the 5090 has no Actions runner and hydrates via `secrets-funnel-from-prod`. Without this row the operator could create the Prod secret, the funnel would report success, and env.tier-data would still have no password -- an optional entry skipped in silence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz
…#2705) * feat(secrets): juicefs_meta's password had no route to any other node B850 mounts pmoves-media as the scoped `juicefs_meta` role. The role exists, the cutover happened, the mount is healthy. The credential, however, never entered the pipeline: /home/pmoves/.pmoves-secrets/juicefs_meta_pw -> bind -> /run/secrets/jfs_meta_pw A hand-placed file, under a different user's home, referenced NOWHERE in this repo and holding no CHIT registry entry. So B850 works and no second node can mount without someone hand-copying a secret -- the exact failure the funnel exists to prevent, and the blocker the 5090 is currently parked on. This is load-bearing now rather than tidy. PR #2702 made pg_hba admit ONLY juicefs_meta from the tailnet and reject every other role there, so a remote mount has no fallback credential: it authenticates as this role or not at all. Registered through the sanctioned path only -- REGISTRY entry, then chit-manifest-register, then chit-manifest-sync. No manifest was hand-edited. "JUICEFS_META_PASSWORD": {"tier": "data", "required": False} tier=data because juicefs-mount and juicefs-gateway both read env.tier-data. required=False unlike its neighbours: only nodes that actually mount pmoves-media need it, and under `--merge` (strict) a required slot fails the whole funnel on every node that legitimately lacks it. The usual argument against required=False -- that it buys silence -- does not apply here: an absent value fails loudly at mount time with an auth error. v2 also emits `docker_secret: pmoves_juicefs_meta_password`, which is the shape B850's hand-bound secret already has, so the pipeline can materialize what is currently placed by hand. Necessary but not sufficient, in #2700's sense: this routes the label. The value still has to be put into CHIT, and the compose defaults for juicefs-format/juicefs-gateway still name supabase_admin -- both tracked separately. Verified: chit-manifest-check reports in sync; chit-manifest-register --check reports complete with no constraint drift; re-running sync is byte-stable; both manifests changed by additions only (+23, 0 deletions). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz * fix(secrets): the route terminated one hop short of the node that needs it Codex, correctly: registering the manifest slot does not deliver the value. sync-secrets-local.yml builds its CGP bundle SOLELY from the workflow's explicit env map, and the v1 sync discards github_secret targets -- so a registered label with no row in that map funnels nothing on any node hydrated from the bundle. #2700 hit this exact trap and wrote the warning inline five lines above where this row now goes. I read that PR and still shipped the half that does not reach anyone. It matters more here than it did there, because the nodes hydrated from the bundle are precisely the ones this slot exists for: the 5090 has no Actions runner and hydrates via `secrets-funnel-from-prod`. Without this row the operator could create the Prod secret, the funnel would report success, and env.tier-data would still have no password -- an optional entry skipped in silence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
#2718) * fix(ci): re-enable the four Claude review workflows on the coding plan All four were down, and two of them invisibly. claude-code-review.yml if: false + empty ANTHROPIC_API_KEY review-comment-monitor.yml if: false + empty ANTHROPIC_API_KEY review-autofix.yml active + empty ANTHROPIC_API_KEY claude.yml active + expired OAuth token The failure chain is the same one #2700 fixed at the fleet level, replayed in CI: the OAuth token expired, so the workflows were repointed at ANTHROPIC_API_KEY -- which was also empty -- so two were switched off with `if: false` to stop them producing failing checks. Each step was locally reasonable. The result was a repo that appears to have Claude review configured and has none, with no failing check to say so, because a disabled job does not appear in the checks list at all. CLAUDE_CODE_OAUTH_TOKEN was re-minted 2026-08-24, so the premise recorded in those comments -- "expired and won't be rotated" -- no longer holds. Changes: - restore the original bot-filter guard in review-comment-monitor.yml, which was preserved in comments rather than deleted (that foresight is why this is a small change) - drop `if: false` from claude-code-review.yml - point all four at claude_code_oauth_token, replacing anthropic_api_key On the credential choice: the old comment argued the API key was "the persistent path" because the subscription token expires. True, and it still expires -- but ANTHROPIC_API_KEY was empty, so persistent meant persistently unauthenticated. An expiring credential that works beats a permanent one that is not set. The coding plan also means these runs draw plan quota rather than per-token spend. This does not displace the fleet review model. Codex bot, CodeRabbit and the CLI agents on self-hosted runners keep doing what they do; this restores the pre-merge lane they cannot cover -- a diff-scoped review on PR open/update, and triage of what the other bots produce. Verified: all four parse, no job is disabled, and no anthropic_api_key reference remains in any of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X9mpK1ZFjexjthZbvcb94f * fix(ci): gate the pull_request reviewer to same-repo; leave triage ungated Review raised two fork-secret P1s. One is right and is fixed; the other rests on a premise that does not hold, and acting on it would have removed working behaviour. claude-code-review.yml fires on `pull_request`. GitHub withholds Actions secrets from fork-originated runs of that event, so CLAUDE_CODE_OAUTH_TOKEN would be empty and the action would fail rather than review -- recreating the exact failing-check noise `if: false` was added to stop. Added the same-repo guard. review-comment-monitor.yml fires on `pull_request_review`, which runs in the BASE repository context and DOES receive secrets. The fork restriction is specific to `pull_request`. Left ungated, with the reasoning recorded in the file so the next reader does not re-derive it. The evidence is in this repo. review-autofix.yml shares the pull_request_review trigger, uses the same secret, and carries a same-repo guard -- but its own comment says why, and it is not about credentials: it holds contents:write and pushes to the PR head branch, so it needs head==base to avoid pushing to the wrong repo. If secrets were unavailable on fork pull_request_review runs, that comment would have said so instead. The triage job holds only pull-requests:write and issues:write. It posts one comment and never pushes, so it works on fork PRs today. Gating it would have turned a working lane off in the name of hardening, which is the failure mode worth avoiding here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X9mpK1ZFjexjthZbvcb94f --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The bug
archonboots healthy, passes its health check, and dies on the first workflow:The cause is not a missing value. It is a missing slot.
env.tier-agent.example:79-83CLAUDE_CODE_OAUTH_TOKEN(since #2359)docker-compose.yml/.agents.yml${CLAUDE_CODE_OAUTH_TOKEN:-}into archonenv.tier-agentMeanwhile
ANTHROPIC_API_KEYis a registered slot, is set fleet-wide fromenv.shared, and the Claude CLI prefers it over the OAuth token. Archon says so in its own source (packages/providers/src/claude/provider.ts:144):So the metered key wins and the coding plan never gets a chance.
MODEL_FABRIC_CONTRACT.md:44already listsClaude Code Maxin the approved coding-plan inventory — this is a missing slot, not a new policy decision.The change
Registered via the sanctioned path only: registry entry, then
chit-manifest-register, thenchit-manifest-sync. No manifest was hand-edited.Necessary but not sufficient — read this before expecting archon to work
This routes the label. It does not conjure a value.
secrets_sync.pyskips an entry entirely when no source value exists:So until a token is provisioned into the CGP source, the key is emitted not at all — not emitted empty. That is the same blank-is-not-absent rule #2688 established: an empty secret is not a secret, so absent stays absent.
archon therefore behaves exactly as it does today until the value lands. No regression, and no fix yet either.
Sequencing after merge: mint the token (
claude setup-token, in a real terminal), thenexport PMOVES_ROTATE_VALUE=...+make -C pmoves secrets-rotate KEY=CLAUDE_CODE_OAUTH_TOKEN(the env-var form keeps a bearer credential out of shell history andps), then restart archon and re-run a workflow. Only after that is confirmed shouldCLAUDE_USE_GLOBAL_AUTHflip tofalse— flipping it first trips archon'sexit(1)boot guard atindex.ts:271-292.Second fix, unavoidable and deliberate
The same sync emits
min_length: 64ontosecret_key_basein the v1 manifest.PR #2688 added that constraint to the registry, to v2, and to the sync logic — but never re-ran the sync. And
secrets_sync.py:336reads v1.So the min-length gate shipped in #2688 has been inert on main: the constraint lived in v2 and in the tooling while the file the enforcer actually reads never carried it. A gate that could not fire.
It cannot be excluded here without hand-editing a machine-emitted file, which the pipeline forbids. So it ships with this change, explicitly rather than silently.
Gates
Negative-tested, not trusted: injecting an unregistered label makes the check fail with
Error 1andwould add: ZZ_NEGATIVE_TEST_KEY -> tier llm. The label was removed and the file restored byte-exact before commit.Noted while verifying, not fixed here
The funnel audit already reports this whole bug class as a non-blocking WARN across seven tier files — for example:
AGENT_ZERO_MESSAGE_PATHin that list is the same var whose absence makes the supervisor proxyPOST /tasksto/api_message, a route the runtime does not serve (it registers only/api/<path:path>, so the real route is/api/api_message). Separate lane; recorded here so it is not rediscovered.🤖 Generated with Claude Code
https://claude.ai/code/session_01X9mpK1ZFjexjthZbvcb94f