w738, w739: each person's Claude token pool; orchestrators and the ops worker on it; the dispatcher's reserve

[bryding]

Request: w738
Request: w739

One PR for both (they share the pick, the account setting and the tests; lothsahn agreed).

TL;DR: each person's runs use only that person's own pool of vault Claude tokens (several per person), picked by weekly reset, with the 80 / 95 / 99 limits. Their orchestrator (claudeAccounts.orchestrator = "vault") and the ops worker (per job and per caller) run on it too. The dispatcher keeps the shared token file and a reserve (5% of the week per day left, 20% of the 5-hour window). Tokens without user:profile are read from the rate-limit headers, as FFBox does. Needs a portal deploy and the new vault.sh on the FFBox host. No live config change.

What it does (the rules are in server/tokenPool.ts, pure functions; docs/vault.md section 4 has them in full)

• Whose tokens: the run's person's Claude entries only. A Claude entry with no owner, or another person's, is never picked (a shared Claude token is retired from the pick; fffctl vault list names it "not in any pool"). Work nobody asked for uses the system payer's pool.
• Order: the token whose weekly window resets soonest, then the 5-hour reset, then the name. Why weekly: the 5-hour window refills on its own every 5 hours, so ranking by it saves nothing; weekly capacity is lost at the reset (reasoning, a guess in the sense of no data).
• Limits (config vault.pool.*, one system-wide set, 0-100, one-at-a-time not above retired, any person's orchestrator may set them, logged with who, read at each process start): 5-hour >= 80 no new process; weekly >= 95 one process at a time; weekly >= 99 retired; 100 used up. No numbers counts as 50%. A session keeps its token only while it passes.
• Capped pool: workers, standing agents and the ops worker are held with the reason and the first reset (a standing run ends "Could not start: ..." and the next one tries again), with no fallback to anyone's token, the dispatcher's or the host token. A person's orchestrator uses the pool up (most room, soonest reset on a tie, one-at-a-time does not bind it) and stops only when every token is used up, with the reset in the message; there is no override.
• Transition: a person with no vault token keeps today's account. So do runs where the vault cannot open the tokens (no key): nothing to wait for.
• Orchestrators: poolRunEnv gives the one token with every other Claude credential removed (as the token file does), read at each session start; an owner's setting; the dispatcher refuses "vault" and keeps the token file (an inherited "vault" resolves to the token file).
• Ops worker: its options pick the caller's pool at each process start; every new job (and a deploy) and every change of caller starts a fresh conversation and process; a follow-up keeps both. A turn still running for the same person is not cut off.
• Dispatcher's reserve: resolved from config at each look (whatever claudeAccounts.dispatcher is, today the token file) plus the host token; 5% of the weekly window per day left to its reset and 20% of the 5-hour window; the dispatcher may always run into it and nothing checks it before it starts (tested). Others on that credential (a person with no vault token) are not refused, since they have nothing else; system_status shows each reserve and the dashboard warns while runs use the buffer. Until the first reading it counts as inside its reserve. One token is one account (no matching of two tokens).
• Numbers: fetchTokenLimits (rate-limit headers) now also runs on HTTP 403, remembered for 6 h; tokens near a limit (5-hour >= 60, weekly >= 90, or no numbers) are polled alone every 2 minutes; the SDK's rate_limit_event raises a meter between polls. FFBox's method: scripts/claude_keys.py in Final-Factory/ffbox, last changed in 57c35a5b4b (master edfddd5160): the usage endpoints 403 for a setup-token ("measured 2026-09-04"), so one Haiku call with max_tokens 1 is made and anthropic-ratelimit-unified-5h-/7d-utilization and -reset are read, also off a 429 reply.
• Visibility: system_status has a line per person's pool (each token's last four characters, state, meters, resets, agents running on it) and per reserved credential; fffctl vault list ends with the pools; a dashboard banner for a pool used up or over its caps and for the dispatcher's buffer in use.
• Host: claude-tokens/<name> files per person; vault-sync loads them as host-<user>-claude-<name>.

Evidence

• Measured here: server/tokenPool.test.ts (9), server/vaultPool.test.ts (18), additions to usage.test.ts (3), opsWorker.test.ts (1), updated vault.test.ts; the full unit suite passes except 4 tests that fail only in this sandbox's temp folder (EPERM on rmSync of ff-msb-*; they are unrelated and run in CI). tsc clean for the server and the web.
• Not measured: the rate-limit headers with an inference-only token (no token touched; the first poll after deploy logs via the API's rate-limit headers), the SDK event's scale and frequency, and the overshoot between polls (no poll history existed; every poll logs its percentages, so a week of journal gives it).
• The e2e step reads fffctl vault list with a second pool token and a refused file name in the nested VM.

🤖 Generated with Claude Code