w738, w739: each person's Claude token pool; orchestrators and the ops worker on it; the dispatcher's reserve

CHANGELOG.md

@@ -10,6 +10,27 @@ Add your change under **[Unreleased]** in the same pull request. `npm run releas
 
 ## [Unreleased]
 
+- **Each person's runs use that person's own pool of Claude tokens, and their orchestrator and the ops worker run on it too** (w738,
+  w739, Lothsahn). A person can have several tokens in the vault (host files `people/<id>/claude-token` and
+  `claude-tokens/<name>`, loaded by `sudo fff-vm vault-sync`). The pick (`server/tokenPool.ts`): that person's tokens only (no shared
+  Claude token, no one else's), the one whose weekly window resets soonest first, the 5-hour reset as the tie-break; no new process
+  on a token at 80% of its 5-hour window, one process at a time at 95% weekly, retired at 99%, used up at 100; a session keeps its
+  token only while it passes. A worker, a standing agent and the ops worker are held with the reason and the next reset when the
+  whole pool is over its caps (no fallback to anyone else's token, the dispatcher's or the host token); a person's orchestrator
+  uses the pool up and then stops with a message. A person with no vault token keeps today's account. New:
+  `claudeAccounts.orchestrator = "vault"` (an owner's setting; the dispatcher stays on the token file; each orchestrator reads its
+  person's pool at every session start with every other credential removed, redacted everywhere); the ops worker starts a
+  fresh conversation and process for every new job and every change of caller, on that caller's pool; the pool limits are config
+  `vault.pool.*` (80 / 95 / 99, 5% a day, 20%), one set that anyone can change with `set_app_config` (checked 0-100, one at a time
+  not above retired, logged with who), read at each process start. The dispatcher's token (whatever `claudeAccounts.dispatcher`
+  names) and the host token keep a reserve of 5% of the weekly window per day left until its reset and 20% of the 5-hour window:
+  the dispatcher may always run into it; `system_status` and a dashboard warning show it. Usage numbers: a token without the
+  `user:profile` scope (every `claude setup-token` token, the dispatcher's too: HTTP 403) is read from the API's rate-limit
+  headers, as FFBox does (`scripts/claude_keys.py`); tokens near a limit are polled every 2 minutes; the SDK's rate-limit events
+  raise a meter between polls. `fffctl vault list` shows each pool; `system_status` shows each pool, its agents and the reserves; a
+  banner shows a pool used up or over its caps. Needs a portal deploy and, on the FFBox host, the new `vault.sh` (`install.sh
+  --guest-only` or `fff-vm` update).
+
 - **The portal's token vault opens its key in the VM** (w736, Lothsahn; `system_status` said "Token vault: 0 entries; key
   unreadable (the vault key /run/credentials/fff-portal.service/fff-vault-key is readable by other users (mode 440); chmod 600
   it)"). Cause: for a service that is not root, systemd writes a credential 0400 root-owned with an ACL entry for the service

README.md

@@ -167,7 +167,7 @@ which is set per role and per machine ([docs/accounts.md](docs/accounts.md)): co
 puts the orchestrators, the dispatcher or this host's own daemon's workers on this host's login instead of the token, and
 `machines.useHostClaudeEnv` puts a Mac's agents on that Mac's login. The token vault
 ([docs/vault.md](docs/vault.md)) holds Claude tokens and the workers' other secrets on the portal and hands each
-machine run what it may have, picking the Claude token by plan headroom. A login is shown by
+machine run what it may have, picking the Claude token from the person's own pool by plan meters. A login is shown by
 its email and one login signed in on several computers is one account; a token by its last 4
 characters, never its value. Each account lists where it is used and which agents run on it, and an
 agent's details name its account when there is more than one. The collapsed footer shows the

deploy/vm/host/vault.sh

@@ -5,11 +5,16 @@
 # lothsahn, 2026-10-06: each person's Claude and GitHub tokens are kept on this host, root only, next to what the
 # installer already keeps (decision D2: root here can read the VM anyway). The VM's vault gets them from here:
 #   /etc/fff-vm/secrets/people/<user id>/claude-token    one 'claude setup-token' token (sk-ant-oat01-...)
+#   /etc/fff-vm/secrets/people/<user id>/claude-tokens/<name>
+#                                                        more of them, one token per file (w739: a person's pool; the
+#                                                        <name> is 1-16 of a-z 0-9 . _ - and starts with a letter or digit)
 #   /etc/fff-vm/secrets/people/<user id>/github-token    one fine-grained GitHub token (github_pat_...)
 #   /etc/fff-vm/secrets/vault.key                        the spare copy of the VM's vault key (kept in step with it)
 # Each value goes into the VM only over ssh's stdin, to `fffctl vault put`, and is never on a command line, in a log or in
 # the VM's daily backup (fff-backup leaves data/vault.json out; this host is where a rebuilt VM gets them back from).
-# The vault entries it makes are named host-<user>-claude and host-<user>-github; one whose file is gone is removed.
+# The vault entries it makes are named host-<user>-claude (claude-token), host-<user>-claude-<name> (each file of
+# claude-tokens/) and host-<user>-github; one whose file is gone is removed. All of a person's Claude entries are their pool:
+# the portal gives their runs those tokens only, soonest weekly reset first (docs/vault.md, section 4).
 
 VAULT_PEOPLE=$FFF_VM_ETC/secrets/people
 VAULT_KEY_COPY=$FFF_VM_ETC/secrets/vault.key
@@ -69,11 +74,27 @@ vault_sync() {
       person=$(basename "$dir")
       [[ "$person" =~ ^[A-Za-z0-9._-]{1,30}$ ]] || { warn "vault: $dir is not named for a portal user id; skipped"; rc=1; continue; }
       lower=$(tr '[:upper:]' '[:lower:]' <<<"$person")
-      for kind in claude github; do
-        file=$dir$kind-token
+      # The single files, then the pool's extra tokens (claude-tokens/<name>): "kind|file|vault entry name" per token.
+      local -a todo=()
+      for kind in claude github; do todo+=("$kind|$dir$kind-token|host-$lower-$kind"); done
+      if [ -d "${dir}claude-tokens" ]; then
+        chmod 0700 "${dir}claude-tokens"
+        for file in "${dir}claude-tokens"/*; do
+          [ -f "$file" ] || continue
+          local slot
+          slot=$(tr '[:upper:]' '[:lower:]' <<<"$(basename "$file")")
+          [[ "$slot" =~ ^[a-z0-9][a-z0-9._-]{0,15}$ ]] || { warn "vault: $file is not named 1-16 of a-z 0-9 . _ - (starting with a letter or digit); skipped"; rc=1; continue; }
+          todo+=("claude|$file|host-$lower-claude-$slot")
+        done
+      fi
+      local item
+      for item in "${todo[@]}"; do
+        kind=${item%%|*}
+        file=${item#*|}
+        name=${file#*|}
+        file=${file%%|*}
         [ -s "$file" ] || continue
         chmod 0600 "$file"
-        name=host-$lower-$kind
         if ! vault_value_ok "$kind" "$file"; then
           warn "vault: $file is not $([ "$kind" = claude ] && echo "one 'claude setup-token' token (sk-ant-oat01-...)" || echo 'one fine-grained GitHub token (github_pat_...)'); skipped (its content is not shown)"
           rc=1

deploy/vm/test/ci-vm-e2e.sh

@@ -271,17 +271,29 @@ echo "ok: the vault key is root's and loaded by the portal; values and credentia
 rnd() { local s; s=$(head -c 96 /dev/urandom | base64 -w0 | tr -dc 'A-Za-z0-9'); printf '%s' "${s:0:$1}"; }
 ct="sk-ant-oat01-$(rnd 44)"
 gt="github_pat_$(rnd 40)"
-install -d -m 0700 /etc/fff-vm/secrets/people/ci /etc/fff-vm/secrets/people/ci2
+ct2="sk-ant-oat01-$(rnd 44)"
+install -d -m 0700 /etc/fff-vm/secrets/people/ci /etc/fff-vm/secrets/people/ci2 /etc/fff-vm/secrets/people/ci/claude-tokens
 printf '%s\n' "$ct" | install -m 0600 /dev/stdin /etc/fff-vm/secrets/people/ci/claude-token
+# A second Claude token for the same person (w739: the pool), and a file the pool refuses by its name.
+printf '%s\n' "$ct2" | install -m 0600 /dev/stdin /etc/fff-vm/secrets/people/ci/claude-tokens/Second
+printf '%s\n' "sk-ant-oat01-$(rnd 44)" | install -m 0600 /dev/stdin "/etc/fff-vm/secrets/people/ci/claude-tokens/bad name!"
 printf '%s\n' "$gt" | install -m 0600 /dev/stdin /etc/fff-vm/secrets/people/ci/github-token
 printf 'ghp_%s\n' "$(rnd 36)" | install -m 0600 /dev/stdin /etc/fff-vm/secrets/people/ci2/github-token
 out=$(/usr/local/sbin/fff-vm vault-sync 2>&1) || true
 echo "$out"
-for v in "$ct" "$gt"; do if printf '%s' "$out" | matches -F "${v:13:24}"; then fail "fff-vm vault-sync printed a token"; fi; done
+for v in "$ct" "$gt" "$ct2"; do if printf '%s' "$out" | matches -F "${v:13:24}"; then fail "fff-vm vault-sync printed a token"; fi; done
+printf '%s' "$out" | matches 'not named 1-16 of a-z' || fail "a pool file with a bad name was not refused: $out"
 printf '%s' "$out" | matches 'not one fine-grained GitHub token' || fail "a classic GitHub token was not refused"
 list=$(g 'sudo fffctl vault list')
 printf '%s' "$list" | matches -F "${ct: -4}" || fail "host-ci-claude is not in the vault: $list"
 printf '%s' "$list" | matches '^host-ci-github ' || fail "host-ci-github is not in the vault: $list"
+# The pool: both Claude tokens are the person's (owner ci), and the CLI lists the pool with their state and last four characters.
+printf '%s' "$list" | matches '^host-ci-claude-second ' || fail "the second token of the pool is not in the vault: $list"
+printf '%s' "$list" | matches -F "${ct2: -4}" || fail "the second token's last four characters are not listed: $list"
+printf '%s' "$list" | matches 'Claude token pools' || fail "fffctl vault list has no pool section: $list"
+printf '%s' "$list" | matches -E '^    host-ci-claude-second +…' || fail "the pool section does not list the second token: $list"
+printf '%s' "$list" | matches -E 'ok +5-hour' || fail "the pool section shows no state and meters: $list"
+if printf '%s' "$list" | matches 'bad'; then fail "the refused pool file went in"; fi
 if printf '%s' "$list" | matches '^host-ci2-'; then fail "the refused token went in"; fi
 [ "$(stat -c '%a %U' /etc/fff-vm/secrets/vault.key)" = "600 root" ] || fail "the key's spare copy is not 0600 root"
 # The file is named to sha256sum under sudo: a "<" redirect would be opened by the admin's own shell, which cannot read it.
@@ -290,6 +302,7 @@ if printf '%s' "$list" | matches '^host-ci2-'; then fail "the refused token went
 out=$(/usr/local/sbin/fff-vm vault-sync 2>&1) || true
 printf '%s' "$out" | matches 'unchanged host-ci-claude' || fail "a second sync was not a no-op: $out"
 printf '%s' "$out" | matches 'unchanged host-ci-github' || fail "a second sync was not a no-op: $out"
+printf '%s' "$out" | matches 'unchanged host-ci-claude-second' || fail "a second sync was not a no-op for the pool's second token: $out"
 rm -rf /etc/fff-vm/secrets/people/ci /etc/fff-vm/secrets/people/ci2
 /usr/local/sbin/fff-vm vault-sync >/dev/null 2>&1 || true
 if g 'sudo fffctl vault list --names' | matches '^host-ci-'; then fail "a removed person's entries stayed in the vault"; fi

deploy/vm/test/fff-watchdog.test.sh

@@ -77,7 +77,7 @@ command -v node >/dev/null || fail "node is needed"
 node -e 'require("net").createServer().listen(process.argv[1]); setInterval(() => {}, 1e6)' "$tmp/claude.sock" &
 NODE_PID=$!
 trap 'kill $NODE_PID 2>/dev/null || true; rm -rf "$tmp"' EXIT
-for _ in $(seq 50); do [ -S "$tmp/claude.sock" ] && break; sleep 0.1; done
+for _ in $(seq 300); do [ -S "$tmp/claude.sock" ] && break; sleep 0.1; done
 [ -S "$tmp/claude.sock" ] || fail "node did not make the test socket"
 
 mkunit() { # UNIT ACTIVE UNITFILESTATE [RESULT] [SINCE]

docs/accounts.md

@@ -14,7 +14,7 @@ Every agent the portal starts runs on one of three kinds of Claude credential:
 
 | Agents | Config | Values | Default |
 |---|---|---|---|
-| The orchestrator | `claudeAccounts.orchestrator` | `"token"` or `"login"` (this host's) | `"token"` |
+| The orchestrator (and the ops worker working for a person) | `claudeAccounts.orchestrator` | `"token"`, `"login"` (this host's), `"tokenfile"` or `"vault"` (each person's own pool, an owner's setting; w738) | `"token"` |
 | The dispatcher (w464) | `claudeAccounts.dispatcher` | same; once set it also overrides the system payer's own token in `userClaudeEnv` | unset: the system payer's own token if any, else `claudeAccounts.orchestrator`'s |
 | Workers and standing agents on a Mac | `machines.useHostClaudeEnv` | `true` (host token) or `false` (the Mac's login); global, or per machine | `true` |
 | Workers and standing agents on this host's own daemon ([beast-machine.md](beast-machine.md)) | `claudeAccounts.workers`, unless `machines.useHostClaudeEnv` names the machine | `"token"` or `"login"`; "login" is this host's login, with the rest of `claudeEnv` (e.g. `CLAUDE_CONFIG_DIR`) kept | `"token"` |
@@ -68,12 +68,31 @@ the access token and the refresh token have both expired (`hostLoginProblem`, `s
 without the `user:profile` scope is accepted: agents need only `user:inference`, and only the usage meters
 need `user:profile`. On macOS a missing file proves nothing, because the login is in the Keychain.
 
-### The token vault (w512)
+### The token vault (w512) and the pools (w738, w739)
+
+`claudeAccounts.orchestrator` also takes `"vault"` (an owner's switch in `set_app_config`): a person's orchestrator
+starts with `CLAUDE_CODE_OAUTH_TOKEN` set to a token of **that person's pool** (their Claude entries in the vault,
+`people/<id>/claude-token` and `claude-tokens/<name>` on the FFBox host), read at each session start so a new token
+applies to the next session with no restart, and every other Claude credential removed first, as for the token file
+(`poolRunEnv`, `server/secrets.ts`). Which of their tokens, and what happens when they are over their caps or used up:
+[vault.md](vault.md) section 4. The orchestration worker works the same way, on the pool of the person whose orchestrator
+gave it the job, and starts a fresh conversation and process for every new job and every change of caller
+([ops-worker.md](ops-worker.md)). **The dispatcher never takes `"vault"`:** it keeps the shared token file (a dispatcher
+that inherits `"vault"` from the orchestrator's setting is on the token file, else the host token), and that
+credential and the host token keep a reserve for their own roles. A person with no vault token yet falls back to the
+token file, as before. `system_status` names the account per role ("each person's own vault pool (token file …dAAA for a
+person with no vault token)"), each person's pool with each token's state, and the reserves. The pool limits are config
+`vault.pool.*`, one set for everyone, which any person's orchestrator may change: `sessionHoldPercent` 80,
+`onePerWeeklyPercent` 95, `retireWeeklyPercent` 99, `reservePerDayPercent` 5, `reserveSessionPercent` 20.
+
+The meters: a session on a pool token shows on that token's account (a live session by the account its process started
+on; a stopped orchestrator by the last token it was handed).
+
+#### Machine runs
 
 Machine runs (workers and standing agents on a machine) can take their Claude token from the portal's token vault
 instead: config `machines.claudeFromVault` (`true`, `false`, or per machine with `"*"`; default `false`; owner-only in
-`set_app_config`, with `machine: "<id>"` for one machine alone) names the machines. Each run then gets one vault token, chosen by plan headroom with the run's
-person's own token first, as `CLAUDE_CODE_OAUTH_TOKEN` alone (`LaunchSpec.login` drops the daemon's own credentials). A
+`set_app_config`, with `machine: "<id>"` for one machine alone) names the machines. Each run then gets one vault token from its person's pool (docs/vault.md section 4), as `CLAUDE_CODE_OAUTH_TOKEN` alone (`LaunchSpec.login` drops the daemon's own credentials). A
 person's own token in `userClaudeEnv` still wins for their work. With no eligible vault token the run falls back to
 what this page describes. The vault, its other secrets and the cut-over: [vault.md](vault.md).
 

docs/machines.md

@@ -85,7 +85,7 @@ before redeploying by hand.
   deletes them. Force pushes and pushes to the game repo's master/main stay refused. The daemon's own folder
   (`~/.ff-factory`, or the machine's `app_dir`; it holds the token) is protected.
 - **Claude account.** With the token vault on for a machine (config `machines.claudeFromVault`), its runs get a vault
-  token chosen per run by plan headroom, and the other secrets granted to that machine, in the same launch spec
+  token from the run person's pool (soonest weekly reset first, within the pool's caps), and the other secrets granted to that machine, in the same launch spec
   ([vault.md](vault.md)). Otherwise: portal-run agents on a Mac (workers and standing agents) use this host's
   `claudeEnv`, so with `CLAUDE_CODE_OAUTH_TOKEN` set (`set_app_config`, write-only) they run on the same
   Claude account as the agents here, not on the Mac's own login. The portal puts it in the launch spec,