Skip to content

fix(secrets): auto-refresh global_secrets on workspace restart (#15) - #64

Merged
HongmingWang-Rabbit merged 1 commit into
mainfrom
fix/issue-15-refresh-oauth-on-restart
Apr 14, 2026
Merged

fix(secrets): auto-refresh global_secrets on workspace restart (#15)#64
HongmingWang-Rabbit merged 1 commit into
mainfrom
fix/issue-15-refresh-oauth-on-restart

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

Resolves #15.

Problem

Provisioner reads CLAUDE_CODE_OAUTH_TOKEN (and all env vars derived from
global_secrets) once at ContainerCreate. When the Claude Max OAuth token
rotates, existing workspaces keep the stale token and the Claude CLI dies
with 401 Invalid authentication credentials. Remediation was a manual
POST /admin/secrets + per-workspace restart loop (incident 2026-04-13:
~15 min recovery, 12 workspaces).

Fix

Restart-time re-read of secrets was already correct — provisionWorkspaceOpts
re-queries global_secrets + workspace_secrets on every call, and
provisioner.Start() always force-removes + recreates the container. The
missing piece was the trigger.

SetGlobal / DeleteGlobal now enqueue RestartByID for every non-paused,
non-removed, non-external workspace that does NOT shadow the key with a
workspace-level override. Mirrors the behaviour of workspace-scoped
Set / Delete (which already auto-restart).

Effectively implements the "bonus" refresh-all step from the issue, without
a new endpoint: updating the global secret via the existing endpoint
propagates automatically.

Files

  • platform/internal/handlers/secrets.gorestartAllAffectedByGlobalKey helper + wiring in SetGlobal / DeleteGlobal
  • platform/internal/handlers/secrets_test.go — two sqlmock tests covering set and delete paths, verifying NOT IN (workspace_secrets WHERE key=...) filter

Test plan

  • go build ./... && go vet ./... clean
  • go test -race ./... pass (platform)
  • New tests TestSetGlobal_AutoRestartsAffectedWorkspaces and TestDeleteGlobal_AutoRestartsAffectedWorkspaces assert restart callback fires for affected workspaces
  • E2E verification pending live platform: set global secret → confirm existing workspaces restart and pick up new value

Generated with Claude Code

Global secrets (e.g. CLAUDE_CODE_OAUTH_TOKEN) are injected as container env
vars at Start() time. Until now, rotating one only propagated to a workspace
on the next full restart-from-zero, which manual ops had to drive via a
`POST /workspaces/:id/restart` loop. Tier-3 Claude Code agents hit the
stale-token path first and surfaced as 401s inside the SDK.

Restart-time re-read of global_secrets + workspace_secrets was already
correct in `provisionWorkspaceOpts` — the missing piece was the trigger.
SetGlobal / DeleteGlobal now enqueue RestartByID for every non-paused,
non-removed, non-external workspace that does NOT shadow the key with a
workspace-level override. Matches the existing behaviour of workspace-scoped
`Set` / `Delete`.

Adds two sqlmock-backed tests exercising both branches.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@HongmingWang-Rabbit
HongmingWang-Rabbit marked this pull request as ready for review April 14, 2026 19:39
@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor Author

noteworthy: secrets-handling — merging after 7-gate verification.

7 gates clean. Code-review: 0 🔴 / 1 🟡 / 4 🔵.

The 🟡 — unbounded goroutine fan-out on global-secret rotation — is a real scaling concern (1000 workspaces × 1 fan = correlated restart storm) but doesn't trigger at current scale (<50 workspaces). Tracked as a follow-up rather than blocker since:

  1. The base-case fix (rotated secrets actually propagate) is the user-facing win
  2. Adding a worker-pool gate would expand scope beyond the issue
  3. Throttling is a 1-line fix when needed

Cross-vendor review attempted but tooling unavailable in cron context.

Other 🔵 nits worth a follow-up cleanup PR: explicit override-shadow SQL assertion, ctx timeout on the affected-workspaces query, log warning on nil restartFunc.

@HongmingWang-Rabbit
HongmingWang-Rabbit merged commit 383582f into main Apr 14, 2026
7 checks passed
@HongmingWang-Rabbit
HongmingWang-Rabbit deleted the fix/issue-15-refresh-oauth-on-restart branch April 14, 2026 19:49
HongmingWang-Rabbit pushed a commit that referenced this pull request Apr 14, 2026
#65)

- edit-history/2026-04-14.md: append tick-4 section covering the 12
  modular guardrail plugins (#63), global-secrets auto-restart fan-out
  (#64, fixes issue #15), and synthetic restart-context A2A message
  (#65, fixes issue #19 Layer 1; Layer 2 deferred to issue #66).
- CLAUDE.md: bump Go test count 699 -> 726 (measured); note global
  secrets auto-restart on SetGlobal/DeleteGlobal in the route table;
  add Workspace Lifecycle paragraph for the restart-context message
  and its system:restart-context caller prefix.
- PLAN.md: bump Go test count in the coverage table; record issues
  #15 and #19 Layer 1 as launched; add new Backlog entry for the
  Layer 2 follow-up (issue #66).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit added a commit that referenced this pull request Apr 14, 2026
docs: sync documentation with 2026-04-14 evening-tick merges (#63, #64, #65)
HongmingWang-Rabbit pushed a commit that referenced this pull request Apr 14, 2026
Backlog items 11-14 used sequential enumeration (#64/#65/#66/#67) as
intra-doc bookkeeping. Those numbers now collide with actual merged
PRs and open issues with completely different scopes:
  - PR #64 = auto-refresh global_secrets (not "delegations list")
  - PR #65 = restart context Layer 1 (not "per-agent repo access")
  - Issue #66 = restart_prompt Layer 2 (not "SDK swallows stderr")
  - PR #67 = docs sync tick-4 (not "MCP localhost default")

Strip the misleading refs and add a footnote explaining the cleanup.
If/when any of these items get prioritized, file real GitHub issues.

Tracked in cron-learnings tick-3 entry.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
…-restart

fix(secrets): auto-refresh global_secrets on workspace restart (#15)
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
#65)

- edit-history/2026-04-14.md: append tick-4 section covering the 12
  modular guardrail plugins (#63), global-secrets auto-restart fan-out
  (#64, fixes issue #15), and synthetic restart-context A2A message
  (#65, fixes issue #19 Layer 1; Layer 2 deferred to issue #66).
- CLAUDE.md: bump Go test count 699 -> 726 (measured); note global
  secrets auto-restart on SetGlobal/DeleteGlobal in the route table;
  add Workspace Lifecycle paragraph for the restart-context message
  and its system:restart-context caller prefix.
- PLAN.md: bump Go test count in the coverage table; record issues
  #15 and #19 Layer 1 as launched; add new Backlog entry for the
  Layer 2 follow-up (issue #66).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
docs: sync documentation with 2026-04-14 evening-tick merges (#63, #64, #65)
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
Backlog items 11-14 used sequential enumeration (#64/#65/#66/#67) as
intra-doc bookkeeping. Those numbers now collide with actual merged
PRs and open issues with completely different scopes:
  - PR #64 = auto-refresh global_secrets (not "delegations list")
  - PR #65 = restart context Layer 1 (not "per-agent repo access")
  - Issue #66 = restart_prompt Layer 2 (not "SDK swallows stderr")
  - PR #67 = docs sync tick-4 (not "MCP localhost default")

Strip the misleading refs and add a footnote explaining the cleanup.
If/when any of these items get prioritized, file real GitHub issues.

Tracked in cron-learnings tick-3 entry.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provisioner: auto-refresh CLAUDE_CODE_OAUTH_TOKEN from global_secrets on workspace restart

1 participant