fix(secrets): auto-refresh global_secrets on workspace restart (#15) - #64
Merged
Merged
Conversation
Global secrets (e.g. CLAUDE_CODE_OAUTH_TOKEN) are injected as container env vars at Start() time. Until now, rotating one only propagated to a workspace on the next full restart-from-zero, which manual ops had to drive via a `POST /workspaces/:id/restart` loop. Tier-3 Claude Code agents hit the stale-token path first and surfaced as 401s inside the SDK. Restart-time re-read of global_secrets + workspace_secrets was already correct in `provisionWorkspaceOpts` — the missing piece was the trigger. SetGlobal / DeleteGlobal now enqueue RestartByID for every non-paused, non-removed, non-external workspace that does NOT shadow the key with a workspace-level override. Matches the existing behaviour of workspace-scoped `Set` / `Delete`. Adds two sqlmock-backed tests exercising both branches. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit
marked this pull request as ready for review
April 14, 2026 19:39
Contributor
Author
|
noteworthy: secrets-handling — merging after 7-gate verification. 7 gates clean. Code-review: 0 🔴 / 1 🟡 / 4 🔵. The 🟡 — unbounded goroutine fan-out on global-secret rotation — is a real scaling concern (1000 workspaces × 1 fan = correlated restart storm) but doesn't trigger at current scale (<50 workspaces). Tracked as a follow-up rather than blocker since:
Cross-vendor review attempted but tooling unavailable in cron context. Other 🔵 nits worth a follow-up cleanup PR: explicit override-shadow SQL assertion, ctx timeout on the affected-workspaces query, log warning on nil restartFunc. |
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Apr 14, 2026
#65) - edit-history/2026-04-14.md: append tick-4 section covering the 12 modular guardrail plugins (#63), global-secrets auto-restart fan-out (#64, fixes issue #15), and synthetic restart-context A2A message (#65, fixes issue #19 Layer 1; Layer 2 deferred to issue #66). - CLAUDE.md: bump Go test count 699 -> 726 (measured); note global secrets auto-restart on SetGlobal/DeleteGlobal in the route table; add Workspace Lifecycle paragraph for the restart-context message and its system:restart-context caller prefix. - PLAN.md: bump Go test count in the coverage table; record issues #15 and #19 Layer 1 as launched; add new Backlog entry for the Layer 2 follow-up (issue #66). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4 tasks
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Apr 14, 2026
Backlog items 11-14 used sequential enumeration (#64/#65/#66/#67) as intra-doc bookkeeping. Those numbers now collide with actual merged PRs and open issues with completely different scopes: - PR #64 = auto-refresh global_secrets (not "delegations list") - PR #65 = restart context Layer 1 (not "per-agent repo access") - Issue #66 = restart_prompt Layer 2 (not "SDK swallows stderr") - PR #67 = docs sync tick-4 (not "MCP localhost default") Strip the misleading refs and add a footnote explaining the cleanup. If/when any of these items get prioritized, file real GitHub issues. Tracked in cron-learnings tick-3 entry. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4 tasks
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
…-restart fix(secrets): auto-refresh global_secrets on workspace restart (#15)
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
#65) - edit-history/2026-04-14.md: append tick-4 section covering the 12 modular guardrail plugins (#63), global-secrets auto-restart fan-out (#64, fixes issue #15), and synthetic restart-context A2A message (#65, fixes issue #19 Layer 1; Layer 2 deferred to issue #66). - CLAUDE.md: bump Go test count 699 -> 726 (measured); note global secrets auto-restart on SetGlobal/DeleteGlobal in the route table; add Workspace Lifecycle paragraph for the restart-context message and its system:restart-context caller prefix. - PLAN.md: bump Go test count in the coverage table; record issues #15 and #19 Layer 1 as launched; add new Backlog entry for the Layer 2 follow-up (issue #66). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
Backlog items 11-14 used sequential enumeration (#64/#65/#66/#67) as intra-doc bookkeeping. Those numbers now collide with actual merged PRs and open issues with completely different scopes: - PR #64 = auto-refresh global_secrets (not "delegations list") - PR #65 = restart context Layer 1 (not "per-agent repo access") - Issue #66 = restart_prompt Layer 2 (not "SDK swallows stderr") - PR #67 = docs sync tick-4 (not "MCP localhost default") Strip the misleading refs and add a footnote explaining the cleanup. If/when any of these items get prioritized, file real GitHub issues. Tracked in cron-learnings tick-3 entry. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #15.
Problem
Provisioner reads
CLAUDE_CODE_OAUTH_TOKEN(and all env vars derived fromglobal_secrets) once atContainerCreate. When the Claude Max OAuth tokenrotates, existing workspaces keep the stale token and the Claude CLI dies
with
401 Invalid authentication credentials. Remediation was a manualPOST /admin/secrets+ per-workspace restart loop (incident 2026-04-13:~15 min recovery, 12 workspaces).
Fix
Restart-time re-read of secrets was already correct —
provisionWorkspaceOptsre-queries
global_secrets+workspace_secretson every call, andprovisioner.Start()always force-removes + recreates the container. Themissing piece was the trigger.
SetGlobal/DeleteGlobalnow enqueueRestartByIDfor every non-paused,non-removed, non-external workspace that does NOT shadow the key with a
workspace-level override. Mirrors the behaviour of workspace-scoped
Set/Delete(which already auto-restart).Effectively implements the "bonus"
refresh-allstep from the issue, withouta new endpoint: updating the global secret via the existing endpoint
propagates automatically.
Files
platform/internal/handlers/secrets.go—restartAllAffectedByGlobalKeyhelper + wiring inSetGlobal/DeleteGlobalplatform/internal/handlers/secrets_test.go— two sqlmock tests covering set and delete paths, verifyingNOT IN (workspace_secrets WHERE key=...)filterTest plan
go build ./... && go vet ./...cleango test -race ./...pass (platform)TestSetGlobal_AutoRestartsAffectedWorkspacesandTestDeleteGlobal_AutoRestartsAffectedWorkspacesassert restart callback fires for affected workspacesGenerated with Claude Code