fix(registry): auto-recover failed/provisioning workspaces on successful heartbeat - #1734
Merged
Merged
Conversation
…ful heartbeat (extracted from #1664) When a workspace is marked "failed" or "provisioning" but is actively sending heartbeats, transition it to "online". Transient boot failures or mid-setup provisioner crashes otherwise leave workspaces stuck in a stale terminal state even after they become healthy. Preserves existing online/degraded/offline transitions; only adds a new conditional branch for the failed/provisioning case with a guarded WHERE clause so a concurrent delete cannot flip 'removed' back to 'online'.
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
…orPanel (#1749) CTO-bypass merge per 2026-05-24 directive — SOP-6 checklist acked, 2 non-author approvals on current HEAD, dispatched-review evidence in PR comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extracted from closed PR #1664. When a workspace is stuck in
failedorprovisioningstate but is actively sending successful heartbeats, the heartbeat handler now transitions it toonline. Transient boot failures or a provisioner crash mid-setup previously left workspaces markedfailedeven after they became healthy, requiring manual intervention.Problem
workspaces.status = 'failed'(or stale'provisioning') in the DB.evaluateStatusonly handled theonline -> degraded,degraded -> online, andoffline -> onlinetransitions.Fix
In
workspace-server/internal/handlers/registry.go::evaluateStatus, after the existingofflinerecovery block, add a new branch:WHERE status IN (...)guarded UPDATE pattern as the existing offline-recovery block, so a concurrent delete that flips toremovedcannot race the recovery back toonline.online,degraded, andofflinetransitions are untouched.WORKSPACE_ONLINEwith arecovered_frompayload field so the UI/event log can distinguish auto-recovery from first-time online.Test plan
go build ./...inworkspace-server/passes cleanly.go test ./internal/handlers/... -count=1 -run "TestRegistry|TestHeartbeat|TestRegister"— the only failures (TestRegister_C18_HijackBlockedNoBearer,TestRegister_ProvisionerURLPreserved) reproduce onorigin/mainwithout this patch, confirmed pre-existing and unrelated.status = 'failed'on a running workspace, confirm next heartbeat flips to'online'and emitsWORKSPACE_ONLINEwithrecovered_from: "failed".status = 'provisioning'on a running workspace, confirm same behaviour withrecovered_from: "provisioning".removedworkspaces are NOT revived (guarded by thestatus IN ('failed', 'provisioning')predicate in the UPDATE).Scope:
workspace-server/internal/handlers/registry.goonly.