Skip to content

fix(registry): auto-recover failed/provisioning workspaces on successful heartbeat - #1734

Merged
HongmingWang-Rabbit merged 1 commit into
mainfrom
fix/registry-heartbeat-autorecover
Apr 23, 2026
Merged

HongmingWang-Rabbit merged 1 commit into
mainfrom
fix/registry-heartbeat-autorecover

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

Summary

Extracted from closed PR #1664. When a workspace is stuck in failed or provisioning state but is actively sending successful heartbeats, the heartbeat handler now transitions it to online. Transient boot failures or a provisioner crash mid-setup previously left workspaces marked failed even after they became healthy, requiring manual intervention.

Problem

  • A flaky boot or a provisioner that crashes mid-setup leaves workspaces.status = 'failed' (or stale 'provisioning') in the DB.
  • The workspace subsequently comes up and begins sending heartbeats, but evaluateStatus only handled the online -> degraded, degraded -> online, and offline -> online transitions.
  • Result: the scheduler and dashboard keep reporting the workspace as dead despite it being healthy. Users have to manually reset status or delete-and-recreate.

Fix

In workspace-server/internal/handlers/registry.go::evaluateStatus, after the existing offline recovery block, add a new branch:

if currentStatus == "failed" || currentStatus == "provisioning" {
    // UPDATE workspaces SET status = 'online' ... WHERE id = $1 AND status IN ('failed', 'provisioning')
    // broadcast WORKSPACE_ONLINE with recovered_from=<prev status>
}
  • Same WHERE status IN (...) guarded UPDATE pattern as the existing offline-recovery block, so a concurrent delete that flips to removed cannot race the recovery back to online.
  • Existing online, degraded, and offline transitions are untouched.
  • Broadcasts WORKSPACE_ONLINE with a recovered_from payload field so the UI/event log can distinguish auto-recovery from first-time online.

Test plan

  • go build ./... in workspace-server/ passes cleanly.
  • go test ./internal/handlers/... -count=1 -run "TestRegistry|TestHeartbeat|TestRegister" — the only failures (TestRegister_C18_HijackBlockedNoBearer, TestRegister_ProvisionerURLPreserved) reproduce on origin/main without this patch, confirmed pre-existing and unrelated.
  • Manual: force status = 'failed' on a running workspace, confirm next heartbeat flips to 'online' and emits WORKSPACE_ONLINE with recovered_from: "failed".
  • Manual: force status = 'provisioning' on a running workspace, confirm same behaviour with recovered_from: "provisioning".
  • Verify removed workspaces are NOT revived (guarded by the status IN ('failed', 'provisioning') predicate in the UPDATE).

Scope: workspace-server/internal/handlers/registry.go only.

…ful heartbeat (extracted from #1664)

When a workspace is marked "failed" or "provisioning" but is actively
sending heartbeats, transition it to "online". Transient boot failures
or mid-setup provisioner crashes otherwise leave workspaces stuck in a
stale terminal state even after they become healthy.

Preserves existing online/degraded/offline transitions; only adds a new
conditional branch for the failed/provisioning case with a guarded
WHERE clause so a concurrent delete cannot flip 'removed' back to
'online'.
@HongmingWang-Rabbit
HongmingWang-Rabbit merged commit d5ec0a9 into main Apr 23, 2026
9 of 11 checks passed
@HongmingWang-Rabbit
HongmingWang-Rabbit deleted the fix/registry-heartbeat-autorecover branch April 24, 2026 00:11
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…orPanel (#1749)

CTO-bypass merge per 2026-05-24 directive — SOP-6 checklist acked, 2 non-author approvals on current HEAD, dispatched-review evidence in PR comments.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant