Skip to content

feat(scheduler): sweepPhantomBusy — clear stuck active_tasks from crashed runs - #1731

Merged
HongmingWang-Rabbit merged 1 commit into
mainfrom
fix/scheduler-sweep-phantom-busy
Apr 23, 2026
Merged

HongmingWang-Rabbit merged 1 commit into
mainfrom
fix/scheduler-sweep-phantom-busy

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

Summary

Extracts the sweepPhantomBusy 5-minute sweeper from closed PR #1664 to address monorepo issue #1372.

Root cause

When a workspace agent crashes mid-task (MiniMax timeout, OOM, panic in the finally block, SSRF kill, etc.), the active_tasks counter gets incremented on task start but never decremented. From then on the scheduler treats the workspace as perpetually busy and skips every cron fire with workspace busy — retry. The only fix was manual SQL against prod.

Fix

A periodic sweeper on the scheduler tick loop:

  • Runs at most every 5 minutes (phantomSweepInterval), gated via lastSweepAt on the Scheduler struct — cheap no-op on non-sweep ticks.
  • Finds workspaces where active_tasks > 0 AND status is not removed AND there has been no activity_logs row in the last 10 minutes (phantomStaleThreshold).
  • UPDATE ... SET active_tasks = 0, current_task = '', updated_at = now() RETURNING id, name — logs each reset by name plus a summary count.

Safeguards

  • Only sweeps when there is no activity log in the configurable stale window (default 10 min). A live agent logs at least a heartbeat/tool-call within that window, so genuinely busy workspaces are never touched.
  • Excludes status = 'removed' workspaces so soft-deleted rows don't churn.
  • Both thresholds are package-level constants (phantomSweepInterval, phantomStaleThreshold) — trivial to tune.
  • Reuses the existing scheduler goroutine + mutex — no new goroutine, no new timer, no new panic surface.

Scope

Only workspace-server/internal/scheduler/scheduler.go changes. Extracted verbatim from the PR #1664 diff (the target section, not the cosmetic formatting noise).

Test plan

  • go build ./... in workspace-server — clean
  • go test ./internal/scheduler/... -count=1 — pass
  • Deploy to staging and watch logs for Scheduler: phantom-busy sweep complete lines
  • Verify that workspaces in the known "stuck" state clear to active_tasks = 0 within 5 min
  • Verify that actively-working workspaces are not swept (their activity_logs are fresh)
  • Confirm the existing 30-min manual SQL cron can be retired

Closes #1372.

🤖 Generated with Claude Code

…shed runs (extracted from #1664)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@HongmingWang-Rabbit
HongmingWang-Rabbit merged commit 3c785bc into main Apr 23, 2026
9 of 10 checks passed
molecule-ai Bot pushed a commit that referenced this pull request Apr 23, 2026
…#1702/#1730/#1731

Confirmed via gh CLI (GH_TOKEN restored): langchain-ai/langgraph PRs #6645, #7113, #7205
still OPEN as of 2026-04-23T17:38Z. A2A live-today positioning vs LangGraph in-progress
remains accurate. Logged PR #1731 (sweepPhantomBusy), PR #1730 (45-min gh-token refresh daemon
fixing 60-min 401 in long sessions), and PR #1702 (SSH-backed file writes for SaaS — P1
regression fix). Blog post for #1702 at docs/marketing/blog/2026-04-23-saas-file-api-fix.md.

Co-Authored-By: Claude PMM <noreply@anthropic.com>
molecule-ai Bot pushed a commit that referenced this pull request Apr 23, 2026
…#1702/#1730/#1731

Confirmed via gh CLI (GH_TOKEN restored): langchain-ai/langgraph PRs #6645, #7113, #7205
still OPEN as of 2026-04-23T17:38Z. A2A live-today positioning vs LangGraph in-progress
remains accurate. Logged PR #1731 (sweepPhantomBusy), PR #1730 (45-min gh-token refresh daemon
fixing 60-min 401 in long sessions), and PR #1702 (SSH-backed file writes for SaaS — P1
regression fix). Blog post for #1702 at docs/marketing/blog/2026-04-23-saas-file-api-fix.md.

Co-Authored-By: Claude PMM <noreply@anthropic.com>
molecule-ai Bot added a commit that referenced this pull request Apr 23, 2026
…#1867)

* PMM: update ecosystem-watch — add LangGraph PR verification deferral note

- Add 2026-04-22 entry: GH API 401 for external repos, LangGraph PRs
  #6645/#7113/#7205 still VERIFY. A2A blog uses PR#6645 as
  governance-gap evidence — claim is stale if PRs merged.
- Update maintenance footer date to 2026-04-22

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* PMM: add Cloudflare Artifacts positioning brief

Source: PR #641, merged 2026-04-17.
Buyer: Platform engineers + enterprise security/compliance.
Headline: 'Give your agents a Git history — without touching a terminal.'
Objections covered: 'Why not GitHub?' + 'Cloudflare Artifacts is beta.'
Blocking: Social Media Brand launch thread.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* PMM: update EC2 SSH launch brief — social copy APPROVED, TTS audio file added as blocker

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* PMM: update ecosystem-watch — verify LangGraph PRs still OPEN, log PRs #1702/#1730/#1731

Confirmed via gh CLI (GH_TOKEN restored): langchain-ai/langgraph PRs #6645, #7113, #7205
still OPEN as of 2026-04-23T17:38Z. A2A live-today positioning vs LangGraph in-progress
remains accurate. Logged PR #1731 (sweepPhantomBusy), PR #1730 (45-min gh-token refresh daemon
fixing 60-min 401 in long sessions), and PR #1702 (SSH-backed file writes for SaaS — P1
regression fix). Blog post for #1702 at docs/marketing/blog/2026-04-23-saas-file-api-fix.md.

Co-Authored-By: Claude PMM <noreply@anthropic.com>

* docs(marketing): add PR #1702 release note + PR #1686 positioning brief

PR #1702 (SSH-backed file writes for SaaS): blog post covers fix, compute
model detection, EIC-based remote write path. Ships same-day after merge.

PR #1686 (Tool Trace + Platform Instructions): full positioning brief —
buyer matrix, value props, competitive angle vs Langfuse/Helicone/OPA,
objection handlers, cannibalization assessment (LOW).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(mmm): add Phase 34 positioning one-pager + messaging matrix

- phase34-positioning.md: one-pager with positioning statement,
  audience matrix, problem/solution, competitive differentiators,
  and proof points for press kit use
- phase34-messaging-matrix.md: 3 candidate taglines (production-grade,
  observability, aspirational) + full 4-feature messaging matrix
  (Partner API Keys, Tool Trace, Platform Instructions, SaaS Fed v2)
- SaaS Federation v2 flagged as content gap — no PM brief exists;
  community copy blocked pending PM confirmation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Molecule AI PMM <pmm@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
@HongmingWang-Rabbit
HongmingWang-Rabbit deleted the fix/scheduler-sweep-phantom-busy branch April 24, 2026 00:12
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
)

The lint-continue-on-error-tracking script requires a tracker comment
within 2 lines of every advisory continue-on-error directive. This adds
the missing internal#418 tracker to ci-arm64-advisory.yml.

Fixes #1731
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant