Skip to content

Revert #16397: Cloud VM creates fail, pool-init migration not applied - #16405

Merged
lawrencecchen merged 1 commit into
mainfrom
revert-16397-pool-init-outage
Oct 1, 2026
Merged

lawrencecchen merged 1 commit into
mainfrom
revert-16397-pool-init-outage

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Production outage, standby revert. Since 2026-10-01 08:44 UTC every Cloud VM create fails with model_plane_unavailable (14 of 14 creates; last success 08:23 UTC, the minute before #16397 deployed).

Cause: #16397 (deployed 08:23:47 UTC) inserts into coderouter_pool_initializations during route-token issue. Its migration 20261001000000_coderouter_vm_pool_initialization was never applied: to_regclass('public.coderouter_pool_initializations') is NULL on both PlanetScale cmux-prod/main and staging (checked read-only). Production migrations do not run at deploy.

Preferred fix is to apply that reviewed, additive migration (CREATE TABLE + backfill INSERT) to staging then production, which keeps #16397. This revert is the alternative if the migration cannot be applied quickly. Merge only one of the two.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Reverts #16397 to restore Cloud VM creation, which has been failing with model_plane_unavailable since #16397 deployed because its migration 20261001000000 was never applied to staging or production. This removes the coderouter_pool_initializations table, the migration, and the pool seeding during token issue, so VM creates no longer reference the missing table. The additive migration can be applied later to re-enable the feature without a revert.

Written for commit a405faf. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Behavior Changes
    • VM default pools are no longer automatically populated with team-visible Codex and Claude accounts when an authorization token is issued.

Production outage: every Cloud VM create fails with model_plane_unavailable
since 2026-10-01 08:44 UTC. #16397 inserts into coderouter_pool_initializations,
but its migration 20261001000000 was never applied to staging or production.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: b01057b5-c2c7-4999-9b21-225ac3295199

📥 Commits

Reviewing files that changed from the base of the PR and between 6aa6343 and a405faf.

📒 Files selected for processing (4)
  • web/db/migrations/20261001000000_coderouter_vm_pool_initialization/migration.sql
  • web/db/schema.ts
  • web/services/coderouter/repository.ts
  • web/tests/coderouter-vm-scope-db-behavior.test.ts
💤 Files with no reviewable changes (4)
  • web/db/schema.ts
  • web/tests/coderouter-vm-scope-db-behavior.test.ts
  • web/services/coderouter/repository.ts
  • web/db/migrations/20261001000000_coderouter_vm_pool_initialization/migration.sql

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

This change removes the VM pool initialization table and migration, stops VM authorization token issuance from initializing default pools or granting team-visible accounts, and removes a test of VM account visibility.

Changes

VM pool initialization removal

Layer / File(s) Summary
Remove initialization storage
web/db/migrations/20261001000000_coderouter_vm_pool_initialization/migration.sql, web/db/schema.ts
The migration no longer creates or seeds the pool initialization table. The schema no longer declares the table.
Remove token-issuance initialization
web/services/coderouter/repository.ts, web/tests/coderouter-vm-scope-db-behavior.test.ts
VM authorization token issuance no longer records pool initialization or grants team-visible Codex and Claude accounts. The test for VM account visibility and pool grant removal is deleted.

Priority: ⬆️ High

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: 🔵 Low · up to a405f

This reverts the VM pool initialization feature to restore Cloud VM creation. Merge it only if the migration cannot be applied quickly, and do not also apply the alternative fix. The feature's behavior, including default pool setup and team-visible accounts, is lost until it is reintroduced.

🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description clearly explains the production failure, cause, affected environments, and alternative remediation. It does not follow the required template because it omits the Testing, Changelog, an… Add the required Summary, Testing, Changelog, and Checklist sections. In Testing, list executed commands or CI checks and any unverified production behavior. Add a present-tense Changelog line, or state none if appropriate. State why no d…
✅ Passed checks (24 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the revert and the Cloud VM creation failure caused by the unapplied pool-initialization migration.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input ✅ Passed The check is not applicable to this diff. The pull request changes only CodeRouter pool initialization: it deletes a migration and schema table, removes pool-account seeding from `issueVmAuthorization…
Cmux Swift Actor Isolation ✅ Passed PASS: The pull request changes only SQL and TypeScript files. The authoritative diff contains no Swift files or Swift code, so it cannot introduce or worsen Swift 6 actor isolation issues.
Cmux Swift Blocking Runtime ✅ Passed PASS: The authoritative PR diff changes only four web/TypeScript/SQL/test files and contains no Swift files. It removes code and adds no blocking or timing-based synchronization primitive in productio…
Cmux Browser Automation Off-Main ✅ Passed PASS: The pull request changes only web database schema, migration, repository, and tests. It does not change Sources/TerminalController.swift, ControlCommandExecutionPolicy.swift, or any browser …
Cmux Expensive Synchronous Load ✅ Passed PASS: The authoritative diff changes only three TypeScript files and one SQL migration, with 86 deletions. It adds or moves no Swift code, agent-history load, transcript/workspace scan, or synchronous…
Cmux Cache Substitution Correctness ✅ Passed The pull request only deletes the pool-initialization migration, schema table, VM pool-grant transaction, and its test. It does not replace an authoritative read with a cached or opportunistic value. …
Cmux No Hacky Sleeps ✅ Passed The PR removes a migration, a schema declaration, and route-token initialization logic. It does not add or expand sleep, timers, polling, fixed delays, or wall-clock waits in production runtime code…
Cmux Algorithmic Complexity ✅ Passed PASS: The pull request only deletes code, a migration, a schema declaration, and a test. It adds no production Swift, TypeScript, JavaScript, shell, or runtime algorithm. Therefore it does not introdu…
Cmux Swift Concurrency ✅ Passed The pull request changes only one SQL migration and three TypeScript files. The authoritative diff contains no Swift files, so it does not introduce or expand Swift concurrency patterns.
Cmux Swift @Concurrent ✅ Passed PASS: The pull request changes only SQL and TypeScript files. The authoritative diff contains no Swift, Objective-C, or Objective-C++ files, so it introduces no Swift concurrency annotation or actor-i…
Cmux Swift Package Boundaries ✅ Passed The PR changes only web TypeScript/SQL and test files. It introduces no Swift source, SwiftPM manifest, or Swift package boundary change, so the Swift package boundary check does not apply.
Cmux Swiftpm Lockfiles ✅ Passed The PR changes only database schema, migration, TypeScript repository code, and a TypeScript test. It changes no SwiftPM package, Package.resolved file, Xcode project reference, .gitignore, workflow, …
Cmux Swift Logging ✅ Passed PASS: The pull request changes only SQL and TypeScript files. It adds no Swift paths or Swift logging statements, so the Swift logging rule does not apply.
Cmux User-Facing Error Privacy ✅ Passed The PR only deletes a migration, its schema declaration, VM pool initialization logic, and a test. The diff adds no user-facing error, alert, API body, command output, or recovery copy. The existing C…
Cmux Full Internationalization ✅ Passed The PR only deletes a database migration, a schema declaration, backend pool-initialization logic, and a test. It adds or changes no user-facing text, API copy, metadata, markdown, or locale data. No …
Cmux Swiftui State Layout ✅ Passed The reviewed diff changes only four web TypeScript/SQL/test files and deletes database/repository code. It contains no SwiftUI, Swift, AppKit, or layout/state changes, so the SwiftUI state-layout chec…
Cmux Architecture Rethink ✅ Passed PASS. The authoritative diff contains only SQL and TypeScript changes in four web files, with no Swift paths or Swift architectural code. The Swift-specific failure conditions do not apply.
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed The pull request changes only four web TypeScript/SQL/test files. It introduces no Swift or standalone cmux-owned window changes, so the auxiliary-window close-shortcut rule does not apply.
Cmux Source Artifacts ✅ Passed The diff changes only intentional source, migration, and test paths. It contains 86 deletions and no added files or artifact directories. The deleted migration is an artifact removal, which the rule e…
Cmux No Test Or Debug Seam In Production Source ✅ Passed PASS: The authoritative PR diff changes only SQL and TypeScript files. It contains no Swift file under a production Sources/ path, so the custom check is not applicable.
Full details: Description check

Explanation

The description clearly explains the production failure, cause, affected environments, and alternative remediation. It does not follow the required template because it omits the Testing, Changelog, and Checklist sections, and it does not state what tests were run or what remains unverified.

Resolution

Add the required Summary, Testing, Changelog, and Checklist sections. In Testing, list executed commands or CI checks and any unverified production behavior. Add a present-tense Changelog line, or state none if appropriate. State why no demo video is needed for this backend-only change, if applicable.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@lawrencecchen
lawrencecchen merged commit 15aa32c into main Oct 1, 2026
67 of 69 checks passed
@lawrencecchen
lawrencecchen deleted the revert-16397-pool-init-outage branch October 1, 2026 15:48
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Merge receipt for a405faffda: every check was green at merge (18 verified; 18 skipped by policy). Full suite runs on main after merge.

lawrencecchen added a commit that referenced this pull request Oct 2, 2026
…#16572)

Reverts #16405. #16397 was reverted because its migration
20261001000000_coderouter_vm_pool_initialization was never applied to
staging or production, so every Cloud VM create failed. This re-land
merges only after that migration is applied to staging and production.

Co-authored-by: austinywang <austinywang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant