Skip to content

Never point a non-production Cloud machine's edge at production coderouter - #18313

Merged
lawrencecchen merged 2 commits into
mainfrom
fix-dev-vm-edge-origin
Oct 7, 2026
Merged

lawrencecchen merged 2 commits into
mainfrom
fix-dev-vm-edge-origin

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

The reflection 401 flood (~41k per 30 min, 2.0M on 2026-10-06) is not credential expiry. After #18307 deployed, every refusal on /api/vm/reflection/name reports cmux.coderouter.auth_failure_detail = signed_unverified (4,138 in 3 min, 0 not_live): the edge injects a signed x-cmux-authorization that production cannot verify.

Cause: dev-backend stacks sign machine credentials with their own key (CMUX_VM_AUTH_SIGNING_KEY_ID=dev-backend-1 in the curated dev env) and set no CMUX_CODEROUTER_EDGE_ORIGIN, so resolveEdgeOrigin fell back to https://coderouter.dev. Every Freestyle machine a dev stack creates (same Freestyle account) therefore sends its reflection, resource-usage and model traffic to production with a token production refuses, for the machine's whole life, also after its dev stack is stopped or purged (299 registered stacks). The guest retries the refusal every 8 s (#18308 fixes that for new snapshots).

Fix: without an explicit origin, only production defaults to coderouter.dev and a Vercel preview to its branch URL. Any other deployment provisions no edge rule and mints no token, with a warning naming CMUX_CODEROUTER_EDGE_ORIGIN. Production can verify neither a dev key nor a dev database row, and the Freestyle edge cannot reach the private dev host, so such a rule could never work. A guest with no rule gets a DNS failure for reflection.cmux.internal and never reaches production.

Diagnostics: a refused signed token now records its unverified key id (cmux.coderouter.auth_key_id) and signed_unknown_key when this deployment holds no such key, so the remaining traffic from already-running dev machines can be confirmed and counted in Axiom.

Red/green on cmux-lawrence-2: tests/coderouter-vm-model-plane.test.ts and tests/coderouter-request-telemetry.test.ts fail at e0bbc24, pass at head. Every coderouter*, vm-reflection*, vm-principal*, vm-model*, vm-freestyle-provider*, vm-create*, vm-route-workflow* test file passes individually. Typecheck, complexity gate and eslint pass (two pre-existing unused-import warnings in the model-plane test).


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Stops dev and local deployments from wiring their machines' edge to production coderouter, ending the reflection 401 flood caused by production refusing tokens signed with dev keys.

Previously any deployment without CMUX_CODEROUTER_EDGE_ORIGIN defaulted to coderouter.dev, so every machine a dev backend created dialed production with a token it cannot verify, forever. Now only production defaults to coderouter.dev and Vercel previews to their branch URL; other deployments provision no edge rule and mint no token, with a warning naming the env var.

Diagnostics

  • Refused signed tokens now record their unverified key id (cmux.coderouter.auth_key_id) and a new signed_unknown_key detail when the deployment holds no such key.

Written for commit 98dabbb. Summary will update on new commits.

Review in cubic Turn on auto-fix


Note

Medium Risk
Changes VM create provisioning and edge-origin defaults (fewer rules on dev stacks); production and preview behavior is preserved, but misconfigured CMUX_CODEROUTER_EDGE_ORIGIN could leave machines unwired.

Overview
Stops dev/local Cloud VM creates from installing Freestyle edge rules that forward guest traffic to production coderouter.dev when the stack signs VM JWTs with its own key and DB—those requests were failing as signed_unverified and flooding reflection 401s.

Model plane: resolveEdgeOrigin now returns null outside production and Vercel preview unless CMUX_CODEROUTER_EDGE_ORIGIN is set; provisionVmModelPlane then skips token minting and returns no edge rules (with a console warning). Production still defaults to coderouter.dev; previews still use the branch URL.

Auth telemetry: Refused signed x-cmux-authorization tokens record the unverified JWT kid on the span (cmux.coderouter.auth_key_id) and use a new detail signed_unknown_key when this deployment does not hold that signing key, via unverifiedVmAuthorizationKeyId / isKnownVmAuthorizationKey and signedRefusal in route token auth.

Docs and tests cover the unwired non-prod path and the unknown-key diagnostic.

Reviewed by Cursor Bugbot for commit 98dabbb. Bugbot is set up for automated code reviews on this repo. Configure here.

…oderouter

Dev backends sign machine credentials with their own key (dev-backend-1)
but set no CMUX_CODEROUTER_EDGE_ORIGIN, so resolveEdgeOrigin sent their
Freestyle machines to coderouter.dev. Production refuses every such
request (signed_unverified), and the machines keep polling it for life,
also after their dev stack stops: the bulk of the reflection 401 flood.
Without an explicit origin, only production defaults to coderouter.dev
and previews to their branch URL; other deployments provision no rule.
Refused signed tokens now name their unverified key id on the span.
@coderabbitai

coderabbitai Bot commented Oct 7, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 51 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 1747c568-776f-42f9-ab55-99ed8a51b731
📥 Commits

Reviewing files that changed from the base of the PR and between 9755545 and 98dabbb.

📒 Files selected for processing (7)
  • web/services/coderouter/requestTelemetry.ts
  • web/services/coderouter/routeTokenAuth.ts
  • web/services/coderouter/vmAuthorization.ts
  • web/services/coderouter/vmModelPlane.ts
  • web/services/vms/README.md
  • web/tests/coderouter-request-telemetry.test.ts
  • web/tests/coderouter-vm-model-plane.test.ts
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@lawrencecchen
lawrencecchen merged commit 3cab11d into main Oct 7, 2026
77 of 78 checks passed
@lawrencecchen
lawrencecchen deleted the fix-dev-vm-edge-origin branch October 7, 2026 08:26
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Merge receipt for 98dabbbcfc: every check was green at merge (20 verified; 20 skipped by policy). Full suite runs on main after merge.

rustybret pushed a commit to rustybret/bmux that referenced this pull request Oct 7, 2026
d5bec00 fix: repaint Codex sessions after pane reattachment (manaflow-ai#18283)
df3e34c fix(cloud): resolve 0.65.0 dogfood papercuts (manaflow-ai#18292)
7a5ff86 fix(cloud): restore from a snapshot with a stopped daemon supervisor (manaflow-ai#18320)
bf0d292 Fix crash when a focused SwiftUI view outlives its hosting view (manaflow-ai#18250)
6b02ff8 feat(gh-merge-green): --revert rolls back a merged PR in one command (manaflow-ai#18312)
3cab11d Never point a non-production Cloud machine's edge at production coderouter (manaflow-ai#18313)
5df12be fix: let Claude Teams launch with integration disabled (manaflow-ai#18276)

# Conflicts:
#	.github/workflows/ci-guards.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant