Skip to content

crash gate: a debug socket that never answers samples the app's main thread - #32239

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/crash-gate-no-socket-sample
Nov 2, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/crash-gate-no-socket-sample

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

When the debug socket never answers, the crash gate now says why: it samples the app's main thread and checks the socket in the app's state folder.

Base-red-job: cmux-next / cmux-next crash ends the app

What the red is

Every red run of the Crash ends the app step (cmux-scheme-compile) ends both launches with:

FAIL exception: the app's debug socket never answered (a Debug build is required); /tmp/cmux-debug-ci-crash-<run>-1.sock is absent, alive=True, last reply {'error': '[Errno 2] No such file or directory'}

The app's own output is only cmux: open-file limit 10240 -> 65536.

It follows the runner, not a commit

The suspected range (9e1c48fa22..e573965d34, including #28295 and #31904) is not the cause. The same failure shows up on owned minis hours before that range, and the same tips pass on Blacksmith macOS:

run built runner gate
41267653860 05f7dc9 (11-01 03:14Z) owned mini FAIL, socket absent
41284987256 6293c80 (11-01 09:10Z) owned mini PASS (socket answered after 31 s)
41287191917 70ca196 (11-01 09:08Z) Blacksmith macOS PASS
41340639162, 41343175400, 41349600162, 41361139761 11-01 19:20Z to 23:16Z owned minis FAIL, socket absent
41391725502 9efaeb7 (11-02 05:05Z) owned mini FAIL, socket absent
41397849902 6924ccb (11-02 06:29Z) owned mini FAIL, socket absent
41395056515 418f3c7 (11-02 06:55Z) owned mini FAIL, socket absent
41396153603, 41386476436, 41403125592 11-02 04:10Z to 07:02Z Blacksmith macOS PASS

Which commit looked like the first failure depended on whether that run's scheme compile landed on Blacksmith or on an owned mini. (#31600 keeps the landing DEV compile off Blacksmith macOS, which may be why owned minis now take more of these runs; I haven't verified that.)

The control socket opens only after settings.waitForLoad(atLeast: 1), the bounded launchSettle.settled(within: 4 s) and the classic-settings migrations (AppDelegate.startSettingsAndControl). On owned minis something there, or a blocked main thread, takes more than 180 s; the one passing owned run took 31 s. The log can't tell which.

This change

scripts/cmux-next/crash-ends-app-gate.py, on a socket that never answered:

  • also reports ~/.local/state/cmux/cmux-debug-<tag>.sock, the real socket. The /tmp path is only its compatibility link, so an absent link alone doesn't show whether a listener exists.
  • samples the still-running app for 2 s, saves the report as no-socket-<kind>.txt in the gate's out folder (already uploaded as the cmux-next-crash-gate-* artifact), and prints the main thread's call graph under the FAIL line.

The verdict is unchanged: still red. With the next red on an owned mini, the cause will be in the log.

Checks: py_compile; main_thread() tried on a synthetic sample report; scripts/cmux-next/prepush-gate.py reports no new violations.

Agent session: agent session

🤖 Generated with Claude Code


Summary by cubic

Improves the crash gate's failure report when the app's debug socket never answers, so the next red run names what kept the socket closed instead of only reporting an absent /tmp link. The gate verdict is unchanged: it still fails.

  • Also checks the real socket in the app's state folder (~/.local/state/cmux/cmux-debug-<tag>.sock), since the /tmp path is only a compatibility link to it.
  • Samples the still-running app for 2 s, saves the report as no-socket-<kind>.txt, and prints the main thread's call graph under the FAIL line.

Written for commit 5aee5c1. Summary will update on new commits.

View guided diff Turn on auto-fix

…thread and checks the moved socket

Since at least 2026-11-01 03:14Z the CI crash gate fails on owned minis with
"the app's debug socket never answered ... absent, alive=True" after 180 s,
while the same commits pass on Blacksmith macOS (and one owned-mini run passed,
answering after 31 s). The message said only that the /tmp link was absent.
It now also says whether the socket in the state folder exists (the /tmp path
is only a link to it), and samples the still-running app for 2 s: the whole
report goes to the out folder (no-socket-<kind>.txt, uploaded with the gate's
artifact) and the main thread's call graph is printed under the FAIL line, so
the next red names the wait or the block that kept the control socket closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-9124da39adbff74677d9
@cursor

cursor Bot commented Nov 2, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 5aee5c1: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr32239 -f ref=lane/crash-gate-no-socket-sample -f platform=macos -f backend_mode=local

Shipped in:

@teamleaderleo teamleaderleo left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (claude-opus-5-5): SHIP at 5aee5c1

Review (subagent claude-opus-5-5): SHIP at 5aee5c1
The no-socket branch still returns a failure string (it only adds detail), so it cannot turn red into green; the moved path matches ControlSocketPath's tagged-debug row, a missing or unparseable sample gives an empty stack and drops the section, and the finally kill plus teardown.end() still run.
Follow-ups: [follow-up] sample() (existing helper) lets subprocess.TimeoutExpired (60 s) propagate; the run still ends red with teardown, but the FAIL line and later kinds are lost. Catching it in sample() would keep the message.

@teamleaderleo
teamleaderleo merged commit 2b2ab03 into feat-cmux-next Nov 2, 2026
155 of 174 checks passed
@teamleaderleo
teamleaderleo deleted the lane/crash-gate-no-socket-sample branch November 2, 2026 12:38

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 5aee5c1c Deployed Nov 2, 2026 by teamleaderleo via wait for same-key tree publisher #101116
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant