Skip to content

crash gate: a fleet step runs the crash gate on any feat commit; a socket timeout says what the socket did - #28813

Merged
lawrencecchen merged 1 commit into
feat-cmux-nextfrom
crash-gate-fleet-step
Oct 31, 2026
Merged

lawrencecchen merged 1 commit into
feat-cmux-nextfrom
crash-gate-fleet-step

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • New fleet step scripts/measure/crash-gate.sh: it builds a tagged Debug cmux-next app at the step's commit (reload.sh, the same route as scripts/measure/tab-switch.sh) and runs scripts/cmux-next/crash-ends-app-gate.py on it. On a feat-cmux-next push the CI crash gate job is skipped, so no tip has run the gate since cmux-tui hook: one queue and one connection per agent; a busy daemon never costs a hook event #28041 merged (00:35:22Z). This gives the tip the gate without a CI change:
    cmux-ci run --class exclusive --label measure --label gui-session-v1 --script scripts/measure/crash-gate.sh --ref SHA
  • crash-ends-app-gate.py: the message "the app's debug socket never answered" (run 41132712722 attempt 1, the only gate result of cmux-tui hook: one queue and one connection per agent; a busy daemon never costs a hook event #28041) gave no cause. The message now names the socket path, whether it exists, whether the app still runs, and the last reply. A timeout means the main thread did not serve debug.crashes; a refused connect means no listener. The pass and fail rules do not change.

The script must be on feat-cmux-next before the controller runs it (an unlisted scripts/measure script runs only at a commit on a reviewed branch, and the worker checks its bytes against the branch tip).

Changelog

none

Observed / Not run

  • Observed: bash -n and py_compile pass; prepush-gate.py reports no new violations. The CI log of run 41132712722 (job 124327772537, cmuxs-Mac-mini-4): both launches ran 180 s, app-*.log holds only the open-file limit line, both runs end "debug socket never answered".
  • Not run: the fleet step itself (it needs this commit on feat-cmux-next). It runs on the tip right after merge; the result goes in a comment here.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Adds a fleet step that runs the cmux-next crash gate on a tagged Debug build of any feat commit, since CI skips that gate on feat-cmux-next pushes. The step builds the app via reload.sh (the same route as tab-switch.sh) and runs crash-ends-app-gate.py; pass and fail rules are unchanged. Run it per commit with cmux-ci run --class exclusive --label measure --label gui-session-v1 --script scripts/measure/crash-gate.sh --ref SHA.

The gate's socket-timeout message now says what the socket did: its path, whether it exists, whether the app still runs, and the last reply. A timeout means the main thread never served debug.crashes; a refused connect means no listener.

Written for commit d2be521. Summary will update on new commits.

View guided diff Turn on auto-fix

… any feat commit; a socket timeout says what the socket did

scripts/measure/crash-gate.sh builds a tagged Debug cmux-next app at the step's commit
(reload.sh, as tab-switch.sh) and runs scripts/cmux-next/crash-ends-app-gate.py on it.
On a feat-cmux-next push the CI crash gate is skipped, so no tip had run it since #28041
merged; this is the fleet route (cmux-ci run --class exclusive --label measure
--label gui-session-v1 --script scripts/measure/crash-gate.sh --ref SHA). No CI change.

crash-ends-app-gate.py: "the app's debug socket never answered" (run 41132712722
attempt 1, #28041's only gate result) gave no cause. The message now names the socket
path, whether it exists, whether the app still runs and the last reply (a timeout means
the main thread did not serve debug.crashes; a refused connect means no listener).
@lawrencecchen lawrencecchen added the risk:low Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Independent review (fresh subagent, read-only, correctness first): no blocking defect.

  • The app name, bundle id and socket match: cmux DEV crashgate-<hex>.app, com.cmuxterm.app.debug.crashgate.<hex>, LSEnvironment CMUX_TAG=crashgate-<hex>. Both sides resolve the socket to /tmp/cmux-debug-crashgate-<hex>.sock. The /tmp link exists, and the socket stays at the /tmp name when the state path is too long.
  • The build is Debug (the default configuration), so the debug.crash* methods are registered.
  • The gate's pass and fail rules do not change. The new diagnostic does not raise.
  • The shell is correct under set -euo pipefail. The trap keeps the gate's exit status.
  • Not blocking: a teardown: line prints after the verdict line. Each SHA gets its own tag, so each build is cold.

Review (claude-subagent): SHIP at d2be521

@lawrencecchen lawrencecchen added ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds and removed ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds labels Oct 31, 2026
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (independent review subagent, Claude Opus 5.5): SHIP at d2be521

Scope: I read the diff origin/feat-cmux-next...d2be5213e63 (scripts/measure/crash-gate.sh, scripts/cmux-next/crash-ends-app-gate.py). I also read scripts/measure/tab-switch.sh, scripts/reload.sh (options, tag naming, Info.plist staging, the "App path:" summary), scripts/lib/mobile-attach.sh (slug and bundle id helpers), the "Crash ends the app" step in .github/workflows/cmux-next.yml, LaunchIdentity.swift, ControlSocketPath.swift, BuildIdentity+AppNames.swift (KeptNames), BuildIdentity.tagSlug, the DEBUG guards in AppControl.swift and the TagTeardown CLI. I did not build, compile or run anything.

Findings (none block):

  1. App path and identity match the gate. reload.sh prints "App path:" and then " " on the next line. The sed expression takes that line and strips only the leading spaces, so the spaces in "cmux DEV crashgate-<8 hex>.app" stay. The tag crashgate-<8 hex> is already a slug (lowercase hex). So APP_NAME is cmux DEV crashgate-<hex>, the bundle id is com.cmuxterm.app.debug.crashgate.<hex> and LSEnvironment CMUX_TAG is crashgate-<hex>. The executable stays Contents/MacOS/cmux DEV. The gate launches the binary directly, so LSEnvironment does not go into the process environment. But LaunchIdentity reads CMUX_TAG from the Info.plist dictionary. KeptNames accepts the bundle id in reload form (segments crashgate, <hex>, all [a-z0-9]). Its suffix slug crashgate-<hex> is equal to the shipped tag, and the tag is not an owned tag. The socket is /tmp/cmux-debug-crashgate-<hex>.sock, or a /tmp link to the same name in ~/.local/state/cmux when the path fits sun_path. That is the path the gate derives from the bundle name. No block.
  2. The build is Debug. The script passes no --config or --release, and reload.sh's default is BUILD_CONFIGURATION="Debug". debug.crashes, debug.crash.app and debug.crash.exception are registered under #if DEBUG || CMUX_BENCH_HOOKS in AppControl.swift. No block.
  3. The .py change keeps pass/fail as it was. rpc(...) or {"error": "empty reply"} followed by "error" not in reply has the same truth value as the old or {"error": 1} form for every reply (dict, list, string, None, empty). wait() always calls answered() at least once, and the message uses last.get. os.path.exists, process.poll() and the f-string cannot raise. The return happens inside the same try, so finally still kills the child and ends the teardown. No block.
  4. The shell is correct under set -euo pipefail. If reload.sh fails, pipefail stops the script with reload's own status, and reload already writes its log to the tee. The [ ] && [ ] && export loop returns non-zero only through an && list, which errexit ignores (the same code as in tab-switch.sh). The EXIT trap is set only after the app is found. It ends with || true and calls no exit, so the script exits with the gate's status. "$@" with no arguments is safe under set -u, and every argument goes to the gate unchanged (for example --runs=...). No block.
  5. I found nothing else that could give a wrong verdict. The gate's last line ("PASS ..." or "FAIL ...") and its exit status are the step's verdict.

Non-blocking notes:

  • The CI step's trap also deletes the tag's state: tags/, the cmux-next marker folder, ~/Library/Caches/SentryCrash/cmux DEV , ~/.acpmux/builds and ~/.cmux/mux/tags. crash-gate.sh does not. With a per-step HOME this does not matter. With a shared HOME, a rerun on the same sha starts with the old crash reports and markers. The suspended check requires fresh mtimes, so this does not change the verdict.
  • Each sha gets a new tag. Without CMUX_DERIVED_DATA, each sha then gets a new per-tag DerivedData, so every run is a cold build. tab-switch.sh uses a fixed tag. This costs time only.
  • When the /tmp name is a dangling link, os.path.exists follows it and the message says "absent". "a socket" does not check S_ISSOCK. Both affect only the message text.

@lawrencecchen lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 31, 2026
@lawrencecchen
lawrencecchen merged commit 2c8d1b8 into feat-cmux-next Oct 31, 2026
118 of 147 checks passed
@lawrencecchen
lawrencecchen deleted the crash-gate-fleet-step branch October 31, 2026 03:09
@github-actions github-actions Bot added risk:normal Risk tier (docs/ci/risk-tiers.md) and removed risk:low Risk tier (docs/ci/risk-tiers.md) labels Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 51ed666; does not block

Shipped in:

@blacksmith-sh

blacksmith-sh Bot commented Oct 31, 2026

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_
a_picked_signing_label_has_a_rerun_alternative)/
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_
a_picked_signing_label_has_a_rerun_alternative)
View Logs
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes
t_no_runs_on_chain_ends_on_the_reserved_side_label)/
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes
t_no_runs_on_chain_ends_on_the_reserved_side_label)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

This branch was previously deployed

1 inactive deployment
ci-routing-agent — d2be5213 Deployed Oct 31, 2026 by lawrencecchen via wait for same-key tree publisher #86919
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant