Repository navigation
ci: fail stalled Swift package tests instead of waiting out the job timeout - #14997
Conversation
…imeout The package loop captured each `swift test` into a shell variable, so a hung test printed nothing and the job burned its 60 minutes before GitHub cancelled it (run 36314061362: 53.6 minutes inside CmuxTerminal, zero output). Each package, and the Bonsplit step, now streams through hung_test_watchdog.py into a log file that the existing tolerance and startup retry checks read. The watchdog gains --stall-seconds: after the build, no test starting or finishing for 120 s fails the package with an annotation naming the unfinished tests, after sampling the test processes. A 15 minute per-package limit stays as a backstop. run-swift-testing-suites.sh uses the same stall limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedNext included review available in 2 minutes. View limit detailsLimit details: You’ve used all 10 included reviews currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Repository: manaflow-ai/cmux/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (6)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
All contributors have signed the CLA ✍️ ✅ |
The package loop exited at the first failing package, so one broken or hung package hid every package after it (on #14876, CmuxTerminal failed and the PR's own CmuxUpdater tests never ran). Each package now runs through test_package, which returns its status; the step prints a result table (package, passed / failed / stalled, seconds), adds it to the job summary, annotates each failed package once and exits with the first failure's status. Stalled packages keep the watchdog's annotation that names the stuck tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
|
@coderabbitai review |
|
|
@coderabbitai review |
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
|
Merge receipt for |
648d5c1 Add a Paste Last Screenshot action with an unbound shortcut (manaflow-ai#14955) ff61677 ci: avoid partial blobs in catch-up merges (manaflow-ai#15023) 4d0d112 ci: retry transient catch-up GraphQL failures (manaflow-ai#15021) 212e808 ci: attribution scores a lone suspect and reports app-host crashes apart (manaflow-ai#14952) 4cabdf4 test: settle the window before measuring the unread sidebar-row invalidation (manaflow-ai#14568) 12ec99b Add a release-media capture tool for changelog screenshots and clips (manaflow-ai#15010) ee2cda0 Backfill Unreleased changelog and draft next release cards (manaflow-ai#14999) be4adf8 Show a brief notice when Cmd+V fails on an oversized image or a timeout (manaflow-ai#14953) 23d22d7 ci: an owned pool the run starts on now beats an earlier one it queues on (manaflow-ai#14993) 05d0190 ci: catch-up posts once per head, says less, and merges inserted declarations (manaflow-ai#15018) 4ee4b21 ci: fail stalled Swift package tests instead of waiting out the job timeout (manaflow-ai#14997) 9ce512a merge-main: run local guards only when asked (manaflow-ai#15016) d60108a ci: clear test-e2e's fixed DerivedData with clear-dirs.sh (manaflow-ai#14994) 1d7895e ci: run the shell and CLI no-socket lanes in parallel (manaflow-ai#14990) 6e7d25f Honor macOS Differentiate Without Color, Increase Contrast and Reduce Transparency (manaflow-ai#14991) 966b355 Stop interrupting focused work: sidebar jumps, Computer Use focus steal, quit dialog on logout (manaflow-ai#14961) e1f1cb2 Strip control characters from feedback attachment filenames (manaflow-ai#14783) 0758c9f test: find the onboarding window the test presented, not a leftover (manaflow-ai#15015) b35c540 fix(spm): resolve GhosttyKit/GhosttyRuntimeTestStubs target name collisions (manaflow-ai#10569) ef33bed Map .purs artifacts to the Haskell highlight.js grammar (manaflow-ai#14202) e2a167a Highlight Elixir and Erlang files in the file editor (manaflow-ai#13732) 972c449 fix: wrap Linux browser download card label (manaflow-ai#11157) f563884 Add Aside to browser data import detection (manaflow-ai#13379) 091d0ea Add cmux send --paste and hint at it for large multi-line sends (manaflow-ai#14937) 3ffcdbb test(ios): keep folder-tap stat tests off the real 2 s deadline (manaflow-ai#15017) 68d3936 test: keep CmuxTerminal pasteboard tests off the cooperative pool (manaflow-ai#15006)
1 participant
Summary
A hung Swift package test used to cost the whole 60 minute
macos / swift-package-testsjob and leave nothing in the log, and any failing package hid every package after it. Run 36314061362 spent 53.6 minutes silent insideCmuxTerminal, was cancelled at the job timeout, and never ran the 9 packages after it. Main hit the same thing on 09-27 inCmuxSwiftRenderUI(run 36295033926). On #14876 aCmuxTerminalfailure meant the PR's ownCmuxUpdatertests never ran.Hangs fail fast and say where. Every package, and the Bonsplit step, now streams
swift testlive throughscripts/ci/hung_test_watchdog.pyinstead of capturing it into a shell variable. The watchdog also writes a log file, and the GhosttyKit binary-name tolerance, the signal 5/6 startup retry andrequire_swift_test_execution.pyread that file, so their behavior is unchanged. The watchdog gets a--stall-secondsmode: once the build is done (Build complete!or the first test event), if no test starts or finishes for 180 s it prints the tests still in flight, runssampleon the xctest / swiftpm-testing-helper processes into the log, kills the process tree and fails the package:Output alone doesn't reset the clock, so a stuck test that keeps logging still fails. A 15 minute per-package total stays as a backstop.
run-swift-testing-suites.sh(CmuxFoundation and the auth/transport packages) uses the same stall limit next to its existing 300 s per-suite limit.CMUX_SWIFT_TEST_STALL_SECONDSandCMUX_SWIFT_PACKAGE_TEST_TIMEOUT_SECONDSoverride the limits. 180 s leaves room even on a slowed-down mini: the longest single test in recent full package runs is 47 s (CmuxAuthRuntime), then 27 s (CmuxCommandPalette), and the gap between test events can't be longer than the longest running test.Every selected package runs. A failed or stalled package no longer stops the loop. The step ends with a table (package, passed / failed / stalled, seconds), also written to the job summary, one
::error::per failed package (a stalled package keeps the watchdog's annotation), and exits with the first failure's status.Testing
tests/test_hung_test_watchdog.py: new cases cover a stall on a chatty process (fails, names the test, kills the child), build output not arming the stall clock, and test events resetting it. 28/28 pass on a fleet mini.tests/test_ci_notification_semantics.py: a new case runs the real step over every package withCMUXAgentLaunch(sorted first) failing, and checks thatCmuxTerminalandCmuxUpdaterstill ran, that the table and exactly one annotation name the failure, and that the step exits nonzero.tests/test_ci_select_package_tests.py,tests/test_swift_package_execution.py,tests/test_swift_testing_suite_timeout.py,tests/test_ci_macos_xcode_selection.pypass. They run the real step shell with a fakeswift, now through the watchdog.CmuxTerminalhangs and printed the stuck tests and their stacks, which led to the root cause fix in test: keep CmuxTerminal pasteboard tests off the cooperative pool #15006 (merged).Changelog
none
🤖 Generated with Claude Code