Skip to content

Fix the flaky crash-report case in swift-test-summary.test.sh - #23129

Merged
teamleaderleo merged 2 commits into
feat-cmux-nextfrom
fix-crash-report-test-mtime
Oct 21, 2026
Merged

teamleaderleo merged 2 commits into
feat-cmux-nextfrom
fix-crash-report-test-mtime

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

The "test helper dies on a signal" case in scripts/cmux-next/tests/swift-test-summary.test.sh flakes in two ways. This PR fixes both in the test only; the sampler script is unchanged.

  1. The sampler's log vanished on CI. On cmux-next sidebar: account chip and Search on one line above the bottom margin; profile picture in the avatar #23076's Linux CI (job 119082777090) the check printed "no crash ::error line". Its log shows awk: fatal: cannot open file /tmp/swift-test-log.RQ7ALW and sed: can't read the same file: the sampler's mktemp log in the shared /tmp was gone before the report step read it. Without the log it never saw the signal line, so it printed no crash error at all. The check runs beside many others under scripts/ci/run-checks-concurrently.sh. I did not find what deletes the file. The test now gives every sampler run TMPDIR=$TMP/tmp, so the log lives in the test's own temp dir.
  2. A whole-second mtime race (macOS, locally). The sampler records started_at=$(date +%s), and ips-crash-summary.py skips reports with mtime below it. A report written at second X.9 with a run start at X+1 counts as from before the run, and the test fails with "the crash error does not name the crashed thread's test frame". The test now sets the fresh report's mtime 60 s ahead with python3 os.utime; nothing in the test sorts by mtime or bounds it from above. The old-report touch -t 202001010000 line is unchanged.

Review (claude-review-subagent) at c55857b found that change 2 alone could not cause the CI failure above; change 1 is the response.

Changelog

none

Observed / Not run

  • Observed (macOS, this Mac, 2 = only the mtime change): 15/15 passed. Base: 14/15 in the same loop, and 1 of 3 on an untouched tree, both with the frame message.
  • Observed (macOS, with both changes at 9093e36): 86 of 87 runs passed. The output of the one failure was not kept, so I don't know its cause.
  • Not reproduced: the CI deletion of the sampler log. Change 1 removes the shared /tmp from the path; whether it fully stops the CI flake shows only on CI over time.
  • Observed: python3 scripts/cmux-next/prepush-gate.py reported no new violations.

Agent session: cosmic-gray

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes the flaky "test helper dies on a signal" case in scripts/cmux-next/tests/swift-test-summary.test.sh where a fresh crash report was sometimes skipped as too old.

The run wrapper records started_at in whole seconds; a report written at second X.9 followed by a start at X+1 is read as predating the run, so ips-crash-summary.py skips it. The test now sets the report's mtime 60 seconds ahead via python3 os.utime, leaving the script's skip rule unchanged. python3 is used instead of date -v/date -d since the test runs on both macOS and Linux.

Written for commit c55857b. Summary will update on new commits.

View guided diff Turn on auto-fix

…st.sh

The crash case wrote its fresh .ips report and then started the wrapper,
which records started_at in whole seconds and skips reports older than it.
A write at second X.9 and a start at X+1 read the fresh report as from
before the run, so the case failed with "no crash ::error line". Set the
report's mtime 60 seconds ahead so it always counts as written during the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d2c-88e4-7fc2-b58e-9a55abdaced8&host=cosmic-gray
@teamleaderleo teamleaderleo added the risk:low Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@cursor

cursor Bot commented Oct 21, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 9093e36: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23129 -f ref=fix-crash-report-test-mtime -f platform=macos -f backend_mode=local

@blacksmith-sh

This comment has been minimized.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by projects-08 on cosmic-gray · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (claude-review-subagent): CHANGES at c55857b

The change is correct as far as it goes, but it does not fix the failure it cites.

  • The mtime race is real: started_at=$(date +%s) rounds down, ips-crash-summary.py drops getmtime < since, so a write at X.9 with a start at X+1 skips the report. Setting the mtime 60 s ahead fixes that. Nothing else in the test cares about a future mtime: copy2 keeps it, the copy-out check only tests that the file exists, nothing sorts by mtime, there is no upper bound, and with CMUX_NEXT_CRASH_WAIT_SECONDS=0 the first scan finds the report. python3 os.utime works on macOS and Linux, and the wrapper needs python3 already. The test passed 3/3 here on macOS.
  • That race cannot produce "no crash ::error line". Once the signal line is in the log, the wrapper always prints the swift test crash ::error line; a skipped report only makes it say "crashed thread: no crash report from this run". I checked by running the wrapper with a report dated 5 s before the run. The test would then fail at "the crash error does not name the crashed thread's test frame".
  • The cited failure (cmux-next sidebar: account chip and Search on one line above the bottom margin; profile picture in the avatar #23076, job 119082777090, a Linux Blacksmith runner inside run-checks-concurrently.sh) has a different cause. Right after the stub output, its log shows:
    awk: fatal: cannot open file '/tmp/swift-test-log.RQ7ALW' for reading: No such file or directory
    sed: can't read /tmp/swift-test-log.RQ7ALW: No such file or directory
    The wrapper's own log was gone before report and the signal sed ran. So crash_signal came out empty and no crash line was printed. The wrapper does not delete the log early: on bash 5.2 its EXIT trap does not run in the watcher or in the process-substitution subshells. The likely cause is something else in the same run deleting files in the shared /tmp. I did not find which check does it.

Requested:

  1. Fix the PR body and commit message: this fixes the whole-second mtime race, not the job 119082777090 flake.
  2. Fix the real flake, or open a follow-up for it. One option is to keep the wrapper's log in a private mktemp -d directory. Another is to have the test set TMPDIR="$TMP" for the wrapper so its log lives inside the test's own temp dir. Either way, find what in the concurrent checks deletes /tmp entries.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-8bf5j created for this PR by the cmux coordinator.

On PR #23076's Linux CI (job 119082777090) the sampler's
/tmp/swift-test-log.* was gone before its report step read it, so the
crash case printed no crash ::error line. The test now points TMPDIR at
its own temp dir, so nothing else running beside it can touch the log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d2c-88e4-7fc2-b58e-9a55abdaced8&host=cosmic-gray
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (claude-review-subagent): SHIP at 9093e36

  • The TMPDIR change is right. TMPDIR="$TMP/tmp" is a prefix assignment on the sampler command inside $(...), so it applies only to that run and leaks into nothing after it; export still covers only STUB_OUTPUT, STUB_STATUS and GITHUB_STEP_SUMMARY, as before. mkdir -p is idempotent across runs, and the existing trap 'rm -rf "$TMP"' EXIT removes $TMP/tmp too.
  • Nothing left in this path writes to the shared /tmp. The sampler's log and its $log.* watcher files come from ${TMPDIR:-/tmp}. The only other default, the crash copy-out dir (RUNNER_TEMP/TMPDIR), is reached only in the signal case, which sets CMUX_NEXT_CRASH_OUT="$TMP/crashes". ips-crash-summary.py makes no temp files. The test's own $TMP is still a private mktemp -d under /tmp, which the body's "shows only on CI over time" caveat covers.
  • The body is accurate: it describes both flakes, says the CI deletion's cause was not found, gives 86 of 87 local runs, and gives the frame message for the mtime race, which matches what that race produces. The cubic summary at the bottom is stale (bot-generated at c55857b).
  • Passed 3/3 on macOS (bash 3.2) and 3/3 on Linux (huge-blue, bash 5.2, the three files copied to a scratch dir). No swift-test-log.* was left in /tmp.

@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:low Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant