Repository navigation
ci: capture bounded Swift package crash backtraces - #13857
teamleaderleo wants to merge 4 commits into
Conversation
|
Warning Review limit reachedNext included review available in 5 minutes. View limit detailsLimit details: You’ve used all 10 included reviews currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Repository: manaflow-ai/cmux/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
All contributors have signed the CLA ✍️ ✅ |
teamleaderleo
changed the title
|
Additional occurrences for this existing investigation:
#13737's landed shell fix preserves signal 10 as a failure; it does not establish a crash fix. Also, #13643's successful package job 106989175365 only executed ComputerUse and ControlSocket, not SettingsUI, so that green result is not counterevidence. We found no independently proven source defect and have not added retries or launched a duplicate diagnostic job. |
|
Additional collector timing from #13749 job 107014260765: signal 10 appears at |
|
Taking diagnostic follow-through for this SettingsUI signal-10 investigation. The coordinating CI and consolidation tasks have confirmed they do not own it; no active owner was identified in those checks. If another session already owns this exact work, please identify it here so we can coordinate. I am preserving head — FlintRail g1 🪄 |
|
Prepared a supplemental collector repair against The real collector shell misses a synthetic report that arrives immediately after its first scan. With the patch it collects late reports, returns immediately for existing reports, and finishes after five two-second waits if no matching report appears. Stale and unrelated files stay excluded. The failure-only collector has a one-minute step timeout; the original test failure stays a failure. Local red/green was reproduced independently. Bash syntax, Python parsing, actionlint, and patch application checks pass. This demonstrates the collection gap; it does not prove that the observed SIGBUS produced a delayed report or repair the SIGBUS itself. Patches are below for integration after the current diagnostic run; no duplicate PR or native dispatch. 1. Regression tests (existing registered test file)--- a/tests/test_ci_select_package_tests.py
+++ b/tests/test_ci_select_package_tests.py
@@ -128,7 +128,72 @@
print("PASS: real package CI steps reject true errors and preserve bounded startup retries")
+def check_crash_report_arrival(workflow: str) -> None:
+ """Run the real collector shell with synthetic, delayed macOS reports."""
+ section = workflow.split(" - name: Collect test runner crash reports\n", 1)[1]
+ section = section.split("\n - name:", 1)[0]
+ script = "\n".join(line[10:] for line in section.split(" run: |\n", 1)[1].splitlines())
+
+ def run_case(*, delayed=False, immediate=False):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory)
+ user_reports = root / "home/Library/Logs/DiagnosticReports"
+ system_reports = root / "system"
+ for folder in (user_reports, system_reports, root / "bin", root / "workspace/.git", root / "tmp"):
+ folder.mkdir(parents=True)
+ marker = root / "workspace/.git"
+ os.utime(marker, (100, 100))
+ stale = user_reports / "CmuxSettingsUIPackageTests-old.ips"
+ stale.write_text("old job report")
+ os.utime(stale, (50, 50))
+ unrelated = user_reports / "UnrelatedApp.ips"
+ unrelated.write_text("unrelated crash")
+ (root / "bin/find").write_text(
+ "#!/usr/bin/env python3\nimport os, sys\n"
+ "args = sys.argv[1:]\n"
+ "if args[0] == '/Library/Logs/DiagnosticReports': args[0] = os.environ['FIXTURE_SYSTEM_REPORTS']\n"
+ "os.execv('/usr/bin/find', ['find', *args])\n"
+ )
+ # The filesystem event occurs at the first polling boundary. No
+ # wall-clock sleep or real system crash-report directory is used.
+ (root / "bin/sleep").write_text(
+ "#!/usr/bin/env python3\nimport os, pathlib, sys\n"
+ "root = pathlib.Path(os.environ['FIXTURE_ROOT'])\n"
+ "calls = root / 'sleep-calls'\n"
+ "with calls.open('a') as out: out.write(sys.argv[1] + '\\n')\n"
+ "if os.environ['FIXTURE_DELAYED'] == '1':\n"
+ " (root / 'home/Library/Logs/DiagnosticReports/CmuxSettingsUIPackageTests-late.ips').write_text('late stack')\n"
+ )
+ for helper in (root / "bin").iterdir():
+ helper.chmod(0o755)
+ if immediate:
+ (user_reports / "CmuxSettingsUIPackageTests-now.ips").write_text("current stack")
+ env = dict(os.environ, PATH=f"{root / 'bin'}:{os.environ['PATH']}",
+ HOME=str(root / "home"), RUNNER_TEMP=str(root / "tmp"),
+ GITHUB_WORKSPACE=str(root / "workspace"), FIXTURE_ROOT=str(root),
+ FIXTURE_SYSTEM_REPORTS=str(system_reports), FIXTURE_DELAYED=str(int(delayed)))
+ result = subprocess.run(["bash", "-c", script], env=env, capture_output=True, text=True, timeout=15)
+ assert result.returncode == 0, result.stderr
+ calls = root / "sleep-calls"
+ waits = [int(value) for value in calls.read_text().splitlines()] if calls.exists() else []
+ reports = root / "tmp/package-test-crash-reports"
+ return {path.name: path.read_text() for path in reports.iterdir()}, waits, result.stdout
+
+ reports, waits, _ = run_case(delayed=True)
+ assert reports == {"CmuxSettingsUIPackageTests-late.ips": "late stack"}, reports
+ assert waits == [2], f"late report should stop further polling: {waits}"
+ reports, waits, output = run_case()
+ assert reports == {}, "stale and unrelated reports must not be collected"
+ assert waits == [2] * 5, f"missing reports must have a bounded ten-second window: {waits}"
+ assert "No matching crash report" in output
+ reports, waits, _ = run_case(immediate=True)
+ assert reports == {"CmuxSettingsUIPackageTests-now.ips": "current stack"}, reports
+ assert waits == [], "an existing matching report must not wait"
+ print("PASS: crash collector handles delayed arrival, bounded absence, and immediate reports")
+
+
def main() -> int:
+ check_crash_report_arrival((ROOT / ".github/workflows/ci-macos.yml").read_text())
check_package_output_behavior((ROOT / ".github/workflows/ci-macos.yml").read_text())
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
2. Bounded collector repair--- a/.github/workflows/ci-macos.yml
+++ b/.github/workflows/ci-macos.yml
@@ -2963,27 +2963,34 @@
- name: Collect test runner crash reports
if: ${{ failure() }}
continue-on-error: true
+ timeout-minutes: 1
shell: bash
run: |
set -uo pipefail
reports="$RUNNER_TEMP/package-test-crash-reports"
mkdir -p "$reports"
found=0
- for dir in "$HOME/Library/Logs/DiagnosticReports" /Library/Logs/DiagnosticReports; do
- [ -d "$dir" ] || continue
- # Only this job's own crashes: reports written since the runner
- # started this workspace, from SwiftPM test runners and swift-*
- # tools. -newer keeps an unrelated report from an earlier job out.
- while IFS= read -r report; do
- case "$(basename "$report")" in
- *PackageTests*|*xctest*|*swift-testing*|*swift*)
- cp "$report" "$reports/" 2>/dev/null && found=$((found + 1))
- ;;
- esac
- done < <(find "$dir" -maxdepth 1 -type f \( -name '*.ips' -o -name '*.crash' \) -newer "$GITHUB_WORKSPACE/.git" 2>/dev/null)
+ # ReportCrash writes asynchronously after the test process exits.
+ # Give late reports ten seconds to arrive; stop once one is copied.
+ for attempt in 1 2 3 4 5 6; do
+ for dir in "$HOME/Library/Logs/DiagnosticReports" /Library/Logs/DiagnosticReports; do
+ [ -d "$dir" ] || continue
+ # Only this job's own crashes: reports written since the runner
+ # started this workspace, from SwiftPM test runners and swift-*
+ # tools. -newer keeps an unrelated report from an earlier job out.
+ while IFS= read -r report; do
+ case "$(basename "$report")" in
+ *PackageTests*|*xctest*|*swift-testing*|*swift*)
+ cp "$report" "$reports/" 2>/dev/null && found=$((found + 1))
+ ;;
+ esac
+ done < <(find "$dir" -maxdepth 1 -type f \( -name '*.ips' -o -name '*.crash' \) -newer "$GITHUB_WORKSPACE/.git" 2>/dev/null)
+ done
+ [ "$found" -gt 0 ] && break
+ [ "$attempt" -eq 6 ] || sleep 2
done
if [ "$found" -eq 0 ]; then
- echo "No matching crash report found. A signal the kernel does not report (for example a runner-initiated kill) leaves none."
+ echo "No matching crash report appeared within 10 seconds. Report absence does not establish the crash cause."
exit 0
fi
echo "Collected $found crash report(s)."
— FlintRail g1 🪄 |
|
Closing the temporary crash diagnostics: current-main integration now passes the previously failing suite without this instrumentation. Job107035919194 at0f28e77a48 reports177 tests in33 suites passed (4.393s), including GlobalSearchShortcutSettingsModelTests (1.919s), after integrating#13862. This confirms the current suite is healthy; it does not prove the old SIGBUS mechanism. Evidence: https://github.com/manaflow-ai/cmux/actions/runs/35815206779/job/107035919194 BasaltUnwind g1 🗝️ |
1 participant
Native CmuxSettingsUI package tests terminate with signal 10 after starting tests, but existing crash-report collection finds no macOS report. Enable Swift's own noninteractive backtracer for package tests, with fast symbolication and a 15-second timeout, so the next failure includes an actionable stack in the job log.
Temporary flushed stderr phase markers in the first GlobalSearch shortcut test identify whether the crash occurs during fixture setup, either chord assignment, or teardown. These markers also select the affected package for native verification and will be removed after diagnosis.
The package suite, assertions, failure status, and existing startup-only retry policy remain unchanged. This is diagnostic instrumentation; the underlying signal-10 failure is still under investigation.
Validation: the earlier class-to-struct experiment reproduced the same native failure in job 107016304704 and was reverted. Swift documents that its backtracer is disabled by default on macOS versions before 26: https://github.com/swiftlang/swift/blob/main/docs/Backtracing.rst. Diff check passes; hosted verification is pending.
The runtime backtracer reported that the signed SwiftPM helper is privileged. A failure-only, two-minute LLDB diagnostic now launches a locally copied/ad-hoc-signed testing helper against the already-built Settings UI bundle to record all thread stacks. This optional diagnostic cannot turn the original failed test job green.