Skip to content

test(cmux-next): run SidebarDocStore-owning live-daemon suites one at a time - #27421

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
ib/classic-sim-isolation
Oct 29, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
ib/classic-sim-isolation

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Confound

AppActions.bind(services) always runs SidebarSectionHandlers.bind, which reads services.sidebarLayout. That lazy var's first read calls SidebarDocWiring.start, and start sets the process-wide static SidebarDocStore.current to that services' store. DocMachines and placement then read the static without checking which AppServices owns it.

These top-level suites each bind AppActions and run a live daemon across awaits:

  • ClassicUpgradeSimulationTests and ClassicUpgradeHugeSimulationTests, through ClassicUpgradeSimulator.init. They are the only callers of ClassicUpgradeSimulator on base.
  • ClassicImportTests, ClassicMigrationTests and FirstLaunchTests.

Each is .serialized only within itself. Swift Testing runs top-level suites in parallel, so a --filter Classic run runs them together. So does CI's daemon-test job (...|FirstLaunchTests|ClassicImportTests|ClassicMigrationTests|ClassicUpgradeSimulationTests). Two daemons end up writing one static. The later bind takes the store, and the earlier simulation places workspaces from the other run's sidebar document. That can explain some of the "order differs" failures. Saving and restoring the static around a test does not help, because both runs are mid-flight when the swap happens.

Fix (tests only)

A new suite trait, .ownsSidebarDocStore (SidebarDocOwnerGate, which follows LiveDaemonGate and WindowLeakCheck). It is non-recursive, and its scope is provided once for the suite itself. That scope holds a process-wide FIFO async mutex for the suite's whole run, which covers every test in the suite, including the huge simulation's phases that share one launch. The five suites above carry the trait. Suites without it keep running in parallel around them, so the process still carries other load.

I chose a trait over nesting the suites under a .serialized parent so that test IDs (the CI filter and per-test base-red matching use them) and file indentation don't change. #26056 edits lines inside ClassicUpgradeSimulationTests, and this PR changes only that file's @Suite line. If #26056 lands, its new ClassicUpgradeSimulationPerfTests also drives ClassicUpgradeSimulator, so it should carry .ownsSidebarDocStore too.

Production code (SidebarDocWiring, SidebarDocStore, DocMachines, placement), assertions and timings are unchanged. The production fix for the race is separate work.

Still exposed in full-package runs (not gated here)

97 test files call AppActions.bind, and every one of them writes SidebarDocStore.current the same way. In a full-package run any of them can still overlap a gated suite. This PR covers the suites the Classic filter and the daemon-test job schedule together. Gating every binder would serialize most of the package.

Measurement

Harness: cmux-ci run --class light --script scripts/ci/package-test-lane.sh --arg=suite --arg=Packages/macOS/CmuxNext with --arg=Classic ×10 and CMUX_CI_XCODE_APP=/Applications/Xcode_26.6.app, on one worker (--target-worker), base then head back to back.

Base a1951db = step c3c0cbc10e9976bffc67c634, head 8081190 = step 60afea902560f174180f206b, same worker, back to back. An earlier base step 7167b321dbf489be0e78bdc9 on a different worker is in the last column for reference.

metric before after delta step (before / after) earlier base
test runs (. Test run with) 10 10 0 c3c0cbc1 / 60afea90 10
runs that failed (✘ Test run with) 10 3 −7 c3c0cbc1 / 60afea90 10
order differs: huge 115 0 −115 c3c0cbc1 / 60afea90 116
order differs: manyGroups 47 14 −33 c3c0cbc1 / 60afea90 78
order differs: small 8 0 −8 c3c0cbc1 / 60afea90 0
order differs: nestedSplits 2 0 −2 c3c0cbc1 / 60afea90 0
is not parked 0 0 0 c3c0cbc1 / 60afea90 0
issues at ClassicUpgradeHugeSimulationTests.swift:70 119 6 −113 c3c0cbc1 / 60afea90 116
issues at ClassicUpgradeHugeSimulationTests.swift:68 2 3 +1 c3c0cbc1 / 60afea90 0
issues at ClassicUpgradeSimulationTests.swift:56 23 0 −23 c3c0cbc1 / 60afea90 47
issues at ClassicUpgradeSimulationTests.swift:49 22 0 −22 c3c0cbc1 / 60afea90 18
most gated suites running a test at once (each of 10 runs) 4 1 −3 c3c0cbc1 / 60afea90 4

The last row is computed from the log's ◇ Test … started and ✔/✘/━ Test … after lines, mapping each test function to its suite. ◇ Suite … started is printed before a suite's scope trait runs, so all four suites still print "started" together. In the head log the suite durations add up one after another instead (for example Import 12.5 s, then Huge ending at 346 s, Migration at 351 s, and the simulations at 412 s).

In the head runs the 14 manyGroups "order differs" lines and the Huge :68/:70 issues remain. That fits the production race (DocMachines reading the static without checking its owner) and not the test overlap.

The isolation fix is not expected to reach zero, because the production race remains. Its purpose is a measurement where the Classic suites no longer share one sidebar document.

Agent session: agent session

🤖 Generated with Claude Code

… a time

AppActions.bind reads services.sidebarLayout, whose first read sets the
process-wide SidebarDocStore.current. The classic upgrade simulations,
ClassicImportTests, ClassicMigrationTests and FirstLaunchTests are separate
top-level .serialized suites, so a Classic filter run or the daemon-test job
ran them at once against that one store. A suite trait (.ownsSidebarDocStore)
holds a process-wide FIFO mutex for each gated suite's whole run; ungated
suites still run in parallel around them. No assertion or timing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cc8bff1c3ddb4ec353c5
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 8081190: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27421 -f ref=ib/classic-sim-isolation -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base c14dc66; does not block

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-09qdts created for this PR by the cmux coordinator.

@github-actions github-actions Bot added the risk:low Risk tier (docs/ci/risk-tiers.md) label Oct 29, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 29, 2026

Copy link
Copy Markdown

Found 6 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmpf5xtvo0a/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpwcls4a0e/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

More evidence of the ClassicUpgradeSimulationTests order flake this PR targets, from the classification of #27052 (which touches no upgrade or archive code):

  • The full daemon-test CI job on cmux NEXT: agent set-model --target pane:|tab: runs with no chat focused #27052 failed bothDamagedFallsBackToTheNewestArchive (:119) and aDowngradeAndReUpgradeMergesOnlyNewWork (:188).
  • Run sequentially, both pass 2/2 at the PR head (fleet steps H1 517a0811, H2 ada2c260) and 2/2 at base 3b3ea3b (B1 d292faf7, B2 d35a1c29).
  • Base step B1 failed a different test, theFirstLaunchBringsEverythingOnce (ClassicUpgradeSimulationTests.swift:49/:56). So the failing test moves with ordering, on base too.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

For awareness, from #26809's red triage: at feat-cmux-next 52b0452, --filter Classic with SWT_EXPERIMENTAL_MAXIMUM_PARALLELIZATION_WIDTH=1 (one test at a time, so no two Classic suites overlap) still fails the "order differs" check:

  • theFirstLaunchBringsEverythingOnce: multiWindow (["w1 3", "w1 2", "w1 1", "w1 0"] for ["w1 0", "w1 1", "w1 2", "w1 3"]) and oldSchema, each at :49 and :56. The cases run one after another in that log.
  • onlyThePreviousCopyLeftStillMigrates (:133), aDamagedSessionFallsBackToItsPreviousCopy (:98), bothDamagedFallsBackToTheNewestArchive (:119), anOffSSHHostDoesNotHoldBackLocalWorkspaces (:243).

Fleet step b50ae2ae6894c55e50dfbb14 (width 1); at width 6, step d2a516728beecc2e8b5c9f80 had 18 order-differs lines against 9. So serializing the suites roughly halves it, and the rest happens with no overlap at all. That fits a SidebarDocStore.current left over from an earlier test's AppServices (read before this test's own sidebar takes it), so a suite gate can't reach zero. It also matches the manyGroups lines left in your head runs.

Agent session: agent session

🤖 Generated with Claude Code

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): SHIP at 8081190

No blocking findings.

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 8081190: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@teamleaderleo
teamleaderleo merged commit f138d12 into feat-cmux-next Oct 29, 2026
262 of 283 checks passed
@teamleaderleo
teamleaderleo deleted the ib/classic-sim-isolation branch October 29, 2026 23:04
@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 80811907 Deployed Oct 29, 2026 by teamleaderleo via namespace autofix #6235
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:low Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant