Repository navigation
Add deterministic iOS connectivity soak workloads - #12735
Conversation
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Repository: manaflow-ai/cmux/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (49)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
All contributors have signed the CLA ✍️ ✅ |
|
Commissioning update: 22 focused iOS tests passed on the fleet, plus 2 deferred-transport tests and 2 shell relay-policy tests. A 602-second basic run completed 58 terminal/RPC cycles but exposed missing route evidence in Settings. The soak now reads identity and path from the active native connection, including through the deferred transport wrapper. Stalled operations now produce a bounded failure. Matching tagged Mac/iOS pairs are being installed for another full run. The earlier Unicode burst stalled after the Mac printed its marker; this is retained as unclassified evidence pending the new run. Recurring service activation and Slack delivery are not yet complete. |
|
Live validation on 3f8ceec: basic PASS, 59 complete cycles over 601.70 seconds, final transaction passed, native relay path and connection identity verified, maximum cycle 3.35 seconds. Stress FAIL at unicode_output_burst after 35.17 seconds with one completed cycle; the 30-second deadline wrote the report and the runner cleaned up. iOS replay logs show repeated waiting_for_baseline followed by replay_followup_cap_reached. The Mac printed the marker, but the probe did not observe completion. This remains an unclassified app/probe interaction, and the full one-hour sequence is not validated. Evidence is retained under the two run IDs 20260916T194028Z-basic-2c8aca and 20260916T194241Z-stress-00fc26 on cmux-mac-mini. Temporary app instances and simulators are stopped; recurring monitoring remains disabled. |
|
Full-window retry completed on cmux-mac-mini, source 39d7669.
This retry exposed a script mistake: retrying a healthy connection keeps that connection, while the stress script demanded a changed connection ID. The corrected action disconnects while preserving the pairing, invokes normal retry, then requires a new connection and successful terminal use. A behavioral regression fails without the correction and passes with it. All 17 focused release-gate tests and 11 supervisor tests pass. The earlier 35-second Unicode timeout did not reproduce; its underlying cause remains unproven. The tested source includes main snapshot be7d9fa fetched at retry start. Main advanced during rebuilding and verification; this is not a claim about the later main tip. Slack delivery still returns missing_scope. Results remain queued, and recurring jobs are not installed. |
* Record per-operation iOS soak latencies * Fix soak latency test dictionary fixtures
* Measure real iOS UI readiness timings * Only report presented terminal frames * Measure workspace tap to detail latency * Keep UI probe reset scoped to runner
|
Verified macOS fleet artifact for aeb554a: pr-12735-aeb554ab. HQ restores/downloads this exact artifact on click. Job This proves a macOS app build and publication; it does not prove iOS, tests, or UI behavior. Fetch the durable receipt with |
|
Mac fleet instructions for head JOB_JSON=$(~/.local/bin/cmux-ci submit --kind cmux --command 'CMUX_FLEET_BUILD_TAG=pr-12735-aeb554ab /Users/Shared/cmux-build-fleet/recipes/cmux.sh https://github.com/manaflow-ai/cmux.git aeb554ab0096d8e85e22c0e8d731316fea8d25d2' --artifact artifacts/cmux.app.zip --workspace https://github.com/manaflow-ai/cmux/pull/12735 --source-digest aeb554ab0096d8e85e22c0e8d731316fea8d25d2 --cache-key cmux:pr-12735 --min-free-bytes 268435456000 --label cmux --label ram48)
JOB_ID=$(python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])' <<<"$JOB_JSON")
~/.local/bin/cmux-ci wait "$JOB_ID" --receipt artifacts/fleet/$JOB_ID.json
~/.local/bin/cmux-ci publish-hq "$JOB_ID"Use an existing campaign job ID if one is already posted; do not submit a duplicate. A wait timeout leaves the remote job running. Published results will include an exact-head artifact link and timing/disk receipt. This recipe validates the macOS app only, not iOS or tests. Never use maclease or put credentials in a PR comment. |
Repair real iOS UI timing and soak terminal ownership
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| break | ||
| } | ||
| let owner = UUID() | ||
| let stream = client.terminalOutputStream(surfaceID: surfaceID, ownerID: owner) |
There was a problem hiding this comment.
The soak opens a second output stream for the foreground surface, but terminal output supports only one registration per surface. Registering this stream replaces the mounted Ghostty renderer’s continuation, token, owner, and delivery queue. The renderer then detects that it lost ownership and exits, while the soak consumer drains and acknowledges the output instead. As a result, the workload can pass even though the mounted renderer stopped receiving terminal output, so it does not provide the claimed sustained renderer-path coverage.
| settleReadiness: { | ||
| try await ContinuousClock().sleep(for: .milliseconds(500)) | ||
| }, |
There was a problem hiding this comment.
This uses a fixed 500 ms sleep as readiness synchronization before taking another state snapshot. The delay does not prove continuous readiness: a disconnect and recovery within that interval remains invisible. This violates the repository directive requiring an owning-subsystem signal or cancellation-aware readiness abstraction instead of fixed sleeps for synchronization, so the requirement must be satisfied before merging.
Rule Used: Flag new blocking or timing-based synchronization in production Swift: semaphores, DispatchGroup.wait, sleeps, Task.sleep, asyncAfter, timers or polling for synchronization, DispatchQueue.main.sync, or manual locks where actor isolation or a real sig... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
34ecef4 perf: coalesce concurrent process snapshots across diagnostics and restore (manaflow-ai#13014) 150d7fa Add app-host test failure census (manaflow-ai#13124) 04ac7a4 Merge pull request manaflow-ai#12735 from manaflow-ai/feat-ios-connectivity-soak c022438 fix: update Ghostty environment lifetime fix (manaflow-ai#13191) eb18207 Fix mobile devices dashboard WebSocket failures and naming (manaflow-ai#13156) c7d961d perf: split BrowserPanelView's modifier chain so it type-checks quickly (manaflow-ai#13130) e188035 refactor: move the Computer Use runtime out of the app module into a package (manaflow-ai#13132) cf2b850 Bound terminal markers and verify restored selection 2d7fc1c Measure terminal latency separately after reconnect 44dcd1e Reconcile iOS monitor stack with main 70034bc Merge pull request manaflow-ai#13116 from manaflow-ai/feat-ios-monitor-e2e-repair deafe6e Skip release gate text scans without a probe a1be46b Release terminal ownership from reader teardown 96d6574 Restore transport target after UI evidence 31ff358 Schedule terminal owner cleanup from deinit e82f6f6 Keep bounded terminal text evidence reliable ccbc4b2 Bound frame evidence scans and handshake setup 301dbad Make terminal evidence capture causal 6b50e3a Finish bounded release gate cleanup 14f2cd9 Stop stale release gate probes and bound frame inspection cd64e8c Bound pairing bootstrap loading 035fd37 Harden release gate evidence and readiness b6ca6e9 Close release gate review races 389026e Make release gate readiness and dismissal causal aa2bb02 Restore main translations for the pairing preparation error 46bbfc9 Restore the pairing preparation handling already present on main 1bcbf60 Give the soak one owned terminal reader across steady-state commands 9d700f9 Test soak terminal consumer lifetime across commands and reconnects 1dd9f69 Fix existing Cloud test imports and nested macro compilation bd5fee0 Give launch-request samples a distinct statistics key bc68fbf Measure UI readiness from the actual simulator launch request e5772e8 Test launch request timing across app initialization 8006ba9 Clear prior UI evidence before each retained-simulator launch d8e1b4c Reuse isolated monitor devices while cold-launching the app 0968fcf Test the dedicated monitor simulator plan boundary e7da214 Wait for the published pairing identity and inject screenshot capture 0af38fe Avoid the Swift task-group isolation checker defect in refresh test 48a0eb9 Own UI measurements per launch and capture composited terminal evidence 7e28e4e Mint pairing tickets with the active v2 device identity f498caf Test pairing tickets against the current transport identity 95f931d Correct the foreground suspension entrypoint in the test f5d5b2b Use the public foreground lifecycle for regression-test cleanup 0eded5a End the UI exercise only after terminal consumer ownership is released c0a61bb Keep UI state on its actor across the task-group boundary 30000a5 Drive and measure the real workspace UI before each soak; decouple background discovery 0d896f5 test: foreground refresh must finish while secondary discovery is blocked 4e739bb test: require real UI selection and stable first-frame measurements aeb554a Measure real iOS UI readiness timings (manaflow-ai#12887) 6a2c896 Record per-operation iOS soak latencies (manaflow-ai#12883) 39d7669 test: advertise workspace actions in the soak reconnect fixture 7e9a676 fix: disconnect the soak session before testing reconnect c51a096 test: require stress reconnect to replace a healthy connection e1a2767 fix: import the workspace model from its owning module e6a6ba4 fix: import mobile workspace preview module 3361141 Merge remote-tracking branch 'origin/main' into feat-ios-connectivity-soak dbfebac Merge remote-tracking branch 'origin/main' into feat-ios-connectivity-soak 3f8ceec fix: forward connection snapshots through deferred Iroh transport d8f30dc test: require deferred transports to forward native path snapshots 3dce84c fix: observe soak path and identity on the native RPC connection 38ee330 test: require native connection path evidence throughout soak 055880c test: cover final soak deadline and name failed usage actions b27046c fix: bound stalled soak cycles with an independent deadline 7f1f748 test: require stalled soak operations to report promptly e116e6b Use accepted boolean spelling for the Mac relay setting aa3dc13 Exercise relay setup command arguments in both modes db17d3d Constrain current Iroh endpoints to relays in app gates 245b55e Reproduce release gate missing current Iroh relay policy c9289ed Add focused Iroh soak harness test plan aec8291 Support an isolated agent account for unattended soaks 915c080 Add deterministic iOS Iroh connectivity soak workloads # Conflicts: # .github/workflows/ci.yml # .github/workflows/iroh-v2.yml
2 participants
Adds deterministic ten-minute and one-hour connectivity workloads inside the real iOS app. Each cycle verifies authenticated Iroh, terminal input and returned output, workspace mutations, events and related mobile RPCs. The stress sequence also exercises navigation, output bursts, scratch workspace creation/closure and deliberate reconnects.
Unexpected connection replacement, slow cycles, missing actions and shortened observation windows fail the run. Results contain workload version, action counts and the last operation. The gate supports an isolated agent account and captures an iOS screenshot alongside its transport report. Contributor instructions and the PR template require maintaining this coverage when the relevant product behavior changes.
Validation: shell syntax, argument validation and whitespace checks passed. Tagged Mac and isolated Simulator builds are running on the fleet; full-duration validation is pending. Added behavior tests for final-transaction requirements, unexpected reconnects, insufficient coverage and failed terminal operations.
Simulator coverage does not prove physical-device radios, background suspension or pixel correctness. Runtime activation is Debug-only and requires an explicit soak profile.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by cubic
Turns the iOS release gate from a one-shot connectivity probe into deterministic 10-minute basic or 60-minute stress soaks in the real iOS app. Soak runs now fail on unexpected connection replacement, invalid native path, stalled operations, incomplete coverage, or a shortened observation window.
Workloads
Gate support
iroh-soak.xctestplanharness and focused tests, including one asserting a stress reconnect replaces a healthy connection.scripts/run-iroh-release-gate.shgains--soak-profile, isolated staging--credentials-file, and--simulator-id, and can reuse a dedicated simulator across cold launches.Written for commit cf2b850. Summary will update on new commits.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation