Skip to content

Fix iPad-simulator flake in renderGridTerminalInputWaitsForLiveEventBeforeReplay (#5911) - #6330

Closed
austinywang wants to merge 4 commits into
mainfrom
issue-5911
Closed

austinywang wants to merge 4 commits into
mainfrom
issue-5911

Conversation

@austinywang

@austinywang austinywang commented Jun 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #5911

Root cause

cmuxFeatureTests' terminal-output collector tests waited for stream chunks to be delivered using a fixed 200ms polling budget (for _ in 0..<200 where collector.lines.count < N { try await Task.sleep(nanoseconds: 1_000_000) }). The renderGridTerminalInputWaitsForLiveEventBeforeReplay() test expects two chunks — the replayed "old" snapshot followed by the live "current" render-grid event. On the slower iPad simulator leg, the second (live) chunk's response flows back through the transport after the 200ms window closes, so the test proceeds with only ["old"] collected and the equality assertion fails.

This flake was previously masked by the test-ios false-green override (being removed in #5906); once that lands it would fail iOS PR runs at roughly the observed rate (3/13 iPad legs in the June 9–11 sample, 0/15 iPhone legs).

Fix

Replace the time-budget poll with a deterministic synchronization point on TerminalOutputCollector:

  • waitForLines(_ count:) parks a CheckedContinuation and is resumed the instant the stream delivers the required number of chunks (resumeSatisfiedWaiters() is called from the stream-consuming task right after each append). No per-platform timing budget for a slow simulator to overrun.
  • unmount() releases any still-parked waiter so a pending wait can never hang.
  • Applied to all three sibling tests that shared the identical fragile poll pattern, so the same latent flake can't resurface elsewhere once the override is gone.

Tradeoff

Test-only change; no production code touched. The wait is unbounded by design (resumes immediately on delivery); a genuinely missing chunk would surface as the job-level test timeout rather than a fixed-window assertion mismatch — an honest signal instead of a flaky one.


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.


Summary by cubic

Deflakes the iPad-simulator run of renderGridTerminalInputWaitsForLiveEventBeforeReplay by replacing the 200ms poll with a deterministic, continuation-based wait for stream chunks (fixes #5911). Adds waitForLines(...) with a 10s watchdog and ensures unmount() releases parked waiters; scoped to this test only, keeping sibling tests on their bounded polls to preserve coverage.

Written for commit 1da3eca. Summary will update on new commits.

Review in cubic

@vercel

vercel Bot commented Jun 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
cmux Ready Ready Preview, Comment Jun 17, 2026 10:04pm
cmux-staging Building Building Preview, Comment Jun 17, 2026 10:04pm

@coderabbitai

coderabbitai Bot commented Jun 17, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

@austinywang, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 2 minutes and 33 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 424b8ddf-6410-4cab-bf25-8dc2b7a58c6a

📥 Commits

Reviewing files that changed from the base of the PR and between 15915ed and 1da3eca.

📒 Files selected for processing (1)
  • ios/cmuxPackage/Tests/cmuxFeatureTests/cmuxFeatureTests.swift
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue-5911

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@greptile-apps

greptile-apps Bot commented Jun 17, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces a waitForLines(_ count:) continuation-based synchronization method on TerminalOutputCollector to replace a fragile 200 ms time-budget poll, fixing the iPad-simulator flake in renderGridTerminalInputWaitsForLiveEventBeforeReplay. No production code is touched.

  • waitForLines parks a CheckedContinuation and resumes it the instant the stream delivers the required number of chunks, with a 10-second watchdog that lets a genuinely missing chunk surface as a fast assertion failure rather than a CI-level timeout.
  • unmount() is updated to release any parked waiters so a pending call can never hang after teardown.
  • The fix is applied only to renderGridTerminalInputWaitsForLiveEventBeforeReplay; the two sibling tests (terminalInputResyncsOutputWhenMacSequenceIsAhead and terminalRenderGridEventsDriveMountedSink) still contain the original for _ in 0..<200 polling loops, contrary to the PR description's claim that all three were updated.

Confidence Score: 3/5

The infrastructure added is correct, but the fix was only applied to one of the three tests the PR explicitly targeted; the other two still have the polling loops that caused the original flake.

The waitForLines implementation is well-designed and the target test is correctly fixed, but terminalInputResyncsOutputWhenMacSequenceIsAhead and terminalRenderGridEventsDriveMountedSink still poll with the original 200 ms budget. The PR description explicitly states all three were converted, making this an incomplete fix — once the test-ios false-green override referenced in #5906 lands, those two tests will resume flaking on iPad under the same conditions described in the root-cause analysis.

ios/cmuxPackage/Tests/cmuxFeatureTests/cmuxFeatureTests.swift — lines 2448, 2456, and 2550 still contain the fragile polling loops in the two unfixed sibling tests.

Important Files Changed

Filename Overview
ios/cmuxPackage/Tests/cmuxFeatureTests/cmuxFeatureTests.swift Adds deterministic waitForLines continuation infrastructure to TerminalOutputCollector and applies it to renderGridTerminalInputWaitsForLiveEventBeforeReplay, but leaves the same fragile 200ms polling loops in terminalInputResyncsOutputWhenMacSequenceIsAhead and terminalRenderGridEventsDriveMountedSink — the two other sibling tests the PR description says were also fixed.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Test
    participant Collector as TerminalOutputCollector
    participant Stream as terminalOutputStream
    participant Watchdog as Watchdog Task

    Test->>Collector: waitForLines(2)
    Collector->>Collector: "park CheckedContinuation (id=0)"
    Collector->>Watchdog: spawn sleep(10s) then resumeWaiter(0)
    Stream-->>Collector: chunk 1 append + resumeSatisfiedWaiters
    Note over Collector: count=1 < 2 waiter stays parked
    Stream-->>Collector: chunk 2 append + resumeSatisfiedWaiters
    Note over Collector: count=2 >= 2 continuation.resume()
    Collector->>Watchdog: watchdog.cancel()
    Collector-->>Test: waitForLines returns
    Test->>Test: assert collector.lines
    Test->>Collector: unmount()
    Collector->>Collector: cancel task and release parked waiters
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Test
    participant Collector as TerminalOutputCollector
    participant Stream as terminalOutputStream
    participant Watchdog as Watchdog Task

    Test->>Collector: waitForLines(2)
    Collector->>Collector: "park CheckedContinuation (id=0)"
    Collector->>Watchdog: spawn sleep(10s) then resumeWaiter(0)
    Stream-->>Collector: chunk 1 append + resumeSatisfiedWaiters
    Note over Collector: count=1 < 2 waiter stays parked
    Stream-->>Collector: chunk 2 append + resumeSatisfiedWaiters
    Note over Collector: count=2 >= 2 continuation.resume()
    Collector->>Watchdog: watchdog.cancel()
    Collector-->>Test: waitForLines returns
    Test->>Test: assert collector.lines
    Test->>Collector: unmount()
    Collector->>Collector: cancel task and release parked waiters
Loading

Reviews (4): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile

Comment thread Sources/PaneMemoryGuardrail.swift Outdated
Comment on lines +151 to +163
static func terminate(processGroupIDs: [Int], graceSeconds: TimeInterval = 3) {
let pgids = processGroupIDs.filter { $0 > 1 }
guard !pgids.isEmpty else { return }
for pgid in pgids {
_ = kill(pid_t(-pgid), SIGTERM)
}
DispatchQueue.global(qos: .userInitiated).asyncAfter(deadline: .now() + graceSeconds) {
for pgid in pgids {
_ = kill(pid_t(-pgid), SIGKILL)
}
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 DispatchQueue.global(qos:).asyncAfter for SIGKILL scheduling

PaneMemoryProcessKiller.terminate uses DispatchQueue.global(qos: .userInitiated).asyncAfter to delay the SIGKILL 3 seconds after SIGTERM. This is new production Swift using both DispatchQueue.global and asyncAfter, both explicitly flagged by the cmux-swift-blocking-runtime and cmux-swift-concurrency-modernization rules. Beyond the pattern violation, the fire-and-forget dispatch has no cancellation path: if the pane is closed or clearAll() is called before the 3-second window expires, the SIGKILL to the old process group IDs still fires. Rapid double-invocations of killActivePaneProcess() also stack unrelated pending SIGKILL closures with no way to discard the earlier one. The Swift-idiomatic replacement is a stored Task.detached { try? await Task.sleep(for: .seconds(graceSeconds)); for pgid in pgids { kill(pid_t(-pgid), SIGKILL) } } that can be cancelled.

Rule Used: Flag new blocking or timing-based synchronization ... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment thread Sources/PaneMemoryGuardrail.swift Outdated
Comment on lines +171 to +185
@MainActor
final class PaneMemoryGuardrail: ObservableObject {
static let shared = PaneMemoryGuardrail()

enum DefaultsKeys {
static let enabled = "terminal.runawayMemoryGuardrail.enabled"
static let thresholdGB = "terminal.runawayMemoryGuardrail.thresholdGB"
}

private static let pollInterval: TimeInterval = 4
private static let defaultThresholdGB: Double = 8
private static let minThresholdGB: Double = 1

/// The banner content for the most recent un-dismissed crossing, or nil.
@Published private(set) var activeBanner: PaneMemoryWarning?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 PaneMemoryGuardrail is a new production class with a single @Published observable property — exactly the pattern cmux-swiftui-state-layout flags as needing @Observable. With @Observable, PaneMemoryGuardrailBannerView replaces @ObservedObject var guardrail with a plain stored property, and SwiftUI tracks only activeBanner reads rather than invalidating on every @Published change.

Suggested change
@MainActor
final class PaneMemoryGuardrail: ObservableObject {
static let shared = PaneMemoryGuardrail()
enum DefaultsKeys {
static let enabled = "terminal.runawayMemoryGuardrail.enabled"
static let thresholdGB = "terminal.runawayMemoryGuardrail.thresholdGB"
}
private static let pollInterval: TimeInterval = 4
private static let defaultThresholdGB: Double = 8
private static let minThresholdGB: Double = 1
/// The banner content for the most recent un-dismissed crossing, or nil.
@Published private(set) var activeBanner: PaneMemoryWarning?
@MainActor
@Observable
final class PaneMemoryGuardrail {
static let shared = PaneMemoryGuardrail()
enum DefaultsKeys {
static let enabled = "terminal.runawayMemoryGuardrail.enabled"
static let thresholdGB = "terminal.runawayMemoryGuardrail.thresholdGB"
}
private static let pollInterval: TimeInterval = 4
private static let defaultThresholdGB: Double = 8
private static let minThresholdGB: Double = 1
/// The banner content for the most recent un-dismissed crossing, or nil.
private(set) var activeBanner: PaneMemoryWarning?

Rule Used: Flag SwiftUI changes that can cause stale state, b... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment thread Sources/PaneMemoryGuardrail.swift Outdated
Comment on lines +195 to +249
private let queue = DispatchQueue(label: "com.cmux.pane-memory-guardrail", qos: .utility)
private var timer: DispatchSourceTimer?
private var isScanning = false
private var lastSamplesByKey: [PaneMemoryPaneKey: PaneMemorySample] = [:]
private var lastWarnedWorkspaceIds: Set<UUID> = []

func start() {
guard timer == nil else { return }
let timer = DispatchSource.makeTimerSource(queue: queue)
timer.schedule(
deadline: .now() + Self.pollInterval,
repeating: Self.pollInterval,
leeway: .seconds(1)
)
timer.setEventHandler { [weak self] in
Task { @MainActor in self?.tick() }
}
self.timer = timer
timer.resume()
}

// MARK: Settings

private var isEnabled: Bool {
UserDefaults.standard.object(forKey: DefaultsKeys.enabled) as? Bool ?? true
}

private func thresholdBytes() -> Int64 {
let configured = UserDefaults.standard.object(forKey: DefaultsKeys.thresholdGB) as? Double
?? Self.defaultThresholdGB
let gb = max(Self.minThresholdGB, configured)
return Int64(gb * 1024 * 1024 * 1024)
}

// MARK: Tick

private func tick() {
guard isEnabled else {
clearAll()
return
}
guard !isScanning, let paneProvider else { return }
let descriptors = paneProvider()
guard !descriptors.isEmpty else {
clearAll()
return
}
let thresholdBytes = thresholdBytes()
isScanning = true
queue.async { [weak self] in
let samples = Self.computeSamples(descriptors: descriptors)
Task { @MainActor in
self?.applySamples(samples, thresholdBytes: thresholdBytes)
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Mixed GCD/Swift concurrency in the scan loop

tick() dispatches onto a private DispatchQueue and creates a nested Task { @MainActor in ... } to bounce results back. start() uses DispatchSource.makeTimerSource(queue:) bridged into Swift concurrency via Task { @MainActor in self?.tick() }. Per cmux-swift-concurrency-modernization, DispatchQueue for background work is the legacy pattern: computeSamples is already nonisolated static, so the canonical shape is Task.detached { let samples = Self.computeSamples(descriptors:); await MainActor.run { self.applySamples(...) } } initiated from a Swift Clock-based timer rather than a GCD source.

Rule Used: Flag new legacy async patterns in cmux-owned Swift... (source)

…eforeReplay (#5911)

The terminal-output collector tests waited for stream chunks with a fixed
200ms polling budget (200×1ms). On the slower iPad simulator leg the live
"current" replay chunk lands after that window, so the assertion saw only the
"old" snapshot and the test failed — a flake previously masked by the test-ios
false-green override (#5906).

Replace the time-budget poll with a deterministic continuation-based
synchronization point on TerminalOutputCollector: waitForLines(_:) parks a
CheckedContinuation and resumes the instant the stream delivers the target
chunk count, with no per-platform timing budget to overrun. unmount() releases
any parked waiter so a pending wait never hangs. Applied to all three sibling
tests that shared the fragile poll pattern.

Closes #5911

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses autoreview: the continuation-based wait had no deadline, so a
genuinely dropped or miscounted output chunk would hang the Swift Testing run
until the global CI timeout. Add a generous (10s) watchdog that resumes the
parked continuation, letting the test fail on its assertion (showing the actual
collected lines) instead of hanging. The success path stays fully event-driven
and resumes the instant the chunk lands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
austinywang and others added 2 commits June 17, 2026 14:06
Addresses autoreview: revert the two sibling tests (selfHealing two-replay and
the render-grid mounted-sink) to their original bounded polls. The mounted-sink
test relied on the bounded poll as a settle window to verify the stale replay
frame is suppressed; switching it to waitForLines(1) asserted the instant the
first chunk landed and weakened that negative coverage. Only
renderGridTerminalInputWaitsForLiveEventBeforeReplay — the test the issue
reported as flaking — now uses the deterministic waitForLines helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lawrencecchen lawrencecchen added the stale-revisit Closed after 30+ days without activity; preserved for possible revisit or reopening. label Sep 23, 2026
@github-project-automation github-project-automation Bot moved this from Todo to Done in cmux backlog Sep 23, 2026

This branch was successfully deployed

1 active deployment
Preview – cmux — 1da3ecaa Deployed Jun 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale-revisit Closed after 30+ days without activity; preserved for possible revisit or reopening.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

iOS CI: renderGridTerminalInputWaitsForLiveEventBeforeReplay flakes on the iPad simulator leg, masked by the test-ios false-green override

3 participants