Skip to content

Fix iOS CI false-green: align manual-pairing tests with encrypted-route restriction, catch Swift Testing failures in success override #5906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lawrencecchen merged 2 commits into from
Jun 11, 2026
Merged

Fix iOS CI false-green: align manual-pairing tests with encrypted-route restriction, catch Swift Testing failures in success override #5906

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aa5fdee
Align manual-pairing auth-contract tests with encrypted-route restric…
lawrencecchen Jun 11, 2026
c5c7860
Catch Swift Testing failures in the ios-simulator success override
lawrencecchen Jun 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .github/workflows/test-ios.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,9 +233,14 @@ jobs:
XCODEBUILD_ARGS+=(test)
selected_tests_passed_despite_xcodebuild_status() {
local log_path="$1"
# The negative grep must catch BOTH XCTest failure output and Swift
# Testing failure output (✘ markers): the final "Failing tests:"
# section is not always flushed into the tee'd log, and without the
# ✘ patterns a real Swift Testing failure exiting 65 was
# misclassified as a runner cleanup failure and turned the job green.
grep -Eq "Test Suite 'Selected tests' passed|Test Suite 'cmuxUITests' passed" "$log_path" &&
grep -Eq "Executed [1-9][0-9]* tests, with 0 failures \\(0 unexpected\\)" "$log_path" &&
! grep -Eq "Test Suite '.*' failed|Test Case '.*' failed|Assertion Failure|Failing tests:|with [1-9][0-9]* failures|with [0-9]+ failures \\([1-9][0-9]* unexpected\\)" "$log_path"
! grep -Eq "Test Suite '.*' failed|Test Case '.*' failed|Assertion Failure|Failing tests:|with [1-9][0-9]* failures|with [0-9]+ failures \\([1-9][0-9]* unexpected\\)|✘ Test|✘ Suite" "$log_path"

@greptile-apps greptile-apps Bot Jun 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Swift Testing crash / fatal-error failures may still slip through

✘ Test and ✘ Suite cover assertion failures and normal test failures, but Swift Testing can also terminate a test runner process via an uncaught thrown error or a fatal error, which in xcodebuild's log may surface as Fatal error:, error: ..., or just a non-zero exit without any ✘ marker. The existing Assertion Failure pattern catches one XCTest form; there is no equivalent for Swift Testing's Issue.record path or for a fatal thrown error outside a #expect. If any test in the suite crashes rather than fails gracefully, the negative grep could still pass and false-green the job. Worth adding a broader safety net (e.g. Issue recorded) once the exact log tokens are confirmed against a real crash log.

}
for attempt in 1 2; do
LOG_PATH="/tmp/cmux-ios-${{ matrix.family }}-attempt-${attempt}.log"
Expand Down
99 changes: 39 additions & 60 deletions ios/cmuxPackage/Tests/cmuxFeatureTests/cmuxFeatureTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -528,11 +528,11 @@ final class TerminalOutputCollector {
}

@MainActor
@Test func manualHostPairingUsesNetworkRouteForPrivateLANIPWithStackAuth() async throws {
let responses = ScriptedTransportResponses([
try rpcErrorFrame(code: "method_not_found", message: "unknown method"),
try rpcWorkspaceListFrame(workspaceID: "lan-workspace", title: "LAN Workspace"),
])
@Test func manualHostPairingRejectsPrivateLANIPWithoutSendingStackToken() async throws {
// Plain private-LAN routes are dialed over unencrypted TCP, so
// routeAllowsStackAuth excludes them: pairing must fail before any RPC
// (and the Stack bearer token) leaves the device.
let responses = ScriptedTransportResponses([])
let runtime = testRuntime(
supportedRouteKinds: [.tailscale],
transportFactory: ScriptedTransportFactory(responses: responses),
Expand All @@ -543,30 +543,20 @@ final class TerminalOutputCollector {
store.signIn()
await store.connectManualHost(name: "Studio LAN", host: " 192.168.1.77 ", port: 15432)

let route = try #require(store.activeRoute)
#expect(store.phase == .workspaces)
#expect(store.connectionState == .connected)
#expect(store.connectionError == nil)
#expect(store.connectedHostName == "Studio LAN")
if case let .hostPort(host, port) = route.endpoint {
#expect(host == "192.168.1.77")
#expect(port == 15432)
} else {
Issue.record("manual LAN route should use host/port")
}
#expect(route.kind == .tailscale)
let requests = try await responses.sentRequests()
#expect(requests.map(\.method) == ["mobile.attach_ticket.create", "workspace.list"])
#expect(requests.allSatisfy { $0.stackAccessToken == "stack-token-for-lan" })
#expect(requests.allSatisfy { $0.attachToken == nil })
#expect(store.phase == .pairing)
#expect(store.connectionState == .disconnected)
#expect(store.activeTicket == nil)
#expect(store.activeRoute == nil)
#expect(store.connectionError == "This pairing route is not allowed. Enter a host and port, or pair with a QR/link from that computer.")
#expect(try await responses.sentRequests().isEmpty)
}

@MainActor
@Test func manualHostPairingUsesNetworkRouteForLocalDNSNameWithStackAuth() async throws {
let responses = ScriptedTransportResponses([
try rpcErrorFrame(code: "method_not_found", message: "unknown method"),
try rpcWorkspaceListFrame(workspaceID: "local-dns-workspace", title: "Local DNS Workspace"),
])
@Test func manualHostPairingRejectsLocalDNSNameWithoutSendingStackToken() async throws {
// `.local`/Bonjour hosts are dialed over unencrypted TCP, so
// routeAllowsStackAuth excludes them: pairing must fail before any RPC
// (and the Stack bearer token) leaves the device.
let responses = ScriptedTransportResponses([])
let runtime = testRuntime(
supportedRouteKinds: [.tailscale],
transportFactory: ScriptedTransportFactory(responses: responses),
Expand All @@ -577,26 +567,19 @@ final class TerminalOutputCollector {
store.signIn()
await store.connectManualHost(name: "", host: "devbox.local", port: 61234)

let route = try #require(store.activeRoute)
#expect(store.phase == .workspaces)
#expect(store.connectionState == .connected)
#expect(store.connectionError == nil)
#expect(store.connectedHostName == "devbox.local")
if case let .hostPort(host, port) = route.endpoint {
#expect(host == "devbox.local")
#expect(port == 61234)
} else {
Issue.record("manual local DNS route should use host/port")
}
#expect(route.kind == .tailscale)
let requests = try await responses.sentRequests()
#expect(requests.map(\.method) == ["mobile.attach_ticket.create", "workspace.list"])
#expect(requests.allSatisfy { $0.stackAccessToken == "stack-token-for-local-dns" })
#expect(requests.allSatisfy { $0.attachToken == nil })
#expect(store.phase == .pairing)
#expect(store.connectionState == .disconnected)
#expect(store.activeTicket == nil)
#expect(store.activeRoute == nil)
#expect(store.connectionError == "This pairing route is not allowed. Enter a host and port, or pair with a QR/link from that computer.")
#expect(try await responses.sentRequests().isEmpty)
}

@MainActor
@Test func manualHostPairingProbesLANHostForAttachTicketBeforeStackAuthFallback() async throws {
@Test func manualHostPairingProbesTailscaleHostForAttachTicketBeforeStackAuthFallback() async throws {
// A trusted (Tailscale) manual host is probed for a real attach ticket
// first; an older Mac that does not implement the probe method falls back
// to a synthetic ticket and Stack-authenticated workspace.list.
let responses = ScriptedTransportResponses([
try rpcErrorFrame(code: "method_not_found", message: "unknown method"),
try rpcWorkspaceListFrame(workspaceID: "manual-workspace", title: "Manual Workspace"),
Expand All @@ -609,12 +592,12 @@ final class TerminalOutputCollector {
let store = CMUXMobileShellStore.preview(runtime: runtime)

store.signIn()
await store.connectManualHost(name: "Studio LAN", host: "192.168.1.77", port: 15432)
await store.connectManualHost(name: "Work Mac", host: "100.71.210.41", port: 15432)

#expect(store.phase == .workspaces)
#expect(store.connectionState == .connected)
#expect(store.connectionError == nil)
#expect(store.connectedHostName == "Studio LAN")
#expect(store.connectedHostName == "Work Mac")
let requests = try await responses.sentRequests()
#expect(requests.map(\.method) == ["mobile.attach_ticket.create", "workspace.list"])
#expect(requests.allSatisfy { $0.stackAccessToken == "stack-token-for-fallback" })
Expand Down Expand Up @@ -1375,11 +1358,11 @@ final class TerminalOutputCollector {
}

@MainActor
@Test func manualHostPairingUsesNetworkRouteForDefaultPortLANHostWithStackAuth() async throws {
let responses = ScriptedTransportResponses([
try rpcErrorFrame(code: "method_not_found", message: "unknown method"),
try rpcWorkspaceListFrame(workspaceID: "default-port-lan-workspace", title: "Default Port LAN Workspace"),
])
@Test func manualHostPairingRejectsDefaultPortLANHostWithoutSendingStackToken() async throws {
// Same encrypted-routes-only contract as the explicit-port LAN test, on
// the default host port: no RPC (and no Stack bearer token) may leave the
// device for a plain-TCP private-LAN route.
let responses = ScriptedTransportResponses([])
let runtime = testRuntime(
supportedRouteKinds: [.tailscale],
transportFactory: ScriptedTransportFactory(responses: responses),
Expand All @@ -1390,16 +1373,12 @@ final class TerminalOutputCollector {
store.signIn()
await store.connectManualHost(name: "Work Mac", host: "192.168.1.77", port: CmxMobileDefaults.defaultHostPort)

#expect(store.phase == .workspaces)
#expect(store.connectionState == .connected)
#expect(store.connectionError == nil)
#expect(store.connectedHostName == "Work Mac")
let route = try #require(store.activeRoute)
#expect(route.kind == .tailscale)
let requests = try await responses.sentRequests()
#expect(requests.map(\.method) == ["mobile.attach_ticket.create", "workspace.list"])
#expect(requests.allSatisfy { $0.stackAccessToken == "stack-token-for-default-lan" })
#expect(requests.allSatisfy { $0.attachToken == nil })
#expect(store.phase == .pairing)
#expect(store.connectionState == .disconnected)
#expect(store.activeTicket == nil)
#expect(store.activeRoute == nil)
#expect(store.connectionError == "This pairing route is not allowed. Enter a host and port, or pair with a QR/link from that computer.")
#expect(try await responses.sentRequests().isEmpty)
}

@MainActor
Expand Down
Loading