Skip to content

fix(serve-sim): capture the active or requested display - #384

Merged
gwdp merged 14 commits into
gwdp/serve-sim-observed-hinge-statefrom
gwdp/serve-sim-screenshot-display-selection
Oct 23, 2026
Merged

gwdp merged 14 commits into
gwdp/serve-sim-observed-hinge-statefrom
gwdp/serve-sim-screenshot-display-selection

Conversation

@gwdp

@gwdp gwdp commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

What

Capture a requested device screen, or freshly select the native active display. Fall back only to a proven sole screen, including tvOS, or a sole integrated panel when other ports are present. Duo and unknown profiles cannot fall back.

One-shot captures share sessions with viewers and release temporary capture on success, failure, or native timeout. UDID case aliases share one registry entry. Publish only validated PNGs; serialize publication with failed-reservation cleanup so a failed capture cannot delete another call's successful image. Keep native details in host logs and reject known non-device display ports before capture. Profile lookup reads only the selected bundle asynchronously. Older iOS profiles also require a matching sole integrated framebuffer and connected screen before fallback or explicit-ID validation.

Why

Duo screenshots could use the inactive inner panel. Fresh native selection verifies a fold independently of the reader. Single-screen simulators retain a safe fallback when native selection is unavailable.

Authorship and LLM use

Primarily generated by Codex: code, tests, documentation, description and local validation. Gabe found issues while testing simulator:control, directed the fixes, and reviewed the earlier diff at a4b51215e. Codex and Devin review led to fresh selection, shared session ownership, cancellation/settings retention and cleanup. SJ's review prompted fallback, safe errors, cleanup, explicit-port validation, a fold-read investigation and Xcode 26 host-action coverage. Brent's review prompted tvOS compatibility, native deadlines, concurrent publication safety, registry identity, generation guards, phase errors and asynchronous profile lookup. Broad testing caught a recording-finalization interaction during that revision; Codex fixed it without weakening the assertion. The Xcode 26.6 runtime job then exposed older profiles without a display-ID array; Codex added verified legacy IO resolution and recorded the real CI profile as a regression fixture. Gabe reviewed the final changes at f174904a3; Gabe owns follow-up.

Evidence

Codex validation for 5475b12 (native and device checks below were run at f0d6375, before this JS-only profile follow-up):

  • JS/native build, formatting, lint, typecheck and LLP checks pass. bun run check remains nonzero at Knip; normalized findings match main.
  • Final isolated suite: 2,620 pass, 225 skip, 0 fail. TV selection, deadline, UDID and publication regressions failed before their fixes. An earlier broad process stalled; the contract suite passed independently and the final full run completed cleanly.
  • FrameCaptureSurfaceTests: 18 pass, including late reads across stop generations and incomplete foldable descriptors. Source review also checked ownership, late startup, recording finalization, fallback refusal and safe errors.
  • host-screenshot.e2e.test.ts: 3 pass on owned iPhone 17 Pro 3B8ACEA7-FBE7-4F42-AF2D-9AC96DA9ACB9, iOS 26.4, Xcode 26.4 (17E192). Default native-active and explicit display 1 return PNGs. Only Desktop copying was stubbed.
  • Owned Duo A073FDE1-E882-425C-A711-5AD79FD1D399, iOS 27.1 (24A94401), Xcode 27.1 (27A9269): default captures followed closed/open/closed/open and returned the expected 1398×2034 and 2007×2853 panel sizes. Both devices were shut down and deleted. Fresh-device bootstatus migration was not clean; these checks validate capture/pose paths, not full app readiness.

The prior Xcode 26.6 runtime job caught a default-screenshot failure with legacy ScreenDimensionsCapability profiles. A standalone bundle of the updated production profile helper plus real simctl capture passed on the same image: actual numeric screen ID 1, valid 1206×2622 PNG (validation run). All six checks pass on 5475b12, including both Xcode 26 runtime jobs and the host-action screenshot tests. Fresh Devin analysis completed with no new posted findings or open review threads. SJ and Brent have been asked to review the follow-ups again. No tvOS runtime or hosted EAS acceptance was run. Earlier validation covered missing-addon iPhone fallback and safe Duo refusal; 30 Duo fold transitions with 394 reads did not reproduce empty reads, so no speculative retries were added.

Rollback

Plain revert; redeploy the matching JS/native package.

— Codex

Hosted pre-merge QA

Codex tested the exact #388 tip 44816833b from its EAS CI build, with worker package pins and logs verified. Owned sessions used an iPhone Duo on Xcode 27.1 and an iPhone 17 on Xcode 26.6.

All 66 stable host/HTTP matrix captures passed across fold states, a fractional angle and rotation. Defaults matched the native active panel; explicit IDs returned the requested panel, including inactive panels. Both tunneled UI downloads passed, and live video kept advancing. Concurrent same-name captures preserved valid pixels; invalid IDs, methods, devices and unauthenticated requests failed safely. HTTP PNG/artifact headers passed. Nineteen focused local tests covered startup/read failures, safe single-display fallback, publication cleanup and session ownership; addon/profile fault injection was local, not hosted.

A separate Duo stress limit remains: overlapping rapid folds and captures can leave panel illumination out of step with the acknowledged hinge angle. It also reproduced on the prior #385 artifact. Independent explicit PNGs confirmed the lit panel; default selection followed it correctly. Reapplying Fully open in the UI recovered the current-tip session; SpringBoard restart alone did not. Native handoff reliability needs separate follow-up. No code changed during this QA. All owned sessions were stopped after testing.

— Codex

@changeset-bot

changeset-bot Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f174904

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@expo/serve-sim Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@gwdp

gwdp commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Codex implemented explicit and native-active screenshot selection, tests, and documentation at Gabe’s direction. Local Duo post-fold captures exposed stale cached selection; Codex replaced it with fresh native display queries and reran the device checks. Gabe reviewed the final diff at a4b51215e and requested review from Krystof and Brent. — Codex

@gwdp

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

devin-ai-integration[bot]

This comment was marked as resolved.

@gwdp

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@gwdp

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

devin-ai-integration[bot]

This comment was marked as resolved.

@gwdp

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

devin-ai-integration[bot]

This comment was marked as resolved.

@gwdp

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

devin-ai-integration[bot]

This comment was marked as resolved.

@gwdp

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@gwdp
gwdp marked this pull request as ready for review October 18, 2026 02:34
@gwdp

gwdp commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto the updated #383, which has all six CI checks passing. All nine commits match their original patches.

Codex ran the native build, lint, typecheck, LLP checks, and 129 focused screenshot, session, settings, and input-state tests; all pass. Knip diagnostics match clean main. Simulator runtime E2E will run in CI.

Only this next layer has been pushed. The remaining PRs will wait for its CI.

— Codex

@gwdp
gwdp force-pushed the gwdp/serve-sim-screenshot-display-selection branch from a4b5121 to c87c30c Compare October 19, 2026 18:00
@gwdp
gwdp requested a review from sjkim-expo October 19, 2026 18:05
@gwdp
gwdp added this pull request to stack #401 October 19, 2026 19:16
@devin-ai-integration

Copy link
Copy Markdown

Starting Devin Review.

Devin Review

@gwdp
gwdp requested review from brentvatne and sjkim-expo and removed request for brentvatne and sjkim-expo October 23, 2026 04:22
gwdp and others added 14 commits October 23, 2026 01:19
Co-authored-by: Codex <codex@openai.com>
Add direct host screenshot coverage for active and numeric displays on Xcode 26.

Co-authored-by: Codex <codex@openai.com>
Use the sole installed integrated panel when native selection is unavailable, reject known external ports, and keep native diagnostics in host logs. Cover fallback, failed validation, profile parsing, and private errors.

Co-authored-by: Codex <codex@openai.com>
Restore sole-screen tvOS fallback and read only the selected display profile asynchronously. Bound native screenshot startup and reads, preserve recording finalization during teardown, and guard late native results. Share case-insensitive session identity and serialize publication with failed reservation cleanup.

Co-authored-by: Codex <codex@openai.com>
Read older iOS main-screen profiles through a matching, unambiguous simctl IO enumeration. Preserve numeric display validation and refuse multi-panel or unknown fallback. Add the real Xcode 26.6 profile and IO regression fixture.

Co-authored-by: Codex <codex@openai.com>
@gwdp
gwdp force-pushed the gwdp/serve-sim-screenshot-display-selection branch from 5475b12 to f174904 Compare October 23, 2026 09:33
@gwdp

gwdp commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Codex restacked this branch onto #383 at b9e3e1bf0. All 14 patches belonging to #384 are unchanged (git range-diff); this update brings in the validated hinge-readback fixes. #383 has six passing checks and no unresolved review threads.

Local validation at the prepared #388 stack tip passed 2,635 tests before the final activity-readback follow-up. After inheriting that follow-up, the native build and four focused test wrappers passed. This is integration coverage of the prepared stack; this exact restacked head has not had a separate full local run. Fresh CI for this restack is pending. Human review requests are unchanged.

/devin review

Please check the integration against the updated parent at this head, f174904a3.

— Codex

@sjkim-expo sjkim-expo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rechecked the current head, and all six points are still addressed. The fallback still requires exactly one proven screen, and the restack left the patches unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants