fix(serve-sim): read observed native hinge state

.changeset/serve-sim-observed-hinge-readback.md

@@ -0,0 +1,7 @@
+---
+"@expo/serve-sim": patch
+---
+
+Report native hinge observations through the CLI and host actions after external folds. Report failed native reads as errors instead of successful partial state.
+
+Preserve readback from older helpers and immediate acknowledged Tent controls. Report tabletop activity only from native samples, and keep unobserved Table Mode availability unknown.

llp/0003-serve-sim-http-api.spec.md

@@ -5,7 +5,7 @@
 **Systems:** ServeSim, HubClient
 **Author:** Gabe Debes
 **Date:** 2026-09-23
-**Revised:** 2026-10-15 (rotation readback and app actions; clipboard; server-wide preview sharing; capture reboot status and recording admission; device-state recording manifest; settings and device actions; action catalog)
+**Revised:** 2026-10-17 (rotation readback and app actions; clipboard; server-wide preview sharing; capture reboot status and recording admission; device-state recording manifest; settings and device actions; action catalog; observed hinge readback)
 **Related:** LLP 0001, LLP 0002
 
 > File paths such as `src/…` are relative to `packages/serve-sim/packages/serve-sim`, unless the text gives a path from the repository root.
@@ -481,7 +481,13 @@ plus `increment` and `decrement` for `textSize`; aliases are refused. `settings.
 - `memoryWarning`: `{udid}`. Sends a memory warning to the device.
 - `caDebug.set`: `{udid, option, value}`. `option` is a name from `src/ca-debug.ts`;
   `value` is `on` or `off`.
-- `hinge.get`: `{udid}`. Reads the hinge state as JSON.
+- `hinge.get`: `{udid}`. Reads hinge state as JSON. New helpers advertise
+  `supportsHingeStateReadback` and answer binary request `0x13` with `0x93`:
+  `{ok, state?, error?}`. The read takes its turn in the hinge queue and uses
+  native eligibility, which reconciles acknowledged controls with newer source
+  samples. `tableMode` remains the acknowledged persistent setting;
+  `tableModeActive` reports native tabletop activity separately. Older helpers
+  retain their config-based fallback. Unknown native pose fields stay unknown.
 - `hinge.set`: `{udid, value}`. `value` is `fold`, `half`, `unfold` or an angle from
   0 to 180.
 - `permissions.list`: `{udid, bundleId?}`. Returns the app permissions as JSON.

llp/0004-serve-sim-hinge-controls.explainer.md

@@ -45,7 +45,7 @@ CLI and existing clients; the additional `observedDeviceState` object is a full
 native snapshot for preview. Once present, omitted fields are unknown and never
 fall back to saved commands. Table Mode eligibility uses physical orientation,
 independently of an app's screen rotation. Before enabling Table Mode or turning
-face-down, a native getter reconciles angle and physical orientation with their latest
+face-down, a native getter reconciles angle, physical orientation and table state with their latest
 successful control sends. A command's value remains eligible until that field has a newer source
 sample, including an unknown sample; unrelated fields and delayed delivery do not
 revoke it. Observation loss and capture reset clear these temporary values. The
@@ -157,9 +157,13 @@ The CLI also accepts an angle, such as `serve-sim hinge 120 -d <udid>`. A comman
 waits for an acknowledgement and reports failures instead of assuming that
 writing to the input socket changed the device.
 
-Run `serve-sim hinge` without a position to print the server's confirmed hinge
-state as JSON: angle, pose, physical orientation, Table Mode, and whether Table
-Mode is available. It does not move the device. Unknown fields are left out.
+Run `serve-sim hinge` without a position to read the observed angle, pose, and
+physical orientation as JSON. `tableMode` is the acknowledged setting;
+`tableModeActive` is the observed native sensor activity, which can differ from
+that setting. The response also reports whether Table Mode is available. It
+does not move the device. Unknown fields are left out; a failed native read
+returns an error instead of a successful partial state. [observed:
+`DeviceSession` command `0x13`, `NativeHid.hingeControlEligibility`]
 
 ## Hinge state at startup
 
@@ -187,6 +191,8 @@ pose command that fails.
 
 ## Confirmed hinge state
 
+Acknowledged control updates share one recorder for legacy command fields and timestamped readback overlays. Raw native snapshots remain separate. The native read returns raw tabletop activity alongside the reconciled control fields in one actor turn; `tableModeActive` never uses the acknowledgement overlay. `hingePose` follows reconciled controls so an acknowledged Tent remains readable before the next sample. Legacy helpers without the readback protocol retain config command fields. Readback omits Table Mode availability until angle and orientation are known; unknown does not mean unavailable.
+
 Recording and live preview observe native pose separately from the successful-command cache and
 saved startup state. It opens the hinge stream with requested 100 ms updates and
 a one-degree threshold, plus a generic motion stream for physical orientation

packages/serve-sim/packages/serve-sim/Sources/SimNative/CoreDeviceBridge.swift

@@ -101,11 +101,30 @@ actor CoreDeviceBridge {
         return hingeControlEligibility[udid, default: HingeControlEligibility()].state(observed: observed)
     }
 
-    func recordHingeControl(udid: String, angle: Double? = nil, orientation: String? = nil,
+    // Reconciled controls and raw activity are read in one actor turn, without
+    // adding a second cache or turning a setting acknowledgement into a sample.
+    func hingeControlReadback(udid: String) -> (state: HingeState, tableModeActive: Bool?) {
+        (hingeControlEligibility(udid: udid), observedDeviceStates[udid]?.state.tableMode)
+    }
+
+    func recordHingeControl(udid: String, angle: Double? = nil, orientation: String? = nil, tableMode: Bool? = nil,
                             sentAt: Double, generation: UInt64) {
         guard hingeResetGenerations[udid, default: 0] == generation else { return }
-        if let angle { hingeControlEligibility[udid, default: HingeControlEligibility()].angle = (angle, sentAt) }
-        if let orientation { hingeControlEligibility[udid, default: HingeControlEligibility()].orientation = (orientation, sentAt) }
+        // @ref LLP 0004#confirmed-hinge-state — successful controls share one recorder
+        // One successful-send path updates legacy command readback and the
+        // timestamped overlay. Raw observations remain owned by their stream.
+        if let angle {
+            hingeStates[udid, default: HingeState()].angle = angle
+            hingeControlEligibility[udid, default: HingeControlEligibility()].angle = (angle, sentAt)
+        }
+        if let orientation {
+            hingeStates[udid, default: HingeState()].orientation = orientation
+            hingeControlEligibility[udid, default: HingeControlEligibility()].orientation = (orientation, sentAt)
+        }
+        if let tableMode {
+            hingeStates[udid, default: HingeState()].tableMode = tableMode
+            hingeControlEligibility[udid, default: HingeControlEligibility()].tableMode = (tableMode, sentAt)
+        }
     }
 
     func subscribeDeviceState(udid: String, onChange: @escaping @Sendable (DeviceStateSnapshot) -> Void) async -> @Sendable () async -> Void {
@@ -284,7 +303,6 @@ actor CoreDeviceBridge {
             self.recordHingeControl(udid: udid, angle: angle, sentAt: sentAt, generation: generation)
         }
         guard hingeResetGenerations[udid, default: 0] == generation else { return false }
-        if sent { hingeStates[udid, default: HingeState()].angle = angle }
         return sent
     }
 
@@ -328,7 +346,7 @@ actor CoreDeviceBridge {
         // still fail, and a real send failure must not be reported as success.
         guard SSCoreDeviceTableModeAvailable() else {
             fputs("[hid] CoreDevice Table Mode unavailable in this Xcode\n", stderr)
-            if !enabled { hingeStates[udid, default: HingeState()].tableMode = false }
+            if !enabled { recordHingeControl(udid: udid, tableMode: false, sentAt: hingeClock(), generation: hingeResetGenerations[udid, default: 0]) }
             return !enabled
         }
         let generation = hingeResetGenerations[udid, default: 0]
@@ -339,14 +357,15 @@ actor CoreDeviceBridge {
                 witnessSymbol: "$s10CoreDevice29UniversalHIDServiceCapabilityVAA0bE0AAWP"
             )
             guard hingeResetGenerations[udid, default: 0] == generation else { return false }
+            let sentAt = hingeClock()
             let sent = SSCoreDeviceSendTableMode(capability.storage, enabled)
-            if sent { hingeStates[udid, default: HingeState()].tableMode = enabled }
+            if sent { recordHingeControl(udid: udid, tableMode: enabled, sentAt: sentAt, generation: generation) }
             if !sent { capabilities.removeValue(forKey: "\(udid):\(metadataSymbol)") }
             return sent
         } catch BridgeError.unavailable {
             fputs("[hid] CoreDevice Table Mode capability unavailable\n", stderr)
             guard hingeResetGenerations[udid, default: 0] == generation else { return false }
-            if !enabled { hingeStates[udid, default: HingeState()].tableMode = false }
+            if !enabled { recordHingeControl(udid: udid, tableMode: false, sentAt: hingeClock(), generation: generation) }
             return !enabled
         } catch {
             fputs("[hid] CoreDevice Table Mode failed: \(error)\n", stderr)
@@ -363,7 +382,6 @@ actor CoreDeviceBridge {
             self.recordHingeControl(udid: udid, orientation: value, sentAt: sentAt, generation: generation)
         }
         guard hingeResetGenerations[udid, default: 0] == generation else { return false }
-        if sent { hingeStates[udid, default: HingeState()].orientation = value }
         return sent
     }
 

packages/serve-sim/packages/serve-sim/Sources/SimNative/CoreDeviceState.swift

@@ -56,7 +56,7 @@ struct ObservedDeviceState {
     private(set) var revision: UInt64 = 0
     private(set) var state = CoreDeviceBridge.HingeState()
     private(set) var angleTimestamp: Double?
-    private var motionTimestamp: Double?
+    private(set) var motionTimestamp: Double?
     private(set) var orientationTimestamp: Double?
 
     init(generation: UInt64) { self.generation = generation }
@@ -108,6 +108,7 @@ struct ObservedDeviceState {
 struct HingeControlEligibility {
     var angle: (value: Double, sentAt: Double)?
     var orientation: (value: String, sentAt: Double)?
+    var tableMode: (value: Bool, sentAt: Double)?
     private var orientationUncertainUntil: Double?
 
     mutating func recordOrientationRead(_ sample: PhysicalOrientationSample, since startedAt: Double) {
@@ -121,6 +122,9 @@ struct HingeControlEligibility {
     func state(observed: ObservedDeviceState) -> CoreDeviceBridge.HingeState {
         var state = observed.state
         if let angle, (observed.angleTimestamp ?? -.infinity) <= angle.sentAt { state.angle = angle.value }
+        if let tableMode, (observed.motionTimestamp ?? -.infinity) <= tableMode.sentAt {
+            state.tableMode = tableMode.value
+        }
         if let orientation {
             if let end = orientationUncertainUntil, orientation.sentAt <= end,
                (observed.orientationTimestamp ?? -.infinity) <= end {

packages/serve-sim/packages/serve-sim/Sources/SimNative/sim-module.swift

@@ -113,7 +113,10 @@ private func u32(_ v: Int) -> UInt32 {
     }
 
     @NodeMethod func hingeControlEligibility() async -> [String: any NodePropertyConvertible] {
-        hingeStateProperties(await CoreDeviceBridge.shared.hingeControlEligibility(udid: udid))
+        let readback = await CoreDeviceBridge.shared.hingeControlReadback(udid: udid)
+        var result = hingeStateProperties(readback.state)
+        if let active = readback.tableModeActive { result["tableModeActive"] = active }
+        return result
     }
 
     private func hingeStateProperties(_ state: CoreDeviceBridge.HingeState) -> [String: any NodePropertyConvertible] {

packages/serve-sim/packages/serve-sim/Tests/SimNativeTests/DeviceStateSubscriptionTests.swift

@@ -204,6 +204,28 @@ final class DeviceStateSubscriptionTests: XCTestCase {
         XCTAssertEqual(observed.revision, 0)
     }
 
+    func testReadbackKeepsTableActivitySeparateFromAcknowledgedControls() async throws {
+        let connected = expectation(description: "producer")
+        let source = DeviceStateProducerProbe(onConnect: { _ in connected.fulfill() })
+        let bridge = makeBridge(source)
+        let unsubscribe = await bridge.subscribeDeviceState(udid: udid) { _ in }
+        await fulfillment(of: [connected], timeout: 2)
+        await bridge.recordHingeControl(udid: udid, angle: 80, orientation: "facedown", tableMode: true, sentAt: 20, generation: 0)
+        let unknown = await bridge.hingeControlReadback(udid: udid)
+        XCTAssertEqual(unknown.state.tableMode, true)
+        XCTAssertNil(unknown.tableModeActive)
+        for (timestamp, activity) in [(11, 0), (21, 1), (22, 99)] {
+            let sample = try JSONDecoder().decode(DeviceMotionSample.self, from: Data(
+                "{\"timestamp\":\(timestamp),\"properties\":{\"propertyC\":{\"int\":\(activity)}}}".utf8
+            ))
+            await source.inject(0, .motion(sample))
+            let readback = await bridge.hingeControlReadback(udid: udid)
+            XCTAssertEqual(readback.tableModeActive, activity == 99 ? nil : activity != 0)
+            XCTAssertEqual(readback.state.tableMode, timestamp <= 20 ? true : readback.tableModeActive)
+        }
+        await unsubscribe()
+    }
+
     func testEligibilityUsesSourceTimeWithoutChangingObservedValues() async {
         let connected = expectation(description: "producer")
         let source = DeviceStateProducerProbe(onConnect: { _ in connected.fulfill() })

packages/serve-sim/packages/serve-sim/Tests/SimNativeTests/ObservedDeviceStateTests.swift

@@ -78,6 +78,23 @@ final class ObservedDeviceStateTests: XCTestCase {
         XCTAssertEqual(json["revision"] as? Int, 1)
     }
 
+    func testTentTableControlOverridesOnlyOlderMotionAndNeverChangesRawObservation() throws {
+        var observed = ObservedDeviceState(generation: 1)
+        observed.apply(.motion(try motion(11, orientation: 1, tabletop: 0)), since: 10)
+        var controls = HingeControlEligibility()
+        controls.angle = (80, 20)
+        controls.orientation = ("facedown", 20)
+        controls.tableMode = (true, 20)
+        XCTAssertEqual(controls.state(observed: observed), .init(angle: 80, orientation: "facedown", tableMode: true))
+        XCTAssertEqual(observed.state, .init(orientation: "portrait", tableMode: false))
+        observed.apply(.motion(try motion(19, orientation: 1, tabletop: 0)), since: 10)
+        XCTAssertEqual(controls.state(observed: observed).tableMode, true)
+        observed.apply(.motion(try motion(21, orientation: 6, tabletop: 0)), since: 10)
+        XCTAssertEqual(controls.state(observed: observed).tableMode, false)
+        observed.apply(.motion(try motion(22, orientation: 6, tabletop: 99)), since: 10)
+        XCTAssertNil(controls.state(observed: observed).tableMode)
+    }
+
     private func motion(_ timestamp: Double, orientation: Int, tabletop: Int) throws -> DeviceMotionSample {
         let json = "{\"timestamp\":\(timestamp),\"properties\":{\"propertyA\":{\"int\":\(orientation)},\"propertyC\":{\"int\":\(tabletop)}}}"
         return try JSONDecoder().decode(DeviceMotionSample.self, from: Data(json.utf8))

packages/serve-sim/packages/serve-sim/src/__tests__/fixtures/device-session-screen-config.child.ts

@@ -714,6 +714,18 @@ describe("native device state", () => {
         physicalOrientation: "portrait",
         observedDeviceState: { hingeAngle: 90, physicalOrientation: "portrait", tableMode: false },
       });
+      const readback = new Promise<unknown>((resolve) => {
+        ws!.on("message", function onReadback(data) {
+          const frame = Buffer.from(data as Buffer);
+          if (frame[0] !== 0x93) return;
+          ws!.off("message", onReadback);
+          resolve(JSON.parse(frame.subarray(1).toString()));
+        });
+      });
+      ws!.send(Buffer.from([0x13]));
+      expect(await readback).toMatchObject(
+        result === "unknown" ? { ok: true } : { ok: false, error: "Hinge state is unavailable" },
+      );
     });
   }