feat(web-shell): paginate restored session history

docs/design/web-shell-history-pagination.md

@@ -0,0 +1,155 @@
+# Web Shell History Pagination
+
+## Problem
+
+Web Shell restores an existing session through `POST /session/:id/load`. The
+daemon currently converts the complete persisted conversation into replay
+updates, bounds the resulting in-memory snapshot by bytes, and returns the
+retained snapshot as one `compactedReplay` response. Web Shell normalizes that
+entire response before painting the transcript.
+
+The byte bound prevents unbounded daemon memory, but a long session can still
+produce a multi-megabyte response and a large synchronous normalization pass.
+Older events are also represented only by a truncation marker; the persisted
+transcript pager is not connected to Web Shell.
+
+The existing transcript API pages from the oldest record toward the newest.
+That order is useful for exports but does not support chat history UX, where the
+latest messages must paint first and earlier messages are requested on demand.
+
+## Goals
+
+- Let Web Shell request a bounded latest page during `session/load`.
+- Preserve the atomic replay snapshot and SSE watermark produced by load.
+- Load older persisted history with an opaque backward cursor.
+- Keep pages chronological when rendered and preserve the viewport when older
+  content is prepended.
+- Bound browser transcript state even when the persisted session is very long.
+- Preserve existing load and forward transcript paging behavior for clients
+  that do not opt in.
+
+## Non-goals
+
+- Changing the model-facing conversation restored by core. The agent still
+  receives the complete resumed conversation required for continuation.
+- Automatically loading the complete transcript.
+- Paging archived transcripts.
+- Replacing SSE replay or reconnect behavior.
+- Making the first persisted-transcript index scan sublinear. The existing
+  frozen-snapshot index remains linear on a cache miss.
+
+## Design
+
+### Stable history record identity
+
+History replay already tracks the active persisted `ChatRecord` while emitting
+ACP updates. The replay collector will stamp that record UUID into private
+update metadata. This metadata is present only on replayed persisted history;
+live events keep their current shape.
+
+The UUID gives the initial load page an exact exclusive boundary. Older paging
+therefore does not rely on timestamps and cannot duplicate the first retained
+record when timestamps collide.
+
+### Paged session load
+
+The TypeScript SDK adds an optional `historyPageSize` restore option. The serve
+route validates it with the transcript pager's existing range of 1 through 500
+and forwards it through the bridge as private ACP load metadata. Servers that
+support this contract advertise `session_transcript_pagination` through
+`/capabilities.features`; Web Shell sends the option only when that feature is
+present, so older servers retain their bulk-load behavior.
+
+When the option is absent, load is unchanged. When present, the ACP agent keeps
+the complete resumed conversation in core but converts only the latest record
+suffix for the UI replay envelope. The suffix begins at a normal user-turn
+boundary. The envelope reports whether older active-chain records exist.
+
+The bridge seeds only that page into the session EventBus. Its response still
+contains the replay events and the EventBus `lastEventId` from one load
+operation, so events emitted after the snapshot remain available through SSE
+without a `resume`/transcript-read race.
+
+### Backward persisted transcript pages
+
+`GET /session/:id/transcript` and its workspace-qualified form accept an
+exclusive `beforeRecordId` only on the first backward request. The response
+uses the existing `nextCursor` field; the signed cursor records that paging is
+backward and freezes file identity, active leaf, byte size, position, and replay
+direction.
+
+Backward pages are returned in chronological display order. Each selected page
+starts at a normal user-turn boundary. The record limit is therefore a soft page
+target: a long turn is returned intact even when it exceeds the requested record
+count, so scrolling never reveals only the tail of the previous turn. The
+workspace route retains its hard source-byte limit: if a complete turn exceeds
+that limit, it returns `transcript_page_too_large` rather than returning a
+partial turn. Forward cursors and responses remain byte-for-byte compatible.
+
+### WebUI transcript state
+
+The provider normalizes an older page in an isolated transcript store,
+allocates non-conflicting block ordinals, and resets the active store with only
+the resulting blocks prepended. Live side-channel state, active streaming
+pointers, and the SSE event cursor are preserved.
+
+Older pages are prepended atomically: if a complete page would exceed
+`maxBlocks`, the UI keeps the current transcript unchanged, stops paging, and
+reports that earlier records remain persisted but cannot be displayed in the
+current view. A complete initial page is never truncated mid-turn; when it
+exceeds `maxBlocks`, the provider retains that page and prevents older paging.
+
+### Web Shell interaction
+
+Web Shell opts into a 100-record initial history page. The session provider
+derives the exclusive boundary from the earliest replay record UUID and exposes
+history state through a small hook:
+
+- whether an older page exists;
+- whether a page request is in flight;
+- an action to load the next page.
+
+When the message list reaches the top, it automatically requests the next page
+and shows a loading status above the transcript. It records the scroll container
+height before the request and restores the visual anchor after prepending. A
+short page that does not fill the viewport automatically advances until the
+transcript becomes scrollable or history is exhausted. A failed or partial page
+leaves the current transcript intact, stops automatic retries, and surfaces the
+existing daemon notice path.
+
+## Consistency and failure handling
+
+- Initial history and the SSE watermark remain coupled through `session/load`.
+- The first older page excludes the exact earliest persisted record already in
+  the load snapshot.
+- Continuation cursors freeze the transcript snapshot; later appends do not
+  change older pages.
+- Session changes invalidate in-flight page results before they can modify the
+  new transcript.
+- Invalid boundaries and cursors return the existing invalid-transcript-cursor
+  error family.
+- Snapshot replacement, rewind, archive, or deletion returns
+  `transcript_snapshot_unavailable`; the provider stops paging and leaves
+  already-rendered content intact.
+- A replay conversion failure does not prepend a partial page or advance its
+  boundary. The provider stops automatic paging and emits a warning notice.
+- Existing clients that omit `historyPageSize` or `beforeRecordId` retain the
+  current bulk-load and forward-page contracts.
+
+## Affected areas
+
+| Layer                    | Change                                                       |
+| ------------------------ | ------------------------------------------------------------ |
+| Core transcript reader   | Backward cursor and exclusive record boundary                |
+| ACP replay               | Record UUID metadata and latest-suffix selection             |
+| ACP bridge / serve route | Paged-load metadata, validation, and `hasMore` propagation   |
+| TypeScript SDK           | Restore option, backward page option, restored history state |
+| WebUI provider           | Isolated prepend, page state, stale-request protection       |
+| Web Shell                | Opt-in page size and automatic top-loading behavior          |
+
+## Open questions and follow-ups
+
+- A session created and kept live for its entire lifetime has no persisted
+  record UUID on its live EventBus frames. This change pages sessions restored
+  through the new load path; adding record identity to live emission is a
+  separate recording/emission coordination change.

integration-tests/cli/qwen-serve-routes.test.ts

@@ -344,6 +344,7 @@ describe('qwen serve — capabilities envelope', () => {
       'session_organization',
       'session_export',
       'session_transcript',
+      'session_transcript_pagination',
       'mcp_guardrails',
       'workspace_mcp_manage',
       'mcp_guardrail_events',

packages/acp-bridge/src/bridge.test.ts

@@ -1581,7 +1581,7 @@ describe('createAcpSessionBridge', () => {
 
     const result = await bridge.getSessionTranscriptPage({
       sessionId: 'session-1',
-      cursor: 'opaque',
+      beforeRecordId: 'record-3',
       limit: 2,
     });
 
@@ -1597,7 +1597,7 @@ describe('createAcpSessionBridge', () => {
         params: {
           cwd: WS_A,
           sessionId: 'session-1',
-          cursor: 'opaque',
+          beforeRecordId: 'record-3',
           limit: 2,
         },
       },
@@ -2775,12 +2775,14 @@ describe('createAcpSessionBridge', () => {
         loadSessionImpl: (p) => {
           expect(p._meta).toMatchObject({
             'qwen.session.loadReplayMode': 'bulk',
+            'qwen.session.loadReplayPageSize': 100,
           });
           return {
             _meta: {
               keep: 'state',
               'qwen.session.loadReplay': {
                 v: 1,
+                hasMore: true,
                 partial: true,
                 replayError: 'replay boom',
                 updates: [
@@ -2808,17 +2810,22 @@ describe('createAcpSessionBridge', () => {
       sessionId: 'persisted-bulk-history',
       workspaceCwd: WS_A,
       historyReplay: 'response',
+      historyPageSize: 100,
     });
 
     expect(handles[0]?.agent.loadSessionCalls[0]).toMatchObject({
       sessionId: 'persisted-bulk-history',
       cwd: WS_A,
       mcpServers: [],
-      _meta: { 'qwen.session.loadReplayMode': 'bulk' },
+      _meta: {
+        'qwen.session.loadReplayMode': 'bulk',
+        'qwen.session.loadReplayPageSize': 100,
+      },
     });
     expect(loaded.state).toEqual({ _meta: { keep: 'state' } });
     expect(loaded.partial).toBe(true);
     expect(loaded.replayError).toBe('replay boom');
+    expect(loaded.historyHasMore).toBe(true);
     expect(loaded.lastEventId).toBe(2);
     expect(loaded.compactedReplay).toHaveLength(2);
     expect(loaded.liveJournal).toEqual([]);
@@ -2916,10 +2923,12 @@ describe('createAcpSessionBridge', () => {
                 {
                   sessionUpdate: 'user_message_chunk',
                   content: { type: 'text', text: `old-${'x'.repeat(600)}` },
+                  _meta: { 'qwen.session.recordId': 'record-1' },
                 },
                 {
                   sessionUpdate: 'agent_message_chunk',
                   content: { type: 'text', text: `new-${'y'.repeat(600)}` },
+                  _meta: { 'qwen.session.recordId': 'record-2' },
                 },
               ],
             },
@@ -2935,6 +2944,7 @@ describe('createAcpSessionBridge', () => {
       sessionId: 'persisted-bounded-history',
       workspaceCwd: WS_A,
       historyReplay: 'response',
+      historyPageSize: 100,
     });
 
     expect(loaded.lastEventId).toBe(2);
@@ -2951,6 +2961,7 @@ describe('createAcpSessionBridge', () => {
       update?: { content?: { text?: string } };
     };
     expect(retained.update?.content?.text).toBe(`new-${'y'.repeat(600)}`);
+    expect(loaded.historyHasMore).toBe(true);
 
     await bridge.shutdown();
   });

packages/acp-bridge/src/bridge.ts

@@ -88,6 +88,7 @@ import {
   LOAD_REPLAY_BULK_MODE,
   LOAD_REPLAY_META_KEY,
   LOAD_REPLAY_MODE_META_KEY,
+  LOAD_REPLAY_PAGE_SIZE_META_KEY,
   LOAD_REPLAY_VERSION,
 } from './bridgeTypes.js';
 import type {
@@ -218,6 +219,7 @@ function extractLoadReplayResponse(state: BridgeSessionState): {
   updates: SessionUpdate[];
   partial?: true;
   replayError?: string;
+  hasMore?: boolean;
 } {
   const meta = isRecord(state._meta) ? state._meta : undefined;
   const replay = meta?.[LOAD_REPLAY_META_KEY];
@@ -256,6 +258,13 @@ function extractLoadReplayResponse(state: BridgeSessionState): {
         `(version=${LOAD_REPLAY_VERSION}, replayError=${describeLoadReplayValue(replayError)})`,
     );
   }
+  const hasMore = replay['hasMore'];
+  if (hasMore !== undefined && typeof hasMore !== 'boolean') {
+    throw new Error(
+      `Invalid qwen.session.loadReplay hasMore ` +
+        `(version=${LOAD_REPLAY_VERSION}, hasMore=${describeLoadReplayValue(hasMore)})`,
+    );
+  }
   const invalidUpdateIndex = rawUpdates.findIndex(
     (update) => !isBulkReplayUpdate(update),
   );
@@ -284,6 +293,7 @@ function extractLoadReplayResponse(state: BridgeSessionState): {
     updates: rawUpdates,
     ...(partial === true ? { partial: true as const } : {}),
     ...(typeof replayError === 'string' ? { replayError } : {}),
+    ...(hasMore === true ? { hasMore: true } : {}),
   };
 }
 
@@ -588,6 +598,7 @@ interface SessionEntry {
   /** Response-mode `session/load` can return a partial replay prefix. */
   restoreReplayPartial?: true;
   restoreReplayError?: string;
+  restoreHistoryHasMore?: true;
   /**
    * Most recent heartbeat across any client on this session (Date.now()
    * epoch ms). Set on every `recordHeartbeat` call regardless of whether
@@ -3637,7 +3648,10 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge {
   const replayFieldsFor = (
     entry: Pick<
       SessionEntry,
-      'events' | 'restoreReplayPartial' | 'restoreReplayError'
+      | 'events'
+      | 'restoreReplayPartial'
+      | 'restoreReplayError'
+      | 'restoreHistoryHasMore'
     >,
     action: 'load' | 'resume',
   ): Pick<
@@ -3647,6 +3661,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge {
     | 'lastEventId'
     | 'partial'
     | 'replayError'
+    | 'historyHasMore'
   > => {
     const replayStatus =
       action === 'load' && entry.restoreReplayPartial === true
@@ -3667,6 +3682,9 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge {
         liveJournal: snapshot.liveJournal,
         lastEventId: snapshot.lastEventId,
         ...replayStatus,
+        ...(entry.restoreHistoryHasMore === true
+          ? { historyHasMore: true }
+          : {}),
       };
     }
     return { lastEventId: snapshot.lastEventId, ...replayStatus };
@@ -3890,6 +3908,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge {
       let replayUpdates: SessionUpdate[] = [];
       let replayPartial: true | undefined;
       let replayError: string | undefined;
+      let replayHasMore: true | undefined;
       try {
         if (action === 'load') {
           state = await Promise.race([
@@ -3907,6 +3926,12 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge {
                   ? {
                       _meta: {
                         [LOAD_REPLAY_MODE_META_KEY]: LOAD_REPLAY_BULK_MODE,
+                        ...(req.historyPageSize !== undefined
+                          ? {
+                              [LOAD_REPLAY_PAGE_SIZE_META_KEY]:
+                                req.historyPageSize,
+                            }
+                          : {}),
                       },
                     }
                   : {}),
@@ -3936,6 +3961,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge {
           replayUpdates = extracted.updates;
           replayPartial = extracted.partial;
           replayError = extracted.replayError;
+          replayHasMore = extracted.hasMore === true ? true : undefined;
         }
       } catch (err) {
         restoreEvents.close();
@@ -4031,6 +4057,9 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge {
       if (replayError !== undefined) {
         entry.restoreReplayError = replayError;
       }
+      if (replayHasMore === true) {
+        entry.restoreHistoryHasMore = true;
+      }
       seedSnapshotCaches(entry, publicState);
       const artifactRestoreWarnings = await entry.artifacts.restore(
         restoredArtifactSnapshot,
@@ -4050,6 +4079,14 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge {
           ingestArtifacts:
             restoredArtifactSnapshot === undefined || artifactRestoreFailed,
         });
+        if (
+          req.historyPageSize !== undefined &&
+          entry.events
+            .snapshotReplay()
+            ?.compactedTurns.some((event) => event.type === 'history_truncated')
+        ) {
+          entry.restoreHistoryHasMore = true;
+        }
         ci.client.drainEarlyEvents(entry.sessionId, entry);
       }
       const clientId = registerClient(entry, req.clientId);