diff --git a/.agents/upstream-review.md b/.agents/upstream-review.md index d1c63961c..8aade218d 100644 --- a/.agents/upstream-review.md +++ b/.agents/upstream-review.md @@ -120,13 +120,14 @@ All four sources in the exact range are accounted for in [#516](https://github.c The maintainer explicitly reopened the four remaining functional exceptions on 2026-09-12, superseding the old earliest-check dates. -| Group | Sources | Outcome and remaining scope | Pylon PR / verification | -| ---------------------------------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Sidebar project scope in search row | `d1d15c67f4a5fb82fd8d5e01e5e3b288296789c3` | Adopted with Pylon environment identity, narrow-layout scope choice and keyboard accessibility retained. No excluded behavior. | [#527](https://github.com/pylon-code/pylon/pull/527); independent review, 196 focused tests, web types and all ten final-head CI jobs. | -| DEF-7: anonymous macOS previews | `33b650a5b3b27382b35d2182dec6b22438c3da56` (#8243); Pylon #111 and `9d112329e` | Closed. Public unsigned previews require explicit `preview:mac` opt-in on same-repository PRs. Read-only builds feed trusted publication and cleanup; Pylon Alpha identity and production updater isolation remain. No preview was published during implementation. | [#528](https://github.com/pylon-code/pylon/pull/528), with trusted helper loading fixed in [#532](https://github.com/pylon-code/pylon/pull/532); independent review, helper/loader regressions, Actionlint and final-head CI. Actual publication/cleanup workflow verification is tracked in #526. | -| DEF-16: live Claude model quota windows | `19d8ab2ae9fc562ee7b216a0d72903fbfafa9572` (#9507) | Closed through bounded cached OAuth reconciliation after unscoped native hints. Named windows retain their account identity, timestamps and 30-minute retention; generation/token fences reject retired instances and switched accounts. No guessed model mapping or duplicate SDK probe. | [#530](https://github.com/pylon-code/pylon/pull/530); independent review, quota/registry/retention/ingestion regressions, server types and all ten final-head CI jobs. | -| Mobile Antigravity catalog and admission | `06336460c9988f29c71e839c4c9c840c4552e077`, `d487dfbf46be344e818725be70ee04be2436bfb4` | Closes #479's mobile exception. Account-owned defaults/aliases, unavailable saved choices, unknown-auth saved sends, bounded targeted catalog discovery at durable dispatch and post-await validation preserve account binding and queued content. No remaining requested catalog behavior. | [#529](https://github.com/pylon-code/pylon/pull/529); independent review, 140 focused tests, mobile types and all ten final-head CI jobs. Native rendered/live-account verification is not claimed. | -| OpenCode exact rollback | `fd5553f1afcef4f410a067687faa991743b5034c`, `efccda9ac9230db22b36990cffabdad218fa41b0` | Extends #513 with immutable native forks, exact idle source/target proof, idempotent selection, original-source compensation and durable restart recovery. Historical checkpoints lacking a proved snapshot, experimental native plan-file mode and externally managed servers with unobservable runtime flags remain unavailable for exact rewind; ordinary plan/history/resume stay usable. Codex and Claude portions remain in #526. | [#531](https://github.com/pylon-code/pylon/pull/531); independent adapter/service review, focused integration tests and isolated installed OpenCode 1.18.29 native fork verification. | +| Group | Sources | Outcome and remaining scope | Pylon PR / verification | +| ---------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Sidebar project scope in search row | `d1d15c67f4a5fb82fd8d5e01e5e3b288296789c3` | Adopted with Pylon environment identity, narrow-layout scope choice and keyboard accessibility retained. No excluded behavior. | [#527](https://github.com/pylon-code/pylon/pull/527); independent review, 196 focused tests, web types and all ten final-head CI jobs. | +| DEF-7: anonymous macOS previews | `33b650a5b3b27382b35d2182dec6b22438c3da56` (#8243); Pylon #111 and `9d112329e` | Closed. Public unsigned previews require explicit `preview:mac` opt-in on same-repository PRs. Read-only builds feed trusted publication and cleanup; Pylon Alpha identity and production updater isolation remain. No preview was published during implementation. | [#528](https://github.com/pylon-code/pylon/pull/528), with trusted helper loading fixed in [#532](https://github.com/pylon-code/pylon/pull/532); independent review, helper/loader regressions, Actionlint and final-head CI. Actual publication/cleanup workflow verification is tracked in #526. | +| DEF-16: live Claude model quota windows | `19d8ab2ae9fc562ee7b216a0d72903fbfafa9572` (#9507) | Closed through bounded cached OAuth reconciliation after unscoped native hints. Named windows retain their account identity, timestamps and 30-minute retention; generation/token fences reject retired instances and switched accounts. No guessed model mapping or duplicate SDK probe. | [#530](https://github.com/pylon-code/pylon/pull/530); independent review, quota/registry/retention/ingestion regressions, server types and all ten final-head CI jobs. | +| Mobile Antigravity catalog and admission | `06336460c9988f29c71e839c4c9c840c4552e077`, `d487dfbf46be344e818725be70ee04be2436bfb4` | Closes #479's mobile exception. Account-owned defaults/aliases, unavailable saved choices, unknown-auth saved sends, bounded targeted catalog discovery at durable dispatch and post-await validation preserve account binding and queued content. No remaining requested catalog behavior. | [#529](https://github.com/pylon-code/pylon/pull/529); independent review, 140 focused tests, mobile types and all ten final-head CI jobs. Native rendered/live-account verification is not claimed. | +| OpenCode exact rollback | `fd5553f1afcef4f410a067687faa991743b5034c`, `efccda9ac9230db22b36990cffabdad218fa41b0` | Extends #513 with immutable native forks, exact idle source/target proof, idempotent selection, original-source compensation and durable restart recovery. Historical checkpoints lacking a proved snapshot, experimental native plan-file mode and externally managed servers with unobservable runtime flags remain unavailable for exact rewind; ordinary plan/history/resume stay usable. Claude portion remains in #526. | [#531](https://github.com/pylon-code/pylon/pull/531); independent adapter/service review, focused integration tests and isolated installed OpenCode 1.18.29 native fork verification. | +| Codex exact rollback | `fd5553f1afcef4f410a067687faa991743b5034c`, `efccda9ac9230db22b36990cffabdad218fa41b0` | Extends #513 with immutable native forks, complete JSONL and inactive goal proof, idempotent selection, compaction retention and same-incarnation recovery. Active/uninspectable goals, paginated forks, unsupported or oversized native histories and uncaptured old/root bindings remain ineligible; ordinary resume is preserved. | [#533](https://github.com/pylon-code/pylon/pull/533); independent native-source/adapter review, 360 focused tests, server types and integration capability checks. | ## Deferred register diff --git a/apps/server/src/provider/Layers/CodexAbsoluteHistory.test.ts b/apps/server/src/provider/Layers/CodexAbsoluteHistory.test.ts new file mode 100644 index 000000000..2e8b75b69 --- /dev/null +++ b/apps/server/src/provider/Layers/CodexAbsoluteHistory.test.ts @@ -0,0 +1,1075 @@ +// @effect-diagnostics nodeBuiltinImport:off +import * as NodeAssert from "node:assert/strict"; + +import { describe, it } from "@effect/vitest"; +import * as Effect from "effect/Effect"; +import * as Schema from "effect/Schema"; +import * as CodexErrors from "effect-codex-app-server/errors"; + +import { + codexConversationDigest, + forkCodexConversation, + readCodexConversation, + type CodexConversationSnapshot, +} from "./CodexAbsoluteHistory.ts"; + +const CWD = "/tmp/codex-history-test"; +const decodeRequest = Schema.decodeUnknownSync( + Schema.Struct({ + threadId: Schema.String, + includeTurns: Schema.optionalKey(Schema.Boolean), + lastTurnId: Schema.optionalKey(Schema.String), + cursor: Schema.optionalKey(Schema.NullOr(Schema.String)), + }), +); +const decodeTurnId = Schema.decodeUnknownSync(Schema.Struct({ id: Schema.String })); +const decodeNativeObject = Schema.decodeUnknownSync(Schema.Record(Schema.String, Schema.Json)); +const decodeThreadRecord = Schema.decodeUnknownSync( + Schema.Struct({ + thread: Schema.Record(Schema.String, Schema.Unknown), + cwd: Schema.optionalKey(Schema.String), + }), +); + +function turn(id: string, overrides: Schema.JsonObject = {}): Schema.Json { + return { + id, + status: "completed", + itemsView: "full", + completedAt: 123, + error: null, + items: [ + { id: `${id}-user`, type: "userMessage", content: [{ type: "text", text: "same prompt" }] }, + ], + ...overrides, + }; +} + +interface MockThread { + turns: ReadonlyArray; + nativeRecords?: ReadonlyArray; + nativeGoal?: Schema.JsonObject | null; + paginated?: boolean; + forkedFromId?: string; +} + +const nativeRecord = (type: string, payload: Schema.JsonObject): Schema.JsonObject => ({ + timestamp: "2026-01-01T00:00:00.000Z", + type, + payload, +}); +const nativeSession = (id: string, forkedFromId?: string): Schema.JsonObject => + nativeRecord("session_meta", { + id, + session_id: id, + timestamp: "2026-01-01T00:00:00.000Z", + cwd: CWD, + context_window: { window_id: `window-${id}` }, + base_instructions: { text: "native system instructions" }, + ...(forkedFromId ? { forked_from_id: forkedFromId } : {}), + }); +const nativeHistory = (id: string, turns: ReadonlyArray): Array => [ + nativeSession(id), + ...turns.flatMap((raw) => { + const { id: turnId } = decodeTurnId(raw); + return [ + nativeRecord("event_msg", { type: "task_started", turn_id: turnId }), + nativeRecord("turn_context", { + turn_id: turnId, + cwd: CWD, + developer_instructions: "native developer instructions", + }), + nativeRecord("response_item", { + type: "message", + id: `${turnId}-user`, + role: "user", + content: [{ type: "input_text", text: "same prompt" }], + }), + nativeRecord("event_msg", { type: "task_complete", turn_id: turnId }), + ]; + }), +]; +const rawNativeRecords = (id: string, thread: MockThread) => + thread.nativeRecords ?? nativeHistory(id, thread.turns); + +function makeClient(initialTurns: ReadonlyArray = [turn("turn-1"), turn("turn-2")]) { + const threads = new Map([ + ["source", { turns: structuredClone(initialTurns) }], + ]); + const calls: Array<{ method: string; params: unknown }> = []; + let forkCount = 0; + const hooks: { + rollout?: (path: string, text: string) => string; + response?: ( + method: string, + params: ReturnType, + response: unknown, + ) => unknown; + } = {}; + const client: Parameters[0] = { + readRollout: (path) => + Effect.sync(() => { + const id = path + .split("/") + .at(-1) + ?.replace(/\.jsonl$/, ""); + const thread = id === undefined ? undefined : threads.get(id); + NodeAssert.ok(thread && id); + const text = `${rawNativeRecords(id, thread) + .map((record) => JSON.stringify(record)) + .join("\n")}\n`; + return hooks.rollout?.(path, text) ?? text; + }), + raw: { + request: (method, params) => + Effect.sync(() => { + calls.push({ method, params }); + const request = decodeRequest(params); + const source = threads.get(request.threadId); + NodeAssert.ok(source, `Unknown thread ${request.threadId}`); + const metadata = (id: string, thread: MockThread, includeTurns: boolean) => ({ + thread: { + id, + cwd: CWD, + path: `${CWD}/${id}.jsonl`, + ephemeral: false, + status: { type: "idle" }, + ...(thread.paginated ? { historyMode: "paginated" } : {}), + ...(thread.forkedFromId ? { forkedFromId: thread.forkedFromId } : {}), + turns: includeTurns ? thread.turns : [], + }, + }); + let response: unknown; + if (method === "thread/goal/get") { + response = { goal: source.nativeGoal ?? null }; + } else if (method === "thread/read") { + response = metadata(request.threadId, source, request.includeTurns === true); + } else if (method === "thread/turns/list") { + const start = request.cursor == null ? 0 : Number(request.cursor); + response = { + data: source.turns.slice(start, start + 1), + nextCursor: start + 1 < source.turns.length ? String(start + 1) : null, + }; + } else if (method === "thread/fork") { + const count = + request.lastTurnId === undefined + ? source.turns.length + : source.turns.findIndex((item) => decodeTurnId(item).id === request.lastTurnId) + + 1; + const forked: MockThread = { + turns: structuredClone(source.turns.slice(0, count)), + ...(source.paginated ? { paginated: true } : {}), + forkedFromId: request.threadId, + }; + const forkId = `fork-${++forkCount}`; + forked.nativeGoal = source.nativeGoal + ? { ...source.nativeGoal, threadId: forkId } + : null; + const native = rawNativeRecords(request.threadId, source); + const turnStart = native.findIndex((record) => { + const payload = record.payload; + return ( + typeof payload === "object" && + payload !== null && + "type" in payload && + payload.type === "task_started" && + "turn_id" in payload && + payload.turn_id === request.lastTurnId + ); + }); + const nextStart = + request.lastTurnId === undefined + ? -1 + : native.findIndex((record, index) => { + const payload = record.payload; + return ( + index > turnStart && + typeof payload === "object" && + payload !== null && + "type" in payload && + payload.type === "task_started" + ); + }); + forked.nativeRecords = [ + nativeSession(forkId, request.threadId), + ...native.slice(0, nextStart < 0 ? undefined : nextStart), + ].map((record, ordinal) => ({ + ...record, + ordinal, + timestamp: "2026-01-02T00:00:00.000Z", + })); + threads.set(forkId, forked); + response = { + ...metadata(forkId, forked, !forked.paginated), + cwd: CWD, + model: "codex-test", + }; + } else { + NodeAssert.fail(`Unexpected mutation or request ${method}`); + } + const isolatedResponse: unknown = structuredClone(response); + return hooks.response?.(method, request, isolatedResponse) ?? isolatedResponse; + }), + }, + }; + return { client, calls, threads, hooks }; +} + +const readSource = (client: Parameters[0]) => + readCodexConversation(client, "source", CWD); + +function replaceThread(response: unknown, update: Record) { + const decoded = decodeThreadRecord(response); + return { ...decoded, thread: { ...decoded.thread, ...update } }; +} + +describe("codexConversationDigest", () => { + it("ignores enclosing identity and key order while retaining the entire native history", () => { + const first: CodexConversationSnapshot = { + threadId: "source", + cwd: CWD, + rolloutPath: `${CWD}/source.jsonl`, + nativeRecords: [nativeRecord("compacted", { replacement_history: [] })], + nativeGoal: null, + turns: [{ id: "one", unknown: { b: 2, a: 1 }, items: [], status: "completed" }], + }; + NodeAssert.equal( + codexConversationDigest(first), + codexConversationDigest({ + threadId: "fork", + cwd: CWD, + rolloutPath: `${CWD}/fork.jsonl`, + nativeRecords: first.nativeRecords, + nativeGoal: null, + turns: [{ status: "completed", items: [], unknown: { a: 1, b: 2 }, id: "one" }], + }), + ); + for (const changed of [ + { ...first, cwd: "/tmp/other" }, + { + ...first, + turns: [{ id: "different", unknown: { a: 1, b: 2 }, items: [], status: "completed" }], + }, + { ...first, turns: [{ id: "one", unknown: { a: 1, b: 3 }, items: [], status: "completed" }] }, + { ...first, turns: [{ id: "one", unknown: { a: 1, b: 2 }, items: [], status: "failed" }] }, + ]) + NodeAssert.notEqual(codexConversationDigest(first), codexConversationDigest(changed)); + }); +}); + +describe("readCodexConversation", () => { + it.effect("preserves unknown turn and item fields, error details, and relationships", () => + Effect.gen(function* () { + const turns = [ + turn("turn-1", { + providerExtension: { callId: "call-1" }, + items: [ + { id: "item-1", type: "futureTool", sourceId: "item-0", result: { rich: [1, "two"] } }, + ], + }), + turn("turn-2", { + status: "failed", + error: { message: "failed", futureDetail: { causeId: "item-1" } }, + }), + ]; + const { client } = makeClient(turns); + NodeAssert.deepEqual((yield* readSource(client)).turns, turns); + }), + ); + + it.effect("reads paginated history in order with full items and rechecks idle metadata", () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.set("source", { turns: [turn("turn-1"), turn("turn-2")], paginated: true }); + const snapshot = yield* readSource(fixture.client); + NodeAssert.deepEqual(snapshot.turns, [turn("turn-1"), turn("turn-2")]); + NodeAssert.deepEqual( + fixture.calls + .filter((call) => call.method === "thread/turns/list") + .map((call) => call.params), + [ + { threadId: "source", cursor: null, limit: 100, sortDirection: "asc", itemsView: "full" }, + { threadId: "source", cursor: "1", limit: 100, sortDirection: "asc", itemsView: "full" }, + ], + ); + NodeAssert.equal(fixture.calls.filter((call) => call.method === "thread/read").length, 2); + }), + ); + + for (const update of [ + { id: "foreign" }, + { cwd: "/tmp/foreign" }, + { ephemeral: true }, + { status: { type: "active", activeFlags: [] } }, + { status: { type: "systemError" } }, + { historyMode: "unknown" }, + ]) { + it.effect(`rejects unverified metadata ${JSON.stringify(update)}`, () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.hooks.response = (_method, _params, response) => replaceThread(response, update); + NodeAssert.equal( + (yield* Effect.flip(readSource(fixture.client)))._tag, + "CodexAppServerRequestError", + ); + }), + ); + } + + for (const turns of [ + [turn("one", { status: "inProgress" })], + [turn("one", { itemsView: "summary" })], + [turn("one", { itemsView: "notLoaded" })], + [turn("one"), turn("one")], + [{ id: "one", status: "completed" }], + [turn("one", { items: [{ type: "agentMessage", text: "missing id" }] })], + [ + turn("one", { + items: [ + { id: "same", type: "plan" }, + { id: "same", type: "plan" }, + ], + }), + ], + ]) { + it.effect(`rejects incomplete or ambiguous history ${JSON.stringify(turns)}`, () => + Effect.gen(function* () { + const { client } = makeClient(turns); + NodeAssert.equal( + (yield* Effect.flip(readSource(client)))._tag, + "CodexAppServerRequestError", + ); + }), + ); + } + + it.effect("rejects pagination cycles before requesting the same page again", () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.set("source", { turns: [], paginated: true }); + fixture.hooks.response = (method, _params, response) => + method === "thread/turns/list" ? { data: [], nextCursor: "cycle" } : response; + NodeAssert.match( + (yield* Effect.flip(readSource(fixture.client))).message, + /repeated a cursor/, + ); + NodeAssert.equal( + fixture.calls.filter((call) => call.method === "thread/turns/list").length, + 2, + ); + }), + ); + + it.effect("rejects a conversation that starts running during pagination", () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.set("source", { turns: [], paginated: true }); + let readCount = 0; + fixture.hooks.response = (method, _params, response) => + method === "thread/read" && ++readCount === 2 + ? replaceThread(response, { status: { type: "active", activeFlags: [] } }) + : response; + NodeAssert.match( + (yield* Effect.flip(readSource(fixture.client))).message, + /persistent and idle/, + ); + }), + ); +}); + +describe("forkCodexConversation", () => { + it.effect("passes captured configuration while preventing source or boundary overrides", () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + yield* forkCodexConversation( + { + ...fixture.client, + forkOptions: { + model: "captured-model", + serviceTier: "fast", + approvalPolicy: "never", + sandbox: "danger-full-access", + threadId: "foreign", + cwd: "/tmp/foreign", + ephemeral: true, + deferGoalContinuation: false, + path: "/tmp/foreign.jsonl", + lastTurnId: "turn-1", + beforeTurnId: "turn-1", + }, + }, + { source }, + ); + NodeAssert.deepEqual(fixture.calls.find((call) => call.method === "thread/fork")?.params, { + model: "captured-model", + serviceTier: "fast", + approvalPolicy: "never", + sandbox: "danger-full-access", + threadId: "source", + cwd: CWD, + ephemeral: false, + deferGoalContinuation: true, + }); + }), + ); + it.effect( + "forks through the exact completed identity inclusively and leaves the source unchanged", + () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + const fork = yield* forkCodexConversation(fixture.client, { source, lastTurnId: "turn-1" }); + NodeAssert.equal(fork.threadId, "fork-1"); + NodeAssert.equal(fork.cwd, CWD); + NodeAssert.deepEqual(fork.turns, [turn("turn-1")]); + NodeAssert.equal(fork.rolloutPath, `${CWD}/fork-1.jsonl`); + NodeAssert.deepEqual(fixture.calls.find((call) => call.method === "thread/fork")?.params, { + threadId: "source", + lastTurnId: "turn-1", + cwd: CWD, + ephemeral: false, + deferGoalContinuation: true, + }); + NodeAssert.deepEqual((yield* readSource(fixture.client)).turns, source.turns); + }), + ); + + it.effect( + "reapplying the same snapshot creates fresh forks with identical semantic history", + () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + const first = yield* forkCodexConversation(fixture.client, { source }); + const second = yield* forkCodexConversation(fixture.client, { source }); + NodeAssert.notEqual(first.threadId, second.threadId); + NodeAssert.equal(codexConversationDigest(first), codexConversationDigest(second)); + NodeAssert.equal(codexConversationDigest(first), codexConversationDigest(source)); + }), + ); + + it.effect("forks an already empty root without treating an omitted boundary as deletion", () => + Effect.gen(function* () { + const fixture = makeClient([]); + const source = yield* readSource(fixture.client); + NodeAssert.deepEqual((yield* forkCodexConversation(fixture.client, { source })).turns, []); + NodeAssert.deepEqual(fixture.calls.find((call) => call.method === "thread/fork")?.params, { + threadId: "source", + cwd: CWD, + ephemeral: false, + deferGoalContinuation: true, + }); + }), + ); + + for (const boundary of ["missing", "failed", "interrupted"]) { + it.effect(`rejects the non-completed boundary ${boundary} before issuing a fork`, () => + Effect.gen(function* () { + const fixture = makeClient([ + turn("completed"), + turn("failed", { status: "failed" }), + turn("interrupted", { status: "interrupted" }), + ]); + const source = yield* readSource(fixture.client); + NodeAssert.match( + (yield* Effect.flip( + forkCodexConversation(fixture.client, { source, lastTurnId: boundary }), + )).message, + /exact completed turn/, + ); + NodeAssert.equal( + fixture.calls.some((call) => call.method === "thread/fork"), + false, + ); + }), + ); + } + + for (const update of [ + { id: "source" }, + { forkedFromId: "foreign" }, + { cwd: "/tmp/other" }, + { ephemeral: true }, + { status: { type: "active", activeFlags: [] } }, + { status: { type: "notLoaded" } }, + ]) { + it.effect(`rejects fork metadata ${JSON.stringify(update)}`, () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, _params, response) => + method === "thread/fork" ? replaceThread(response, update) : response; + NodeAssert.equal( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source })))._tag, + "CodexAppServerRequestError", + ); + }), + ); + } + + for (const changed of [ + [turn("renamed"), turn("turn-2")], + [ + turn("turn-1", { + items: [ + { + id: "renamed-item", + type: "userMessage", + content: [{ type: "text", text: "same prompt" }], + }, + ], + }), + turn("turn-2"), + ], + [ + turn("turn-1", { + items: [ + { + id: "turn-1-user", + type: "userMessage", + content: [{ type: "text", text: "changed prompt" }], + }, + ], + }), + turn("turn-2"), + ], + [turn("turn-2"), turn("turn-1")], + [turn("turn-1", { futureRelationship: "foreign-item" }), turn("turn-2")], + ]) { + it.effect(`rejects equal-length fork history changes ${JSON.stringify(changed)}`, () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, _params, response) => + method === "thread/fork" ? replaceThread(response, { turns: changed }) : response; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /changed retained history/, + ); + }), + ); + } + + it.effect("does not trust a correct fork response when rereading reveals different history", () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, params, response) => + method === "thread/read" && params.threadId === "fork-1" && params.includeTurns + ? replaceThread(response, { turns: [turn("foreign"), turn("turn-2")] }) + : response; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /changed retained turn content or identities/, + ); + }), + ); + + it.effect("rejects a source changed since capture before creating a fork", () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + fixture.threads.set("source", { turns: [turn("turn-1"), turn("turn-2"), turn("turn-3")] }); + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /source changed after capture/, + ); + NodeAssert.equal( + fixture.calls.some((call) => call.method === "thread/fork"), + false, + ); + }), + ); + + it.effect("rejects a source changed during fork verification", () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, _params, response) => { + if (method === "thread/fork") + fixture.threads.set("source", { + turns: [turn("turn-1"), turn("turn-2"), turn("turn-3")], + }); + return response; + }; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /source changed during fork verification/, + ); + }), + ); + + it.effect("verifies paginated forks using full history despite empty response turns", () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.set("source", { turns: [turn("turn-1"), turn("turn-2")], paginated: true }); + const source = yield* readSource(fixture.client); + const fork = yield* forkCodexConversation(fixture.client, { source, lastTurnId: "turn-1" }); + NodeAssert.deepEqual(fork.turns, [turn("turn-1")]); + }), + ); +}); + +describe("native conversation proof", () => { + const hiddenContexts = [ + nativeRecord("compacted", { + replacement_history: [ + { type: "message", role: "developer", content: "retained native memory" }, + ], + window_number: 2, + }), + nativeRecord("turn_context", { + turn_id: "turn-2", + model: "codex", + developer_instructions: "retained native instructions", + future: { enabled: true }, + }), + nativeRecord("world_state", { + full: true, + state: { files: { "private.txt": "native baseline" } }, + }), + nativeRecord("inter_agent_communication", { + sender_thread_id: "child-1", + receiver_thread_id: "source", + trigger_turn: false, + content: "native child message", + }), + nativeRecord("future_native_context", { + relationships: ["item-1", "child-1"], + inference: { hidden: "future context" }, + }), + ]; + + it.effect("reads unloaded persistent snapshots after restart without resuming them", () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.hooks.response = (method, _params, response) => + method === "thread/read" + ? replaceThread(response, { status: { type: "notLoaded" } }) + : response; + const source = yield* readSource(fixture.client); + NodeAssert.equal(source.turns.length, 2); + NodeAssert.equal( + fixture.calls.some((call) => call.method === "thread/resume"), + false, + ); + NodeAssert.match( + (yield* Effect.flip( + readCodexConversation(fixture.client, "source", CWD, { requireLoaded: true }), + )).message, + /persistent and idle/, + ); + }), + ); + + it.effect("accepts regenerated native Git metadata but retains unknown Git context", () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, _params, response) => { + if (method === "thread/fork") { + const fork = fixture.threads.get("fork-1")!; + const header = nativeSession("fork-1", "source"); + const payload = decodeNativeObject(header.payload); + fork.nativeRecords = [ + { + ...header, + payload: { + ...payload, + git: { commit_hash: "new-checkout", branch: "restored", repository_url: "example" }, + }, + }, + ...fork.nativeRecords!.slice(1), + ]; + } + return response; + }; + NodeAssert.equal( + codexConversationDigest(yield* forkCodexConversation(fixture.client, { source })), + codexConversationDigest(source), + ); + fixture.hooks.rollout = (path, text) => + path.endsWith("fork-2.jsonl") + ? text.replace('"context_window":', '"git":{"futureContext":"changed"},"context_window":') + : text; + delete fixture.hooks.response; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /changed retained turn content or identities/, + ); + }), + ); + + for (const hidden of hiddenContexts) { + it.effect(`includes ${String(hidden.type)} context omitted by the public transcript`, () => + Effect.gen(function* () { + const fixture = makeClient(); + const sourceThread = fixture.threads.get("source")!; + sourceThread.nativeRecords = [...nativeHistory("source", sourceThread.turns), hidden]; + const source = yield* readSource(fixture.client); + const forked = yield* forkCodexConversation(fixture.client, { source }); + NodeAssert.equal(codexConversationDigest(forked), codexConversationDigest(source)); + NodeAssert.deepEqual(forked.nativeRecords.at(-1), { + type: hidden.type, + payload: hidden.payload, + }); + fixture.hooks.response = (method, _params, response) => { + if (method === "thread/fork") { + const fork = fixture.threads.get("fork-2")!; + fork.nativeRecords = [ + ...fork.nativeRecords!.slice(0, -1), + nativeRecord(String(hidden.type), { changedInference: "same public transcript" }), + ]; + } + return response; + }; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /changed retained turn content or identities/, + ); + }), + ); + } + + it.effect( + "retains opaque records after the terminal event through the exact next native start", + () => + Effect.gen(function* () { + const fixture = makeClient(); + const sourceThread = fixture.threads.get("source")!; + const native = nativeHistory("source", sourceThread.turns); + const interturn = nativeRecord("world_state", { + full: false, + state: { retainedBetweenTurns: true }, + }); + sourceThread.nativeRecords = [...native.slice(0, 5), interturn, ...native.slice(5)]; + const source = yield* readSource(fixture.client); + const fork = yield* forkCodexConversation(fixture.client, { source, lastTurnId: "turn-1" }); + NodeAssert.deepEqual(fork.nativeRecords.at(-1), { + type: interturn.type, + payload: interturn.payload, + }); + NodeAssert.deepEqual(fork.turns, [turn("turn-1")]); + }), + ); + + it.effect("does not infer a native boundary from synthetic public turn identities", () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.get("source")!.nativeRecords = [nativeSession("source")]; + const source = yield* readSource(fixture.client); + NodeAssert.match( + (yield* Effect.flip( + forkCodexConversation(fixture.client, { source, lastTurnId: "turn-1" }), + )).message, + /native boundary/, + ); + NodeAssert.equal( + fixture.calls.some((call) => call.method === "thread/fork"), + false, + ); + }), + ); + + it.effect("detects native-only source edits before creating a private fork", () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + fixture.threads.get("source")!.nativeRecords = [ + ...nativeHistory("source", source.turns), + hiddenContexts[0]!, + ]; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /source changed after capture/, + ); + NodeAssert.equal( + fixture.calls.some((call) => call.method === "thread/fork"), + false, + ); + }), + ); + + it.effect("detects native-only source edits made while forking", () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, _params, response) => { + if (method === "thread/fork") + fixture.threads.get("source")!.nativeRecords = [ + ...nativeHistory("source", source.turns), + hiddenContexts[0]!, + ]; + return response; + }; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /source changed during fork verification/, + ); + }), + ); + + it.effect("detects a native file changing across the public history read", () => + Effect.gen(function* () { + const fixture = makeClient(); + let reads = 0; + fixture.hooks.rollout = (_path, text) => + ++reads === 1 ? text : `${text}${JSON.stringify(hiddenContexts[0])}\n`; + NodeAssert.match( + (yield* Effect.flip(readSource(fixture.client))).message, + /native history changed while reading/, + ); + }), + ); + + for (const metadata of [ + { id: "foreign" }, + { cwd: "/tmp/foreign" }, + { history_base: { thread_id: "unread-base", end_ordinal_exclusive: 4, end_byte_offset: 100 } }, + { context_window: { window_id: 7 } }, + ]) + it.effect(`rejects unprovable native session metadata ${JSON.stringify(metadata)}`, () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.get("source")!.nativeRecords = [ + nativeRecord("session_meta", { + id: "source", + session_id: "source", + timestamp: "2026-01-01T00:00:00.000Z", + cwd: CWD, + ...metadata, + }), + ]; + NodeAssert.equal( + (yield* Effect.flip(readSource(fixture.client)))._tag, + "CodexAppServerRequestError", + ); + }), + ); + + it.effect("fails closed without a provider-owned rollout reader", () => + Effect.gen(function* () { + const fixture = makeClient(); + NodeAssert.match( + (yield* Effect.flip(readSource({ raw: fixture.client.raw }))).message, + /proof is unavailable/, + ); + }), + ); + + for (const path of [null, "relative.jsonl", "", "/tmp/invalid\0.jsonl"]) + it.effect(`rejects native path ${JSON.stringify(path)}`, () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.hooks.response = (method, _params, response) => + method === "thread/read" ? replaceThread(response, { path }) : response; + NodeAssert.match( + (yield* Effect.flip(readSource(fixture.client))).message, + /proof is unavailable/, + ); + }), + ); + + for (const [label, invalid] of [ + ["truncated", '{"type":"session_meta"}'], + ["malformed", "not JSON\n"], + ["empty", "\n"], + ["missing envelope", "{}\n"], + ["oversized line", `${" ".repeat(1024 * 1024 + 1)}\n`], + ["oversized file", `${" ".repeat(16 * 1024 * 1024)}\n`], + ["too many records", "{}\n".repeat(100_001)], + ]) + it.effect(`rejects ${label} native JSONL without disclosing content`, () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.hooks.rollout = () => invalid!; + const error = yield* Effect.flip(readSource(fixture.client)); + NodeAssert.equal(error._tag, "CodexAppServerRequestError"); + NodeAssert.doesNotMatch(error.message, /native system instructions|retained native memory/); + }), + ); + + it.effect("preserves unknown header context instead of treating it as generated identity", () => + Effect.gen(function* () { + const fixture = makeClient(); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, _params, response) => { + if (method === "thread/fork") { + const fork = fixture.threads.get("fork-1")!; + const first = fork.nativeRecords![0]!; + fork.nativeRecords = [ + { ...first, providerInferenceExtension: { changed: true } }, + ...fork.nativeRecords!.slice(1), + ]; + } + return response; + }; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /changed retained turn content or identities/, + ); + }), + ); +}); + +describe("native goal proof", () => { + const goal = (status: string): Schema.JsonObject => ({ + threadId: "source", + objective: "preserve goal", + status, + tokenBudget: 100, + tokensUsed: 10, + timeUsedSeconds: 2, + createdAt: 1, + updatedAt: 2, + unknown: { futureGoalState: true }, + }); + + for (const status of ["paused", "blocked", "usageLimited", "budgetLimited", "complete"]) + it.effect(`retains an idle ${status} goal through native deferred copying`, () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.get("source")!.nativeGoal = goal(status); + const source = yield* readSource(fixture.client); + const fork = yield* forkCodexConversation(fixture.client, { source }); + NodeAssert.deepEqual(fork.nativeGoal, source.nativeGoal); + NodeAssert.equal(codexConversationDigest(fork), codexConversationDigest(source)); + NodeAssert.equal( + typeof fork.nativeGoal === "object" && + fork.nativeGoal !== null && + "threadId" in fork.nativeGoal, + false, + ); + }), + ); + + for (const nativeGoal of [ + goal("active"), + goal("futureActive"), + { ...goal("paused"), threadId: "foreign" }, + { threadId: "source", status: "paused" }, + { ...goal("paused"), tokensUsed: "unknown" }, + ]) + it.effect(`rejects unproven native goal ${JSON.stringify(nativeGoal)}`, () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.get("source")!.nativeGoal = nativeGoal; + NodeAssert.equal( + (yield* Effect.flip(readSource(fixture.client)))._tag, + "CodexAppServerRequestError", + ); + }), + ); + + it.effect( + "does not establish exact eligibility when native goal introspection is unsupported", + () => + Effect.gen(function* () { + const fixture = makeClient(); + const unsupported: Parameters[0] = { + ...fixture.client, + raw: { + request: (method, params) => + method === "thread/goal/get" + ? Effect.fail(CodexErrors.CodexAppServerRequestError.internalError("unsupported")) + : fixture.client.raw.request(method, params), + }, + }; + NodeAssert.equal( + (yield* Effect.flip(readSource(unsupported)))._tag, + "CodexAppServerRequestError", + ); + }), + ); + + it.effect("detects changed native goal state with identical public turns and JSONL", () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.get("source")!.nativeGoal = goal("paused"); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, _params, response) => { + if (method === "thread/fork") + fixture.threads.get("fork-1")!.nativeGoal = { + ...goal("paused"), + threadId: "fork-1", + tokensUsed: 11, + }; + return response; + }; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /changed retained turn content or identities/, + ); + }), + ); + + it.effect("detects a native goal appearing while history is read", () => + Effect.gen(function* () { + const fixture = makeClient(); + let reads = 0; + fixture.hooks.response = (method, _params, response) => + method === "thread/goal/get" && ++reads === 2 ? { goal: goal("paused") } : response; + NodeAssert.match( + (yield* Effect.flip(readSource(fixture.client))).message, + /native goal changed while reading/, + ); + }), + ); + + it.effect("rejects dropped or changed unknown goal state even when history matches", () => + Effect.gen(function* () { + for (const changed of [ + null, + { ...goal("paused"), threadId: "fork-1", unknown: { futureGoalState: false } }, + ]) { + const fixture = makeClient(); + fixture.threads.get("source")!.nativeGoal = goal("paused"); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, _params, response) => { + if (method === "thread/fork") fixture.threads.get("fork-1")!.nativeGoal = changed; + return response; + }; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /changed retained turn content or identities/, + ); + } + }), + ); + + it.effect("rejects source goal edits during native fork verification", () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.get("source")!.nativeGoal = goal("paused"); + const source = yield* readSource(fixture.client); + fixture.hooks.response = (method, _params, response) => { + if (method === "thread/fork") + fixture.threads.get("source")!.nativeGoal = { ...goal("paused"), tokensUsed: 12 }; + return response; + }; + NodeAssert.match( + (yield* Effect.flip(forkCodexConversation(fixture.client, { source }))).message, + /source changed during fork verification/, + ); + }), + ); + + it.effect("does not infer prior goal state from an earlier public turn", () => + Effect.gen(function* () { + const fixture = makeClient(); + fixture.threads.get("source")!.nativeGoal = goal("paused"); + const source = yield* readSource(fixture.client); + NodeAssert.match( + (yield* Effect.flip( + forkCodexConversation(fixture.client, { source, lastTurnId: "turn-1" }), + )).message, + /historical goal state/, + ); + NodeAssert.equal( + fixture.calls.some((call) => call.method === "thread/fork"), + false, + ); + }), + ); +}); diff --git a/apps/server/src/provider/Layers/CodexAbsoluteHistory.ts b/apps/server/src/provider/Layers/CodexAbsoluteHistory.ts new file mode 100644 index 000000000..9336182fc --- /dev/null +++ b/apps/server/src/provider/Layers/CodexAbsoluteHistory.ts @@ -0,0 +1,583 @@ +// @effect-diagnostics nodeBuiltinImport:off +// Node builtins here are pure digest/path operations; runtime-owned Effect services perform I/O. +import * as NodeCrypto from "node:crypto"; +import * as NodePath from "node:path"; + +import * as Effect from "effect/Effect"; +import * as Schema from "effect/Schema"; +import type * as CodexClient from "effect-codex-app-server/client"; +import * as CodexErrors from "effect-codex-app-server/errors"; + +export interface CodexConversationSnapshot { + readonly threadId: string; + readonly cwd: string; + readonly turns: ReadonlyArray; + readonly rolloutPath: string; + readonly nativeRecords: ReadonlyArray; + readonly nativeGoal: Schema.Json; +} + +type CodexHistoryClient = { + readonly raw: Pick; + readonly readRollout?: (path: string) => Effect.Effect; + readonly forkOptions?: Readonly>; +}; + +const CODEX_ROLLOUT_MAX_BYTES = 16 * 1024 * 1024; +const MAX_ROLLOUT_RECORDS = 100_000; +const MAX_ROLLOUT_LINE_BYTES = 1024 * 1024; +const RESERVED_FORK_OPTIONS = new Set([ + "threadId", + "cwd", + "ephemeral", + "deferGoalContinuation", + "path", + "lastTurnId", + "beforeTurnId", +]); + +const ConversationMetadata = Schema.Struct({ + thread: Schema.Struct({ + id: Schema.NonEmptyString, + cwd: Schema.NonEmptyString, + ephemeral: Schema.Boolean, + status: Schema.Struct({ type: Schema.String }), + historyMode: Schema.optionalKey(Schema.Literals(["legacy", "paginated"])), + forkedFromId: Schema.optionalKey(Schema.NullOr(Schema.String)), + path: Schema.optionalKey(Schema.NullOr(Schema.String)), + // Json keeps every native field, including fields newer than the pinned protocol. + turns: Schema.optionalKey(Schema.Array(Schema.Json)), + }), + cwd: Schema.optionalKey(Schema.String), +}); +const ConversationTurn = Schema.Struct({ + id: Schema.NonEmptyString, + status: Schema.Literals(["completed", "interrupted", "failed", "inProgress"]), + itemsView: Schema.optionalKey(Schema.Literals(["full", "summary", "notLoaded"])), + items: Schema.Array(Schema.Struct({ id: Schema.NonEmptyString, type: Schema.NonEmptyString })), +}); +const ConversationTurnsPage = Schema.Struct({ + data: Schema.Array(Schema.Json), + nextCursor: Schema.NullOr(Schema.String), +}); + +const decodeMetadata = Schema.decodeUnknownEffect(ConversationMetadata); +const decodeTurn = Schema.decodeUnknownEffect(ConversationTurn); +const decodeTurnsPage = Schema.decodeUnknownEffect(ConversationTurnsPage); +const decodeNativeRecord = Schema.decodeUnknownEffect( + Schema.fromJsonString(Schema.Record(Schema.String, Schema.Json)), +); +const decodeNativeEnvelope = Schema.decodeUnknownEffect( + Schema.Struct({ + timestamp: Schema.NonEmptyString, + ordinal: Schema.optionalKey(Schema.Number), + type: Schema.NonEmptyString, + payload: Schema.Record(Schema.String, Schema.Json), + }), +); +const decodeNativeSession = Schema.decodeUnknownEffect( + Schema.Struct({ + id: Schema.NonEmptyString, + cwd: Schema.NonEmptyString, + timestamp: Schema.NonEmptyString, + session_id: Schema.optionalKey(Schema.NonEmptyString), + forked_from_id: Schema.optionalKey(Schema.NullOr(Schema.NonEmptyString)), + context_window: Schema.optionalKey( + Schema.NullOr(Schema.Struct({ window_id: Schema.NonEmptyString })), + ), + }), +); +const decodeNativeGoal = Schema.decodeUnknownEffect( + Schema.Struct({ + goal: Schema.NullOr(Schema.Record(Schema.String, Schema.Json)), + }), +); +const decodeIdleNativeGoal = Schema.decodeUnknownEffect( + Schema.Struct({ + threadId: Schema.NonEmptyString, + status: Schema.Literals(["paused", "blocked", "usageLimited", "budgetLimited", "complete"]), + objective: Schema.NonEmptyString, + tokenBudget: Schema.NullOr(Schema.Int), + tokensUsed: Schema.Int, + timeUsedSeconds: Schema.Int, + createdAt: Schema.Int, + updatedAt: Schema.Int, + }), +); +const isNativeTurnStart = Schema.is( + Schema.Struct({ + type: Schema.Literal("event_msg"), + payload: Schema.Struct({ + type: Schema.Literals(["task_started", "turn_started"]), + turn_id: Schema.NonEmptyString, + }), + }), +); +const isNativeContextWindow = Schema.is(Schema.Record(Schema.String, Schema.Json)); + +const invalidHistory = (method: string, message: string) => + CodexErrors.CodexAppServerRequestError.internalError(message, undefined, { + method, + operation: "decode-payload", + }); + +const validateMetadata = Effect.fn("CodexAbsoluteHistory.validateMetadata")(function* ( + response: unknown, + method: string, + threadId: string | undefined, + cwd: string, + requireLoaded = false, +) { + const metadata = yield* decodeMetadata(response).pipe( + Effect.mapError((cause) => + CodexErrors.CodexAppServerRequestError.invalidPayload(method, "decode-payload", cause), + ), + ); + if ( + (threadId !== undefined && metadata.thread.id !== threadId) || + metadata.thread.cwd !== cwd || + (metadata.cwd !== undefined && metadata.cwd !== cwd) + ) { + return yield* invalidHistory( + method, + "Codex conversation identity or working directory changed.", + ); + } + if ( + metadata.thread.ephemeral || + (metadata.thread.status.type !== "idle" && + (requireLoaded || metadata.thread.status.type !== "notLoaded")) + ) { + return yield* invalidHistory(method, "Codex conversation must be persistent and idle."); + } + return metadata.thread; +}); + +const validateTurns = Effect.fn("CodexAbsoluteHistory.validateTurns")(function* ( + turns: ReadonlyArray, + method: string, +) { + const ids = new Set(); + const decoded = []; + for (const turn of turns) { + const metadata = yield* decodeTurn(turn).pipe( + Effect.mapError((cause) => + CodexErrors.CodexAppServerRequestError.invalidPayload(method, "decode-payload", cause), + ), + ); + if ( + metadata.status === "inProgress" || + (metadata.itemsView !== undefined && metadata.itemsView !== "full") || + ids.has(metadata.id) + ) { + return yield* invalidHistory( + method, + "Codex conversation history is incomplete or ambiguous.", + ); + } + if (new Set(metadata.items.map((item) => item.id)).size !== metadata.items.length) { + return yield* invalidHistory( + method, + "Codex conversation contains duplicate item identities.", + ); + } + ids.add(metadata.id); + decoded.push(metadata); + } + return decoded; +}); + +function canonicalJson(value: Schema.Json): string { + if (value === null || typeof value !== "object") { + return JSON.stringify(value); + } + if (Array.isArray(value)) { + return `[${value.map(canonicalJson).join(",")}]`; + } + return `{${Object.entries(value) + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)) + .map(([key, entry]) => `${JSON.stringify(key)}:${canonicalJson(entry)}`) + .join(",")}}`; +} + +const readNativeRecords = Effect.fn("CodexAbsoluteHistory.readNativeRecords")(function* ( + client: CodexHistoryClient, + path: string | null | undefined, + threadId: string, + cwd: string, + expectedParent?: string, +) { + if (!client.readRollout || !path || !NodePath.isAbsolute(path) || path.includes("\0")) + return yield* invalidHistory("thread/read", "Codex native history proof is unavailable."); + const text = yield* client.readRollout(path); + if (Buffer.byteLength(text, "utf8") > CODEX_ROLLOUT_MAX_BYTES || !text.endsWith("\n")) + return yield* invalidHistory("thread/read", "Codex native history is oversized or incomplete."); + const lines = text.slice(0, -1).split("\n"); + if (lines.length > MAX_ROLLOUT_RECORDS) + return yield* invalidHistory("thread/read", "Codex native history exceeds the record bound."); + const records: Array = []; + let leadingSessionHeaders = true; + for (const [index, line] of lines.entries()) { + if (!line.trim() || Buffer.byteLength(line, "utf8") > MAX_ROLLOUT_LINE_BYTES) + return yield* invalidHistory( + "thread/read", + "Codex native history contains an invalid record.", + ); + const record = yield* decodeNativeRecord(line).pipe( + Effect.mapError(() => + invalidHistory("thread/read", "Codex native history is not valid JSONL."), + ), + ); + const envelope = yield* decodeNativeEnvelope(record).pipe( + Effect.mapError(() => + invalidHistory("thread/read", "Codex native history has an invalid envelope."), + ), + ); + if (index === 0 && envelope.type !== "session_meta") + return yield* invalidHistory( + "thread/read", + "Codex native history omitted its session identity.", + ); + if ( + envelope.ordinal !== undefined && + (!Number.isSafeInteger(envelope.ordinal) || envelope.ordinal < 0) + ) + return yield* invalidHistory("thread/read", "Codex native history has an invalid ordinal."); + let payload = envelope.payload; + if (envelope.type === "session_meta") { + const session = yield* decodeNativeSession(payload).pipe( + Effect.mapError(() => + invalidHistory("thread/read", "Codex native history has an invalid session identity."), + ), + ); + if (index === 0 && (session.id !== threadId || session.cwd !== cwd)) + return yield* invalidHistory( + "thread/read", + "Codex native history belongs to another conversation.", + ); + if (index === 0 && expectedParent !== undefined && session.forked_from_id !== expectedParent) + return yield* invalidHistory( + "thread/read", + "Codex native history has unverified fork provenance.", + ); + if (payload.history_base != null) + return yield* invalidHistory( + "thread/read", + "Codex native history depends on an unverified external base.", + ); + payload = Object.fromEntries( + Object.entries(payload).filter( + ([key]) => !["id", "session_id", "forked_from_id", "timestamp"].includes(key), + ), + ); + if (isNativeContextWindow(payload.context_window)) { + const { context_window: _contextWindow, ...rest } = payload; + const contextWindow = Object.fromEntries( + Object.entries(payload.context_window).filter(([key]) => key !== "window_id"), + ); + payload = { + ...rest, + ...(Object.keys(contextWindow).length === 0 ? {} : { context_window: contextWindow }), + }; + } + if (isNativeContextWindow(payload.git)) { + const { git: _git, ...rest } = payload; + // recorder::write_session_meta recollects these informational fields from the current + // checkout. Unknown Git fields remain part of the proof. + const git = Object.fromEntries( + Object.entries(payload.git).filter( + ([key]) => !["commit_hash", "branch", "repository_url"].includes(key), + ), + ); + payload = { ...rest, ...(Object.keys(git).length === 0 ? {} : { git }) }; + } + } + // The pinned native recorder rewrites only these enclosing fields when replaying a fork. + // Payload timestamps, context, tool relationships, and every unknown field remain semantic. + const normalized = { + ...Object.fromEntries( + Object.entries(record).filter( + ([key]) => !["timestamp", "ordinal", "payload"].includes(key), + ), + ), + payload, + }; + // A native fork prepends its own session header to the copied history. Only an identical + // normalized adjacent header can be elided; differences in context must fail verification. + if ( + leadingSessionHeaders && + envelope.type === "session_meta" && + records.length > 0 && + canonicalJson(records.at(-1)!) === canonicalJson(normalized) + ) + continue; + if (envelope.type !== "session_meta") leadingSessionHeaders = false; + records.push(normalized); + } + return { path, records, text }; +}); + +/** Public identities and full native inference context are semantic; native file identity is not. */ +export function codexConversationDigest(snapshot: CodexConversationSnapshot): string { + return NodeCrypto.createHash("sha256") + .update( + canonicalJson({ + cwd: snapshot.cwd, + turns: snapshot.turns, + nativeRecords: snapshot.nativeRecords, + nativeGoal: snapshot.nativeGoal, + }), + ) + .digest("hex"); +} + +const readNativeGoal = Effect.fn("CodexAbsoluteHistory.readNativeGoal")(function* ( + client: CodexHistoryClient, + threadId: string, +) { + const { goal } = yield* decodeNativeGoal( + yield* client.raw.request("thread/goal/get", { threadId }), + ).pipe( + Effect.mapError(() => + invalidHistory("thread/goal/get", "Codex native goal proof is unavailable."), + ), + ); + if (goal === null) return null; + const metadata = yield* decodeIdleNativeGoal(goal).pipe( + Effect.mapError(() => + invalidHistory("thread/goal/get", "Codex native goal is not provably idle."), + ), + ); + if (metadata.threadId !== threadId) + return yield* invalidHistory( + "thread/goal/get", + "Codex native goal belongs to another conversation.", + ); + return Object.fromEntries(Object.entries(goal).filter(([key]) => key !== "threadId")); +}); + +export const readCodexConversation = Effect.fn("readCodexConversation")(function* ( + client: CodexHistoryClient, + threadId: string, + cwd: string, + options?: { readonly requireLoaded?: boolean }, +): Effect.fn.Return { + const metadata = yield* validateMetadata( + yield* client.raw.request("thread/read", { threadId, includeTurns: false }), + "thread/read", + threadId, + cwd, + options?.requireLoaded, + ); + const nativeGoal = yield* readNativeGoal(client, threadId); + const native = yield* readNativeRecords( + client, + metadata.path, + threadId, + cwd, + metadata.forkedFromId ?? undefined, + ); + const turns: Array = []; + if (metadata.historyMode !== "paginated") { + const history = yield* validateMetadata( + yield* client.raw.request("thread/read", { threadId, includeTurns: true }), + "thread/read", + threadId, + cwd, + options?.requireLoaded, + ); + if ( + history.historyMode === "paginated" || + history.turns === undefined || + history.path !== native.path + ) { + return yield* invalidHistory( + "thread/read", + "Codex conversation did not return full history.", + ); + } + turns.push(...history.turns); + } else { + const requestedCursors = new Set(); + let cursor: string | null = null; + do { + if (requestedCursors.has(cursor)) { + return yield* invalidHistory( + "thread/turns/list", + "Codex history pagination repeated a cursor.", + ); + } + requestedCursors.add(cursor); + const page: typeof ConversationTurnsPage.Type = yield* decodeTurnsPage( + yield* client.raw.request("thread/turns/list", { + threadId, + cursor, + limit: 100, + sortDirection: "asc", + itemsView: "full", + }), + ).pipe( + Effect.mapError((cause) => + CodexErrors.CodexAppServerRequestError.invalidPayload( + "thread/turns/list", + "decode-payload", + cause, + ), + ), + ); + turns.push(...page.data); + cursor = page.nextCursor; + } while (cursor !== null); + } + yield* validateTurns(turns, "thread/read"); + const after = yield* validateMetadata( + yield* client.raw.request("thread/read", { threadId, includeTurns: false }), + "thread/read", + threadId, + cwd, + options?.requireLoaded, + ); + if (after.historyMode !== metadata.historyMode || after.path !== native.path) + return yield* invalidHistory( + "thread/read", + "Codex conversation history scope changed while reading.", + ); + const verifiedGoal = yield* readNativeGoal(client, threadId); + if (canonicalJson(verifiedGoal) !== canonicalJson(nativeGoal)) + return yield* invalidHistory("thread/goal/get", "Codex native goal changed while reading."); + const verifiedNative = yield* readNativeRecords( + client, + after.path, + threadId, + cwd, + after.forkedFromId ?? undefined, + ); + if (verifiedNative.text !== native.text) + return yield* invalidHistory("thread/read", "Codex native history changed while reading."); + return { + threadId, + cwd, + turns, + rolloutPath: native.path, + nativeRecords: native.records, + nativeGoal, + }; +}); + +export const forkCodexConversation = Effect.fn("forkCodexConversation")(function* ( + client: CodexHistoryClient, + input: { readonly source: CodexConversationSnapshot; readonly lastTurnId?: string }, +): Effect.fn.Return { + const { source, lastTurnId } = input; + const sourceTurns = yield* validateTurns(source.turns, "thread/fork"); + const boundaryIndex = + lastTurnId === undefined + ? sourceTurns.length - 1 + : sourceTurns.findIndex((turn) => turn.id === lastTurnId); + if ( + lastTurnId !== undefined && + (boundaryIndex < 0 || sourceTurns[boundaryIndex]?.status !== "completed") + ) { + return yield* invalidHistory( + "thread/fork", + "Codex fork target must identify an exact completed turn.", + ); + } + if (source.nativeGoal !== null && boundaryIndex !== sourceTurns.length - 1) + return yield* invalidHistory( + "thread/fork", + "Codex historical goal state has no verified native boundary.", + ); + let nativeBoundary = source.nativeRecords.length; + if (lastTurnId !== undefined) { + // Pinned truncate_rollout_after_turn_id keeps every record through the next explicit + // TurnStarted boundary. Public item counts cannot identify this native context boundary. + const starts = source.nativeRecords.flatMap((record, index) => + isNativeTurnStart(record) ? [{ index, turnId: record.payload.turn_id }] : [], + ); + const nativeStart = starts.findIndex(({ turnId }) => turnId === lastTurnId); + if (nativeStart < 0 || starts.filter(({ turnId }) => turnId === lastTurnId).length !== 1) + return yield* invalidHistory( + "thread/fork", + "Codex fork target has no unambiguous native boundary.", + ); + nativeBoundary = starts[nativeStart + 1]?.index ?? source.nativeRecords.length; + } + const expected = { + ...source, + turns: source.turns.slice(0, boundaryIndex + 1), + nativeRecords: source.nativeRecords.slice(0, nativeBoundary), + }; + const expectedDigest = codexConversationDigest(expected); + const sourceDigest = codexConversationDigest(source); + const before = yield* readCodexConversation(client, source.threadId, source.cwd); + if ( + before.rolloutPath !== source.rolloutPath || + codexConversationDigest(before) !== sourceDigest + ) { + return yield* invalidHistory("thread/fork", "Codex fork source changed after capture."); + } + const forked = yield* validateMetadata( + yield* client.raw.request("thread/fork", { + ...Object.fromEntries( + Object.entries(client.forkOptions ?? {}).filter(([key]) => !RESERVED_FORK_OPTIONS.has(key)), + ), + threadId: source.threadId, + ...(lastTurnId !== undefined ? { lastTurnId } : {}), + cwd: source.cwd, + ephemeral: false, + deferGoalContinuation: true, + }), + "thread/fork", + undefined, + source.cwd, + true, + ); + if ( + forked.id === source.threadId || + forked.path === source.rolloutPath || + (forked.forkedFromId != null && forked.forkedFromId !== source.threadId) + ) { + return yield* invalidHistory( + "thread/fork", + "Codex fork did not return a distinct private conversation.", + ); + } + yield* readNativeRecords(client, forked.path, forked.id, source.cwd, source.threadId); + // Paginated forks may omit their turns from this response; the complete reread is authoritative. + if (forked.historyMode !== "paginated") { + if (forked.turns === undefined) { + return yield* invalidHistory("thread/fork", "Codex fork response omitted retained history."); + } + yield* validateTurns(forked.turns, "thread/fork"); + if ( + codexConversationDigest({ ...expected, threadId: forked.id, turns: forked.turns }) !== + expectedDigest + ) { + return yield* invalidHistory("thread/fork", "Codex fork response changed retained history."); + } + } + const snapshot = yield* readCodexConversation(client, forked.id, source.cwd, { + requireLoaded: true, + }); + const retainedTurns = yield* validateTurns(snapshot.turns, "thread/fork"); + if ( + retainedTurns.length !== boundaryIndex + 1 || + snapshot.rolloutPath !== forked.path || + retainedTurns.some((turn, index) => turn.id !== sourceTurns[index]?.id) || + codexConversationDigest(snapshot) !== expectedDigest + ) { + return yield* invalidHistory( + "thread/fork", + "Codex fork changed retained turn content or identities.", + ); + } + const after = yield* readCodexConversation(client, source.threadId, source.cwd); + if (after.rolloutPath !== source.rolloutPath || codexConversationDigest(after) !== sourceDigest) { + return yield* invalidHistory( + "thread/fork", + "Codex fork source changed during fork verification.", + ); + } + return snapshot; +}); diff --git a/apps/server/src/provider/Layers/CodexAbsoluteRollback.test.ts b/apps/server/src/provider/Layers/CodexAbsoluteRollback.test.ts new file mode 100644 index 000000000..c4a3e9ca2 --- /dev/null +++ b/apps/server/src/provider/Layers/CodexAbsoluteRollback.test.ts @@ -0,0 +1,477 @@ +// @effect-diagnostics nodeBuiltinImport:off +import * as NodeAssert from "node:assert/strict"; +import { + CheckpointRef, + ProviderInstanceId, + RuntimeSessionId, + ThreadId, + TurnId, +} from "@t3tools/contracts"; +import { describe, it } from "@effect/vitest"; +import * as Effect from "effect/Effect"; +import * as Exit from "effect/Exit"; +import * as Schema from "effect/Schema"; +import * as CodexErrors from "effect-codex-app-server/errors"; +import { makeCodexAbsoluteRollback, readCodexExactCursor } from "./CodexAbsoluteRollback.ts"; +import type { CodexConversationSnapshot } from "./CodexAbsoluteHistory.ts"; +import type { CodexSessionRuntimeShape } from "./CodexSessionRuntime.ts"; +import { ProviderAdapterValidationError } from "../Errors.ts"; +import { ProviderDriverKind } from "@t3tools/contracts"; + +const decodeSnapshotTurns = Schema.decodeUnknownEffect( + Schema.Struct({ snapshot: Schema.Struct({ turns: Schema.Array(Schema.Json) }) }), +); +const threadId = ThreadId.make("pylon-thread"); +const instanceId = ProviderInstanceId.make("codex-test"); +const incarnationId = RuntimeSessionId.make("incarnation-1"); +const cwd = "/tmp/codex-absolute-test"; +const turn = (id: string, text = id): Schema.Json => ({ + id, + status: "completed", + itemsView: "full", + items: [{ id: `${id}-item`, type: "userMessage", content: [{ type: "text", text }] }], +}); +const checkpoint = (id: string | null, count: number) => ({ + kind: "checkpoint" as const, + checkpointTurnCount: count, + turnId: id === null ? null : TurnId.make(id), + checkpointRef: CheckpointRef.make(`checkpoint-${count}`), + checkpointOid: `oid-${count}`, + sourceRevision: count, +}); +const source = (id: string | null, count: number) => ({ + ...checkpoint(id, count), + kind: "source" as const, +}); + +function harness(initial: ReadonlyArray = []) { + let selected = "live"; + let counter = 0; + let idle = true; + let current = true; + let quarantined = false; + const histories = new Map([ + [ + selected, + { + threadId: selected, + cwd, + turns: initial, + nativeGoal: null, + nativeRecords: [{ type: "test-native-proof" }, ...initial], + rolloutPath: "/tmp/codex-absolute-test/native.jsonl", + }, + ], + ]); + const quarantineStates: boolean[] = []; + const read = (id = selected) => + Effect.suspend(() => { + const value = histories.get(id); + return value + ? Effect.succeed(structuredClone(value)) + : Effect.fail(CodexErrors.CodexAppServerRequestError.internalError("missing snapshot")); + }); + const native: NonNullable = { + isIdle: Effect.sync(() => idle), + read, + fork: ({ source: snapshot }) => + Effect.sync(() => { + const forked = { ...structuredClone(snapshot), threadId: `fork-${++counter}` }; + histories.set(forked.threadId, forked); + return forked; + }), + select: (snapshot) => + Effect.sync(() => { + selected = snapshot.threadId; + }), + quarantine: (held) => + Effect.sync(() => { + quarantined = held; + quarantineStates.push(held); + }), + }; + const make = ( + options: { + incarnation?: RuntimeSessionId; + recovering?: boolean; + targetThreadId?: ThreadId; + targetCwd?: string; + } = {}, + ) => + makeCodexAbsoluteRollback({ + threadId: options.targetThreadId ?? threadId, + instanceId, + incarnationId: options.incarnation ?? incarnationId, + cwd: options.targetCwd ?? cwd, + runtime: { absoluteConversation: native }, + recovering: options.recovering ?? false, + requireCurrent: Effect.suspend(() => + current + ? Effect.void + : Effect.fail( + new ProviderAdapterValidationError({ + provider: ProviderDriverKind.make("codex"), + operation: "test", + issue: "retired owner", + }), + ), + ), + }); + const controller = make(); + return { + controller, + make, + histories, + quarantineStates, + native, + setTurns: (turns: ReadonlyArray) => + histories.set(selected, { + threadId: selected, + cwd, + turns, + nativeGoal: null, + nativeRecords: [{ type: "test-native-proof" }, ...turns], + rolloutPath: "/tmp/codex-absolute-test/native.jsonl", + }), + setIdle: (value: boolean) => { + idle = value; + }, + retire: () => { + current = false; + }, + get selected() { + return selected; + }, + get quarantined() { + return quarantined; + }, + get forks() { + return counter; + }, + }; +} +const capture = ( + controller: ReturnType, + binding: ReturnType | ReturnType, +) => controller.operations.captureAnchor({ threadId, binding }); +const twoTurns = Effect.fn("test.twoTurns")(function* () { + const h = harness(); + yield* h.controller.initialize(); + const root = yield* capture(h.controller, checkpoint(null, 0)); + h.setTurns([turn("one")]); + yield* h.controller.recordCompleted("one"); + const first = yield* capture(h.controller, checkpoint("one", 1)); + h.setTurns([turn("one"), turn("two")]); + yield* h.controller.recordCompleted("two"); + const original = yield* capture(h.controller, source("two", 2)); + return { h, root, first, original }; +}); + +describe("Codex exact rollback ownership", () => { + it.effect( + "selects an immutable absolute target, retries idempotently, and compensates to the intact source", + () => + Effect.gen(function* () { + const { h, first, original } = yield* twoTurns(); + const originalLive = h.selected; + const originalContents = structuredClone(h.histories.get(originalLive)); + NodeAssert.equal(h.quarantined, true); + yield* h.controller.operations.applyAnchor(threadId, first.anchor); + NodeAssert.notEqual(h.selected, originalLive); + NodeAssert.equal( + (yield* h.controller.operations.inspectAnchor(threadId)).digest, + first.digest, + ); + const count = h.forks; + yield* h.controller.operations.applyAnchor(threadId, first.anchor); + NodeAssert.equal(h.forks, count); + NodeAssert.deepEqual(h.histories.get(originalLive), originalContents); + yield* h.controller.operations.applyAnchor(threadId, original.anchor); + NodeAssert.equal( + (yield* h.controller.operations.inspectAnchor(threadId)).digest, + original.digest, + ); + yield* h.controller.operations.releaseAnchor(threadId, original.anchor); + NodeAssert.equal(h.quarantined, false); + NodeAssert.ok(readCodexExactCursor(h.controller.cursor())); + }), + ); + it.effect("supports root only from the proven empty startup snapshot", () => + Effect.gen(function* () { + const { h, root } = yield* twoTurns(); + yield* h.controller.operations.applyAnchor(threadId, root.anchor); + NodeAssert.deepEqual(h.histories.get(h.selected)?.turns, []); + const old = harness([turn("old")]); + yield* old.controller.initialize(); + NodeAssert.equal(yield* old.controller.operations.isAvailable(threadId), false); + NodeAssert.ok( + Exit.isFailure(yield* Effect.exit(capture(old.controller, checkpoint(null, 0)))), + ); + }), + ); + it.effect( + "refuses an unrecorded completion even when native history is idle and has that turn ID", + () => + Effect.gen(function* () { + const h = harness([turn("one")]); + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(capture(h.controller, source("one", 1))))); + NodeAssert.equal(h.quarantined, true); + }), + ); + it.effect("rejects external edits retaining the recorded tail ID", () => + Effect.gen(function* () { + const h = harness([turn("one")]); + yield* h.controller.recordCompleted("one"); + h.setTurns([turn("one", "external replacement")]); + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(capture(h.controller, source("one", 1))))); + NodeAssert.equal(h.controller.cursor(), undefined); + }), + ); + it.effect("duplicate completion cannot replace the first immutable Pylon turn binding", () => + Effect.gen(function* () { + const h = harness([turn("one")]); + yield* h.controller.recordCompleted("one"); + const original = yield* capture(h.controller, checkpoint("one", 1)); + const forks = h.forks; + yield* h.controller.recordCompleted("one"); + NodeAssert.equal(h.forks, forks); + h.setTurns([turn("one", "external replacement")]); + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(h.controller.recordCompleted("one")))); + NodeAssert.equal( + (yield* capture(h.controller, checkpoint("one", 1))).digest, + original.digest, + ); + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(capture(h.controller, source("one", 1))))); + }), + ); + it.effect("refuses count guesses and keeps a checkpoint tied to its original full contents", () => + Effect.gen(function* () { + const h = harness([turn("one")]); + yield* h.controller.recordCompleted("one"); + NodeAssert.ok( + Exit.isFailure(yield* Effect.exit(capture(h.controller, checkpoint("unknown", 1)))), + ); + NodeAssert.ok( + Exit.isFailure( + yield* Effect.exit(capture(h.controller, { ...checkpoint("one", 1), sourceRevision: 2 })), + ), + ); + h.setTurns([turn("one", "external edit")]); + const anchored = yield* capture(h.controller, checkpoint("one", 1)); + const decoded = yield* decodeSnapshotTurns(anchored.anchor); + NodeAssert.deepEqual(decoded.snapshot.turns, [turn("one")]); + }), + ); + it.effect("requires recovery preparation and release before input or output resumes", () => + Effect.gen(function* () { + const { h, first, original } = yield* twoTurns(); + const cursor = h.controller.cursor(); + NodeAssert.ok(cursor); + const recovered = h.make({ recovering: true }); + yield* recovered.recover(cursor); + NodeAssert.equal(recovered.outputAllowed(), false); + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(recovered.beforeMutation))); + yield* recovered.operations.prepareRecovery!({ + threadId, + sourceAnchor: original.anchor, + desiredAnchor: first.anchor, + expectedAnchor: original.anchor, + }); + yield* recovered.activate; + NodeAssert.equal(recovered.outputAllowed(), false); + yield* recovered.operations.applyAnchor(threadId, first.anchor); + yield* recovered.operations.releaseAnchor(threadId, first.anchor); + NodeAssert.equal(recovered.outputAllowed(), true); + yield* recovered.beforeMutation; + NodeAssert.equal(recovered.cursor(), undefined); + }), + ); + it.effect( + "rejects same-incarnation recovery for a different incarnation, Pylon thread, cwd, or changed snapshot", + () => + Effect.gen(function* () { + const { h } = yield* twoTurns(); + const cursor = h.controller.cursor(); + NodeAssert.ok(cursor); + for (const options of [ + { incarnation: RuntimeSessionId.make("other") }, + { targetThreadId: ThreadId.make("other") }, + { targetCwd: "/elsewhere" }, + ]) { + NodeAssert.ok( + Exit.isFailure( + yield* Effect.exit(h.make({ ...options, recovering: true }).recover(cursor)), + ), + ); + } + h.setTurns([turn("one"), turn("two", "changed")]); + NodeAssert.ok( + Exit.isFailure(yield* Effect.exit(h.make({ recovering: true }).recover(cursor))), + ); + }), + ); + it.effect( + "ordinary Stop/resume rebinds verified history to a new incarnation without losing history", + () => + Effect.gen(function* () { + const { h, original } = yield* twoTurns(); + yield* h.controller.operations.releaseAnchor(threadId, original.anchor); + const previous = h.controller.cursor(); + NodeAssert.ok(previous); + const contents = structuredClone(h.histories.get(h.selected)); + const resumed = h.make({ incarnation: RuntimeSessionId.make("next-incarnation") }); + yield* resumed.initialize(previous); + NodeAssert.deepEqual(h.histories.get(h.selected), contents); + NodeAssert.equal( + resumed.cursor()?.exactRollback.anchor.sessionIncarnationId, + "next-incarnation", + ); + NodeAssert.equal((yield* capture(resumed, source("two", 2))).digest, original.digest); + }), + ); + it.effect( + "ordinary resume preserves changed native history but drops old exact eligibility", + () => + Effect.gen(function* () { + const { h, original } = yield* twoTurns(); + yield* h.controller.operations.releaseAnchor(threadId, original.anchor); + const previous = h.controller.cursor(); + NodeAssert.ok(previous); + h.setTurns([turn("new-external-turn")]); + const selected = h.selected; + const resumed = h.make({ incarnation: RuntimeSessionId.make("next-incarnation") }); + yield* resumed.initialize(previous); + NodeAssert.equal(h.selected, selected); + NodeAssert.deepEqual(h.histories.get(selected)?.turns, [turn("new-external-turn")]); + NodeAssert.equal(resumed.cursor(), undefined); + NodeAssert.equal(yield* resumed.operations.isAvailable(threadId), false); + }), + ); + it.effect( + "owned compaction refreshes source proof while retaining the original checkpoint through recovery", + () => + Effect.gen(function* () { + const h = harness([turn("one")]); + yield* h.controller.recordCompleted("one"); + const original = yield* capture(h.controller, checkpoint("one", 1)); + yield* h.controller.beforeCompaction; + NodeAssert.equal(h.controller.cursor(), undefined); + const compactedTurns = [ + turn("one"), + { + id: "compact-1", + status: "completed", + itemsView: "full", + items: [{ id: "compaction-item", type: "contextCompaction" }], + }, + ]; + h.setTurns(compactedTurns); + yield* h.controller.finishCompaction(); + NodeAssert.equal( + h.controller.cursor(), + undefined, + "acknowledgement alone cannot establish a proof", + ); + h.controller.compactionReceipt("compact-1"); + yield* h.controller.finishCompaction(); + const cursor = readCodexExactCursor(h.controller.cursor()); + NodeAssert.ok(cursor); + NodeAssert.deepEqual(cursor.exactRollback.anchor.compactionCheckpoint?.turns, [ + turn("one"), + ]); + const current = yield* capture(h.controller, source("one", 1)); + NodeAssert.notEqual(current.digest, original.digest); + NodeAssert.equal( + (yield* capture(h.controller, checkpoint("one", 1))).digest, + original.digest, + ); + const recovered = h.make({ recovering: true }); + yield* recovered.recover(cursor); + yield* recovered.operations.prepareRecovery!({ + threadId, + sourceAnchor: current.anchor, + desiredAnchor: original.anchor, + expectedAnchor: current.anchor, + }); + yield* recovered.activate; + yield* recovered.operations.applyAnchor(threadId, original.anchor); + yield* recovered.operations.releaseAnchor(threadId, original.anchor); + NodeAssert.deepEqual(h.histories.get(h.selected)?.turns, [turn("one")]); + NodeAssert.equal((yield* capture(recovered, source("one", 1))).digest, original.digest); + }), + ); + it.effect("compaction cannot use an old receipt or incomplete/non-compaction native tail", () => + Effect.gen(function* () { + const h = harness([turn("one")]); + yield* h.controller.recordCompleted("one"); + yield* capture(h.controller, checkpoint("one", 1)); + yield* h.controller.beforeCompaction; + h.controller.compactionReceipt("one"); + yield* h.controller.finishCompaction(); + NodeAssert.equal(h.controller.cursor(), undefined); + h.setTurns([turn("one"), turn("not-compaction")]); + h.controller.compactionReceipt("not-compaction"); + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(h.controller.finishCompaction()))); + NodeAssert.equal(h.controller.cursor(), undefined); + }), + ); + it.effect( + "normal resume after compaction preserves both current and original immutable histories", + () => + Effect.gen(function* () { + const h = harness([turn("one")]); + yield* h.controller.recordCompleted("one"); + const original = yield* capture(h.controller, checkpoint("one", 1)); + yield* h.controller.beforeCompaction; + h.setTurns([ + turn("one"), + { + id: "compact-1", + status: "completed", + itemsView: "full", + items: [{ id: "compact-item", type: "contextCompaction" }], + }, + ]); + h.controller.compactionReceipt("compact-1"); + yield* h.controller.finishCompaction(); + const previous = readCodexExactCursor(h.controller.cursor()); + NodeAssert.ok(previous); + const resumed = h.make({ incarnation: RuntimeSessionId.make("new-incarnation") }); + yield* resumed.initialize(previous); + NodeAssert.ok(readCodexExactCursor(resumed.cursor())); + NodeAssert.equal((yield* capture(resumed, checkpoint("one", 1))).digest, original.digest); + NodeAssert.equal( + (yield* capture(resumed, source("one", 1))).digest, + previous.exactRollback.anchor.digest, + ); + }), + ); + it.effect("rejects busy and retired owners and never accepts a malformed private cursor", () => + Effect.gen(function* () { + const { h } = yield* twoTurns(); + const cursor = h.controller.cursor(); + NodeAssert.ok(cursor); + NodeAssert.equal(readCodexExactCursor({ threadId: h.selected }), undefined); + NodeAssert.equal( + readCodexExactCursor({ + ...cursor, + exactRollback: { + ...cursor.exactRollback, + anchor: { ...cursor.exactRollback.anchor, digest: "changed" }, + }, + }), + undefined, + ); + h.setIdle(false); + NodeAssert.equal(yield* h.controller.operations.isAvailable(threadId), false); + NodeAssert.ok( + Exit.isFailure(yield* Effect.exit(h.controller.operations.inspectAnchor(threadId))), + ); + h.setIdle(true); + h.retire(); + NodeAssert.ok( + Exit.isFailure(yield* Effect.exit(h.controller.operations.inspectAnchor(threadId))), + ); + }), + ); +}); diff --git a/apps/server/src/provider/Layers/CodexAbsoluteRollback.ts b/apps/server/src/provider/Layers/CodexAbsoluteRollback.ts new file mode 100644 index 000000000..9f3439548 --- /dev/null +++ b/apps/server/src/provider/Layers/CodexAbsoluteRollback.ts @@ -0,0 +1,568 @@ +import { + ProviderDriverKind, + type ProviderInstanceId, + type RuntimeSessionId, + type ThreadId, +} from "@t3tools/contracts"; +import * as Effect from "effect/Effect"; +import * as Schema from "effect/Schema"; +import { ProviderAdapterValidationError, type ProviderAdapterError } from "../Errors.ts"; +import type { + ProviderAbsoluteConversationRollback, + ProviderConversationAnchorBinding, + ProviderConversationAnchorReceipt, +} from "../Services/ProviderAdapter.ts"; +import type { CodexSessionRuntimeShape } from "./CodexSessionRuntime.ts"; +import { codexConversationDigest, type CodexConversationSnapshot } from "./CodexAbsoluteHistory.ts"; + +const Snapshot = Schema.Struct({ + threadId: Schema.String, + cwd: Schema.String, + turns: Schema.Array(Schema.Json), + nativeRecords: Schema.Array(Schema.Json), + nativeGoal: Schema.Json, + rolloutPath: Schema.String, +}); +const Anchor = Schema.Struct({ + version: Schema.Literal(1), + provider: Schema.Literal("codex-exact"), + providerInstanceId: Schema.String, + sessionIncarnationId: Schema.String, + pylonThreadId: Schema.String, + cwd: Schema.String, + snapshot: Snapshot, + digest: Schema.String, + binding: Schema.optionalKey(Schema.Json), + compactionCheckpoint: Schema.optionalKey(Snapshot), + compactionTurnId: Schema.optionalKey(Schema.String), +}); +const Cursor = Schema.Struct({ + threadId: Schema.String, + exactRollback: Schema.Struct({ selectedThreadId: Schema.String, anchor: Anchor }), +}); +export type CodexExactCursor = typeof Cursor.Type; +const isCursor = Schema.is(Cursor); +const isAnchor = Schema.is(Anchor); +const coherentSnapshots = (anchor: typeof Anchor.Type) => + anchor.snapshot.cwd === anchor.cwd && + anchor.snapshot.rolloutPath.length > 0 && + anchor.snapshot.nativeRecords.length > 0 && + codexConversationDigest(anchor.snapshot) === anchor.digest && + ((anchor.compactionCheckpoint === undefined && anchor.compactionTurnId === undefined) || + (anchor.compactionCheckpoint !== undefined && + anchor.compactionTurnId !== undefined && + anchor.compactionCheckpoint.cwd === anchor.cwd && + anchor.compactionCheckpoint.rolloutPath.length > 0 && + anchor.compactionCheckpoint.nativeRecords.length > 0 && + anchor.compactionCheckpoint.threadId !== anchor.snapshot.threadId)); + +/** Only the private directory's complete idle proof is eligible for same-incarnation adoption. */ +export const readCodexExactCursor = (raw: unknown): CodexExactCursor | undefined => + isCursor(raw) && + raw.threadId.length > 0 && + raw.threadId === raw.exactRollback.selectedThreadId && + raw.threadId !== raw.exactRollback.anchor.snapshot.threadId && + raw.exactRollback.anchor.snapshot.threadId.length > 0 && + raw.exactRollback.anchor.snapshot.cwd === raw.exactRollback.anchor.cwd && + raw.exactRollback.anchor.providerInstanceId.length > 0 && + raw.exactRollback.anchor.sessionIncarnationId.length > 0 && + raw.exactRollback.anchor.pylonThreadId.length > 0 && + coherentSnapshots(raw.exactRollback.anchor) + ? raw + : undefined; + +export function makeCodexAbsoluteRollback(input: { + readonly threadId: ThreadId; + readonly instanceId: ProviderInstanceId; + readonly incarnationId: RuntimeSessionId | undefined; + readonly cwd: string; + readonly runtime: Pick; + readonly requireCurrent: Effect.Effect; + readonly recovering: boolean; +}) { + let recoveryHeld = input.recovering; + let rollbackHeld = false; + let outputEpoch = 0; + let source: typeof Anchor.Type | undefined; + let target: typeof Anchor.Type | undefined; + let proof: CodexExactCursor | undefined; + let root: typeof Anchor.Type | undefined; + const completed = new Map(); + const compacted = new Map(); + let compactionInFlight = false; + let pendingCompaction: + | { + selectedThreadId: string; + checkpointTurnId: string | null; + anchor: typeof Anchor.Type; + before: CodexConversationSnapshot; + receiptTurnId?: string; + } + | undefined; + const held = () => recoveryHeld || rollbackHeld; + const invalid = (issue: string) => + new ProviderAdapterValidationError({ + provider: ProviderDriverKind.make("codex"), + operation: "absoluteConversationRollback", + issue, + }); + const native = input.runtime.absoluteConversation; + const requireIdle = Effect.gen(function* () { + yield* input.requireCurrent; + if (!native || !input.incarnationId || !(yield* native.isIdle)) { + return yield* invalid( + "Exact Codex rollback needs a current, idle session with an incarnation and complete native history.", + ); + } + return native; + }); + const read = Effect.fn("CodexAbsoluteRollback.read")(function* (threadId?: string) { + const runtime = yield* requireIdle; + const snapshot = yield* runtime + .read(threadId) + .pipe(Effect.mapError(() => invalid("The exact Codex conversation could not be verified."))); + yield* input.requireCurrent; + return snapshot; + }); + const verifyAnchor = (raw: unknown) => + Effect.gen(function* () { + if ( + !isAnchor(raw) || + raw.providerInstanceId !== input.instanceId || + raw.sessionIncarnationId !== input.incarnationId || + raw.pylonThreadId !== input.threadId || + raw.cwd !== input.cwd || + raw.snapshot.cwd !== input.cwd || + !coherentSnapshots(raw) + ) { + return yield* invalid("The private Codex rollback identity or snapshot is invalid."); + } + return raw; + }); + const anchorFor = ( + snapshot: CodexConversationSnapshot, + binding?: ProviderConversationAnchorBinding, + ): typeof Anchor.Type => ({ + version: 1, + provider: "codex-exact", + providerInstanceId: input.instanceId, + sessionIncarnationId: input.incarnationId!, + pylonThreadId: input.threadId, + cwd: input.cwd, + snapshot, + digest: codexConversationDigest(snapshot), + ...(binding ? { binding: { ...binding } } : {}), + }); + const receipt = (anchor: typeof Anchor.Type): ProviderConversationAnchorReceipt => ({ + anchor, + digest: anchor.digest, + }); + const remember = (snapshot: CodexConversationSnapshot, anchor: typeof Anchor.Type) => { + proof = { + threadId: snapshot.threadId, + exactRollback: { selectedThreadId: snapshot.threadId, anchor }, + }; + }; + const fork = Effect.fn("CodexAbsoluteRollback.fork")(function* ( + snapshot: CodexConversationSnapshot, + lastTurnId?: string, + ) { + const runtime = yield* requireIdle; + const result = yield* runtime + .fork({ source: snapshot, ...(lastTurnId === undefined ? {} : { lastTurnId }) }) + .pipe( + Effect.mapError(() => + invalid("The immutable Codex conversation snapshot could not be verified."), + ), + ); + yield* input.requireCurrent; + return result; + }); + const inspect = Effect.fn("CodexAbsoluteRollback.inspect")(function* () { + const current = yield* read(); + const digest = codexConversationDigest(current); + const known = [source, target, proof?.exactRollback.anchor].find( + (anchor) => anchor?.digest === digest, + ); + if (!known) + return yield* invalid( + "The selected Codex conversation is outside the verified rollback boundary.", + ); + // Inspect can finish a selection that succeeded before its caller failed. Publish routing + // proof here as well, so the saga cannot verify a target while persisting a stale source. + const immutable = yield* read(known.snapshot.threadId); + if (codexConversationDigest(immutable) !== known.digest) + return yield* invalid("The private Codex snapshot changed."); + remember(current, known); + return receipt(known); + }); + const completedTail = (snapshot: CodexConversationSnapshot, turnId: string) => { + const last = snapshot.turns.at(-1); + return ( + last !== null && + typeof last === "object" && + "id" in last && + last?.id === turnId && + last.status === "completed" + ); + }; + const originalCheckpoint = (anchor: typeof Anchor.Type): typeof Anchor.Type => { + const { compactionCheckpoint, compactionTurnId: _compactionTurnId, ...plain } = anchor; + return compactionCheckpoint + ? { + ...plain, + snapshot: compactionCheckpoint, + digest: codexConversationDigest(compactionCheckpoint), + } + : plain; + }; + const retainBinding = (anchor: typeof Anchor.Type) => { + const binding = anchor.binding; + if (!binding || typeof binding !== "object" || !("kind" in binding) || !("turnId" in binding)) + return false; + if (binding.kind !== "checkpoint" && binding.kind !== "source") return false; + const original = originalCheckpoint(anchor); + if ( + anchor.compactionTurnId !== undefined && + !completedTail(anchor.snapshot, anchor.compactionTurnId) + ) + return false; + if (binding.turnId === null && original.snapshot.turns.length === 0) { + root = original; + if (anchor.compactionCheckpoint) compacted.set(null, anchor); + return true; + } + if (typeof binding.turnId === "string" && completedTail(original.snapshot, binding.turnId)) { + completed.set(binding.turnId, original); + if (anchor.compactionCheckpoint) compacted.set(binding.turnId, anchor); + return true; + } + return false; + }; + const beforeCompaction = Effect.gen(function* () { + yield* input.requireCurrent; + if (held()) return yield* invalid("Codex input is held until exact recovery is released."); + const previous = proof?.exactRollback.anchor; + proof = undefined; + pendingCompaction = undefined; + compactionInFlight = true; + yield* Effect.gen(function* () { + const current = yield* read(); + const digest = codexConversationDigest(current); + const known = [previous, root, ...completed.values(), ...compacted.values()].find( + (anchor) => anchor?.digest === digest, + ); + if (!known) return; + const original = originalCheckpoint(known); + const tail = original.snapshot.turns.at(-1); + const checkpointTurnId = + original.snapshot.turns.length === 0 + ? null + : tail !== null && typeof tail === "object" && "id" in tail && typeof tail.id === "string" + ? tail.id + : undefined; + if (checkpointTurnId === undefined) return; + pendingCompaction = { + selectedThreadId: current.threadId, + checkpointTurnId, + anchor: known, + before: current, + }; + }).pipe(Effect.ignore); + }); + const compactionReceipt = (turnId: string) => { + if ( + pendingCompaction && + !pendingCompaction.before.turns.some( + (turn) => turn !== null && typeof turn === "object" && "id" in turn && turn.id === turnId, + ) + ) { + pendingCompaction.receiptTurnId = turnId; + } + }; + const finishCompaction = Effect.fn("CodexAbsoluteRollback.finishCompaction")(function* () { + const pending = pendingCompaction; + if (!pending?.receiptTurnId) return; + const current = yield* read(); + if ( + current.threadId !== pending.selectedThreadId || + !completedTail(current, pending.receiptTurnId) + ) + return yield* invalid("Codex compaction has no exact completed native receipt."); + const tail = current.turns.at(-1); + if ( + !tail || + typeof tail !== "object" || + !("items" in tail) || + !Array.isArray(tail.items) || + !tail.items.some( + (item: unknown) => + item !== null && + typeof item === "object" && + "type" in item && + item.type === "contextCompaction", + ) + ) + return yield* invalid("The completed Codex receipt has no native compaction item."); + const original = originalCheckpoint(pending.anchor); + const immutable = yield* read(original.snapshot.threadId); + if (codexConversationDigest(immutable) !== original.digest) + return yield* invalid("The original Codex checkpoint changed during compaction."); + const snapshot = yield* fork(current); + const anchor = { + ...anchorFor(snapshot), + ...(pending.anchor.binding ? { binding: pending.anchor.binding } : {}), + compactionCheckpoint: original.snapshot, + compactionTurnId: pending.receiptTurnId, + }; + compacted.set(pending.checkpointTurnId, anchor); + if (anchor.binding) remember(current, anchor); + pendingCompaction = undefined; + compactionInFlight = false; + }); + // The event consumer records an immutable full history before publishing a Pylon completion. + // Later source capture compares this digest, so an external edit cannot masquerade as that turn. + const recordCompleted = Effect.fn("CodexAbsoluteRollback.recordCompleted")(function* ( + turnId: string, + ) { + yield* requireIdle; + if (held()) return yield* invalid("Codex completion is held during exact recovery."); + return yield* Effect.gen(function* () { + const current = yield* read(); + if (!completedTail(current, turnId)) + return yield* invalid("The completed Codex turn is not the authoritative history tail."); + const existing = completed.get(turnId); + if (existing) { + const immutable = yield* read(existing.snapshot.threadId); + if ( + codexConversationDigest(current) !== existing.digest || + codexConversationDigest(immutable) !== existing.digest + ) + return yield* invalid( + "A repeated Codex completion changed its original immutable history.", + ); + return; + } + const snapshot = yield* fork(current); + completed.set(turnId, anchorFor(snapshot)); + }); + }); + const initialize = Effect.fn("CodexAbsoluteRollback.initialize")(function* ( + previous?: CodexExactCursor, + ) { + yield* requireIdle; + return yield* Effect.gen(function* () { + const current = yield* read(); + if (current.turns.length === 0) { + root = anchorFor(yield* fork(current)); + return; + } + // An ordinary Stop/resume gets a new incarnation. It retains the selected native history, + // but may only rebind old checkpoint eligibility after both histories are verified. + const old = previous?.exactRollback.anchor; + if ( + !old || + previous.threadId !== current.threadId || + old.providerInstanceId !== input.instanceId || + old.pylonThreadId !== input.threadId || + old.cwd !== input.cwd || + codexConversationDigest(current) !== old.digest + ) + return; + const immutable = yield* read(old.snapshot.threadId); + if (codexConversationDigest(immutable) !== old.digest) return; + if (old.compactionCheckpoint) { + const original = yield* read(old.compactionCheckpoint.threadId); + if (codexConversationDigest(original) !== codexConversationDigest(old.compactionCheckpoint)) + return; + } + const snapshot = yield* fork(current); + const rebound = { + ...anchorFor(snapshot), + ...(old.binding ? { binding: old.binding } : {}), + ...(old.compactionCheckpoint + ? { + compactionCheckpoint: old.compactionCheckpoint, + compactionTurnId: old.compactionTurnId, + } + : {}), + }; + if (retainBinding(rebound)) remember(current, rebound); + }); + }); + const operations: ProviderAbsoluteConversationRollback = { + isAvailable: () => + requireIdle.pipe( + Effect.map( + () => + !compactionInFlight && + (root !== undefined || completed.size > 0 || proof !== undefined), + ), + Effect.orElseSucceed(() => false), + ), + captureAnchor: ({ binding }) => + Effect.gen(function* () { + const runtime = yield* requireIdle; + if (compactionInFlight) + return yield* invalid("Codex compaction has not reached a verified idle completion."); + if ( + binding.kind === "checkpoint" && + (binding.sourceRevision !== binding.checkpointTurnCount || + (binding.checkpointTurnCount === 0) !== (binding.turnId === null)) + ) + return yield* invalid("The exact Codex checkpoint binding is invalid."); + if (binding.kind === "source") { + rollbackHeld = true; + outputEpoch += 1; + yield* runtime.quarantine(true); + } + return yield* Effect.gen(function* () { + const current = yield* read(); + const checkpoint = binding.turnId === null ? root : completed.get(binding.turnId); + const known = + binding.kind === "source" + ? [compacted.get(binding.turnId), checkpoint].find( + (anchor) => anchor?.digest === codexConversationDigest(current), + ) + : checkpoint; + if (!known) + return yield* invalid("This Pylon turn has no verified immutable Codex checkpoint."); + const immutable = yield* read(known.snapshot.threadId); + if (codexConversationDigest(immutable) !== known.digest) + return yield* invalid("The private Codex checkpoint changed."); + const anchor = { ...known, binding: { ...binding } }; + if (binding.kind === "source") { + if (codexConversationDigest(current) !== known.digest) + return yield* invalid( + "The current Codex history differs from Pylon's recorded completed turn.", + ); + source = anchor; + target = undefined; + remember(current, anchor); + } else if (codexConversationDigest(current) === anchor.digest) { + remember(current, anchor); + } + return receipt(anchor); + }).pipe( + Effect.ensuring( + Effect.suspend(() => + binding.kind === "source" ? runtime.quarantine(held()) : Effect.void, + ), + ), + ); + }), + inspectAnchor: () => inspect(), + applyAnchor: (_threadId, raw) => + Effect.gen(function* () { + const runtime = yield* requireIdle; + const desired = yield* verifyAnchor(raw); + if (!rollbackHeld || !source) + return yield* invalid("An exact Codex source must be captured before applying a target."); + const current = yield* inspect(); + const previousTarget = target; + if (desired.digest !== source.digest) target = desired; + if ( + current.digest !== source.digest && + current.digest !== previousTarget?.digest && + current.digest !== desired.digest + ) { + return yield* invalid("Codex moved outside the exact source and target."); + } + yield* runtime.quarantine(true); + const immutable = yield* read(desired.snapshot.threadId); + if (codexConversationDigest(immutable) !== desired.digest) + return yield* invalid("The desired private Codex snapshot changed."); + if (current.digest === desired.digest) return; + // Selecting a new fork makes retries idempotent and leaves both recovery snapshots intact. + const selected = yield* fork(immutable); + yield* runtime + .select(selected) + .pipe(Effect.mapError(() => invalid("Selecting the exact Codex conversation failed."))); + yield* input.requireCurrent; + remember(selected, desired); + const verified = yield* inspect(); + if (verified.digest !== desired.digest) + return yield* invalid("The selected Codex target could not be verified."); + }), + prepareRecovery: ({ sourceAnchor, desiredAnchor, expectedAnchor }) => + Effect.gen(function* () { + const runtime = yield* requireIdle; + const restoredSource = yield* verifyAnchor(sourceAnchor); + const restoredTarget = yield* verifyAnchor(desiredAnchor); + const expected = yield* verifyAnchor(expectedAnchor); + if (expected.digest !== restoredSource.digest && expected.digest !== restoredTarget.digest) + return yield* invalid("The recovered Codex boundary is invalid."); + rollbackHeld = true; + outputEpoch += 1; + yield* runtime.quarantine(true); + source = restoredSource; + target = restoredTarget; + const current = yield* inspect(); + if (current.digest !== expected.digest) + return yield* invalid("The recovered Codex owner is not at its expected exact snapshot."); + }), + releaseAnchor: (_threadId, raw) => + Effect.gen(function* () { + const runtime = yield* requireIdle; + const expected = yield* verifyAnchor(raw); + const actual = yield* inspect(); + if (actual.digest !== expected.digest) + return yield* invalid("The exact Codex release snapshot does not match."); + rollbackHeld = false; + outputEpoch += 1; + source = undefined; + target = undefined; + yield* runtime.quarantine(held()); + }), + }; + return { + operations, + initialize, + recordCompleted, + beforeCompaction, + compactionReceipt, + finishCompaction, + outputAllowed: () => !held(), + outputEpoch: () => outputEpoch, + cursor: () => proof, + invalidate: () => { + proof = undefined; + }, + beforeMutation: Effect.gen(function* () { + yield* input.requireCurrent; + if (held()) return yield* invalid("Codex input is held until exact recovery is released."); + proof = undefined; + pendingCompaction = undefined; + compactionInFlight = false; + }), + recover: (cursor: CodexExactCursor) => + Effect.gen(function* () { + const anchor = yield* verifyAnchor(cursor.exactRollback.anchor); + const selected = yield* read(); + const immutable = yield* read(anchor.snapshot.threadId); + if ( + selected.threadId !== cursor.threadId || + codexConversationDigest(selected) !== anchor.digest || + codexConversationDigest(immutable) !== anchor.digest + ) + return yield* invalid("The private Codex idle recovery proof no longer matches."); + if (anchor.compactionCheckpoint) { + const original = yield* read(anchor.compactionCheckpoint.threadId); + if ( + codexConversationDigest(original) !== + codexConversationDigest(anchor.compactionCheckpoint) + ) + return yield* invalid("The original Codex checkpoint recovery snapshot changed."); + } + if (!retainBinding(anchor)) + return yield* invalid("The recovered Codex proof has no completed Pylon turn binding."); + remember(selected, anchor); + }), + activate: Effect.gen(function* () { + const runtime = yield* requireIdle; + recoveryHeld = false; + outputEpoch += 1; + yield* runtime.quarantine(held()); + }), + }; +} diff --git a/apps/server/src/provider/Layers/CodexAdapter.test.ts b/apps/server/src/provider/Layers/CodexAdapter.test.ts index e01c62a15..3fe845537 100644 --- a/apps/server/src/provider/Layers/CodexAdapter.test.ts +++ b/apps/server/src/provider/Layers/CodexAdapter.test.ts @@ -5,6 +5,7 @@ import * as NodeOS from "node:os"; import * as NodePath from "node:path"; import { ApprovalRequestId, + CheckpointRef, CodexSettings, CommandId, EnvironmentId, @@ -27,6 +28,7 @@ import * as NodeServices from "@effect/platform-node/NodeServices"; import { it, vi } from "@effect/vitest"; import * as Context from "effect/Context"; +import * as Deferred from "effect/Deferred"; import * as Effect from "effect/Effect"; import * as Exit from "effect/Exit"; import * as Fiber from "effect/Fiber"; @@ -52,6 +54,9 @@ import { type CodexThreadSnapshot, } from "./CodexSessionRuntime.ts"; import { makeCodexAdapter } from "./CodexAdapter.ts"; +import type { EventNdjsonLogger } from "./EventNdjsonLogger.ts"; +import { readCodexExactCursor } from "./CodexAbsoluteRollback.ts"; +import type { CodexConversationSnapshot } from "./CodexAbsoluteHistory.ts"; const decodeCodexSettings = Schema.decodeSync(CodexSettings); // Test-local service tag so the rest of the file can keep using `yield* CodexAdapter`. @@ -65,6 +70,7 @@ const asEventId = (value: string): EventId => EventId.make(value); const asItemId = (value: string): ProviderItemId => ProviderItemId.make(value); class FakeCodexRuntime implements CodexSessionRuntimeShape { + declare absoluteConversation?: NonNullable; private readonly eventQueue = Effect.runSync(Queue.unbounded()); private readonly now = "2026-01-01T00:00:00.000Z"; @@ -174,10 +180,11 @@ class FakeCodexRuntime implements CodexSessionRuntimeShape { } } -function makeRuntimeFactory() { +function makeRuntimeFactory(configure?: (runtime: FakeCodexRuntime) => void) { const runtimes: Array = []; const factory = vi.fn((options: CodexSessionRuntimeOptions) => { const runtime = new FakeCodexRuntime(options); + configure?.(runtime); runtimes.push(runtime); return Effect.succeed(runtime); }); @@ -3158,3 +3165,468 @@ usageLimitLayer("CodexAdapterLive usage limits", (it) => { }), ); }); + +function makeExactAdapterFixture( + nativeEventLogger?: EventNdjsonLogger, + onRead?: Effect.Effect, +) { + const histories = new Map(); + let selected = "native-live"; + let count = 0; + const factory = makeRuntimeFactory((runtime) => { + selected = runtime.options.resumeCursor?.threadId ?? "native-live"; + if (!histories.has(selected)) + histories.set(selected, { + threadId: selected, + cwd: runtime.options.cwd, + turns: [], + nativeGoal: null, + nativeRecords: [{ type: "test-native-proof" }], + rolloutPath: "/tmp/codex-adapter-native.jsonl", + }); + const originalStart = runtime.startImpl.getMockImplementation()!; + runtime.startImpl.mockImplementation(async () => ({ + ...(await originalStart()), + resumeCursor: { threadId: selected }, + })); + runtime.absoluteConversation = { + isIdle: Effect.succeed(true), + read: (id) => + Effect.gen(function* () { + if (onRead) yield* onRead; + const value = histories.get(id ?? selected); + NodeAssert.ok(value); + return structuredClone(value); + }), + fork: ({ source }) => + Effect.sync(() => { + const snapshot = { ...structuredClone(source), threadId: `private-${++count}` }; + histories.set(snapshot.threadId, snapshot); + return snapshot; + }), + select: (snapshot) => + Effect.sync(() => { + selected = snapshot.threadId; + }), + quarantine: () => Effect.void, + }; + }); + const layer = Layer.effect( + CodexAdapter, + makeCodexAdapter(decodeCodexSettings({}), { + makeRuntime: factory.factory, + ...(nativeEventLogger ? { nativeEventLogger } : {}), + }), + ).pipe( + Layer.provideMerge(ServerConfig.layerTest(process.cwd(), process.cwd())), + Layer.provideMerge(ServerSettingsService.layerTest()), + Layer.provideMerge(providerSessionDirectoryTestLayer), + Layer.provideMerge(NodeServices.layer), + ); + const threadId = asThreadId("thread-1"); + const binding = { + kind: "checkpoint" as const, + checkpointTurnCount: 1, + turnId: asTurnId("turn-exact"), + checkpointRef: CheckpointRef.make("checkpoint-exact"), + checkpointOid: "oid-exact", + sourceRevision: 1, + }; + const start = { + provider: ProviderDriverKind.make("codex"), + threadId, + runtimeMode: "full-access" as const, + cwd: process.cwd(), + sessionIncarnationId: RuntimeSessionId.make("exact-incarnation"), + }; + const completed = (adapter: CodexAdapterShape, owned = true) => + Effect.gen(function* () { + const runtime = factory.lastRuntime; + NodeAssert.ok(runtime); + runtime.sendTurnImpl.mockImplementation(async () => ({ + threadId, + turnId: asTurnId("turn-exact"), + })); + if (owned) yield* adapter.sendTurn({ threadId, input: "owned prompt" }); + histories.set(selected, { + threadId: selected, + cwd: process.cwd(), + nativeGoal: null, + nativeRecords: [{ type: "test-native-proof" }], + rolloutPath: "/tmp/codex-adapter-native.jsonl", + turns: [ + { + id: "turn-exact", + status: "completed", + itemsView: "full", + items: [ + { + id: "message-exact", + type: "userMessage", + content: [{ type: "text", text: "retained history" }], + }, + ], + }, + ], + }); + const receipt = yield* adapter.streamEvents.pipe( + Stream.filter((event) => event.type === "turn.completed"), + Stream.runHead, + Effect.forkChild, + ); + yield* runtime.emit(codexTurnEvent("turn/completed", "turn-exact")); + yield* Fiber.join(receipt); + return yield* adapter.absoluteConversationRollback!.captureAnchor({ threadId, binding }); + }); + return { + layer, + factory, + histories, + start, + binding, + completed, + get selected() { + return selected; + }, + }; +} + +it.effect( + "exact Codex checkpoints precede Pylon completion and ordinary Stop/mode resume preserves native history", + () => { + const fixture = makeExactAdapterFixture(); + return Effect.gen(function* () { + const adapter = yield* CodexAdapter; + yield* adapter.startSession(fixture.start); + const anchor = yield* fixture.completed(adapter); + const cursor = readCodexExactCursor((yield* adapter.listSessions())[0]?.resumeCursor); + NodeAssert.ok(cursor); + const history = structuredClone(fixture.histories.get(fixture.selected)); + for (const runtimeMode of ["full-access", "approval-required"] as const) { + yield* adapter.stopSession(fixture.start.threadId); + const started: ProviderSession = yield* adapter.startSession({ + ...fixture.start, + runtimeMode, + sessionIncarnationId: RuntimeSessionId.make(`resumed-${runtimeMode}`), + resumeCursor: cursor, + }); + const runtime = fixture.factory.lastRuntime; + NodeAssert.ok(runtime); + NodeAssert.equal(runtime.options.strictResume, true); + NodeAssert.deepEqual(runtime.options.resumeCursor, { threadId: cursor.threadId }); + NodeAssert.deepEqual(started.resumeCursor, { threadId: cursor.threadId }); + NodeAssert.deepEqual(fixture.histories.get(fixture.selected), history); + const rebound = readCodexExactCursor((yield* adapter.listSessions())[0]?.resumeCursor); + NodeAssert.ok(rebound); + NodeAssert.equal( + rebound.exactRollback.anchor.sessionIncarnationId, + `resumed-${runtimeMode}`, + ); + const verified = yield* adapter.absoluteConversationRollback!.captureAnchor({ + threadId: fixture.start.threadId, + binding: { ...fixture.binding, kind: "source" }, + }); + NodeAssert.equal(verified.digest, anchor.digest); + yield* adapter.absoluteConversationRollback!.releaseAnchor( + fixture.start.threadId, + verified.anchor, + ); + } + }).pipe(Effect.provide(fixture.layer)); + }, +); + +it.effect( + "Codex same-incarnation adoption starts quarantined, strips private proof, and holds input until prepared release", + () => { + const fixture = makeExactAdapterFixture(); + return Effect.gen(function* () { + const adapter = yield* CodexAdapter; + yield* adapter.startSession(fixture.start); + const target = yield* fixture.completed(adapter); + const source = yield* adapter.absoluteConversationRollback!.captureAnchor({ + threadId: fixture.start.threadId, + binding: { ...fixture.binding, kind: "source" }, + }); + const cursor = readCodexExactCursor((yield* adapter.listSessions())[0]?.resumeCursor); + NodeAssert.ok(cursor); + yield* adapter.stopSession(fixture.start.threadId); + const recovered = yield* adapter.recoverSession!({ + ...fixture.start, + providerInstanceId: ProviderInstanceId.make("codex"), + resumeCursor: cursor, + }); + NodeAssert.ok(recovered); + const runtime = fixture.factory.lastRuntime; + NodeAssert.ok(runtime); + NodeAssert.equal(runtime.options.quarantined, true); + NodeAssert.deepEqual(recovered.resumeCursor, { threadId: cursor.threadId }); + const send = { + threadId: fixture.start.threadId, + input: "next", + runtimeMode: "full-access" as const, + }; + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(adapter.sendTurn(send)))); + yield* adapter.absoluteConversationRollback!.prepareRecovery!({ + threadId: fixture.start.threadId, + sourceAnchor: source.anchor, + desiredAnchor: target.anchor, + expectedAnchor: source.anchor, + }); + yield* adapter.activateRecoveredSession!(fixture.start.threadId); + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(adapter.sendTurn(send)))); + yield* adapter.absoluteConversationRollback!.releaseAnchor( + fixture.start.threadId, + source.anchor, + ); + yield* adapter.sendTurn(send); + NodeAssert.equal( + readCodexExactCursor((yield* adapter.listSessions())[0]?.resumeCursor), + undefined, + ); + }).pipe(Effect.provide(fixture.layer)); + }, +); + +it.effect("Codex rejects public or mismatched recovery cursors before opening a runtime", () => { + const fixture = makeExactAdapterFixture(); + return Effect.gen(function* () { + const adapter = yield* CodexAdapter; + const input = { + ...fixture.start, + providerInstanceId: ProviderInstanceId.make("codex"), + resumeCursor: { threadId: "native-live" }, + }; + NodeAssert.equal(yield* adapter.recoverSession!(input), null); + NodeAssert.equal(fixture.factory.factory.mock.calls.length, 0); + yield* adapter.startSession(fixture.start); + yield* fixture.completed(adapter); + const cursor = readCodexExactCursor((yield* adapter.listSessions())[0]?.resumeCursor); + NodeAssert.ok(cursor); + const calls = fixture.factory.factory.mock.calls.length; + NodeAssert.equal( + yield* adapter.recoverSession!({ + ...input, + sessionIncarnationId: RuntimeSessionId.make("other"), + resumeCursor: cursor, + }), + null, + ); + NodeAssert.equal(fixture.factory.factory.mock.calls.length, calls); + }).pipe(Effect.provide(fixture.layer)); +}); + +it.effect("Codex external completion cannot establish an owned Pylon checkpoint", () => { + const fixture = makeExactAdapterFixture(); + return Effect.gen(function* () { + const adapter = yield* CodexAdapter; + yield* adapter.startSession(fixture.start); + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(fixture.completed(adapter, false)))); + NodeAssert.equal( + readCodexExactCursor((yield* adapter.listSessions())[0]?.resumeCursor), + undefined, + ); + NodeAssert.ok( + Exit.isFailure( + yield* Effect.exit( + adapter.absoluteConversationRollback!.captureAnchor({ + threadId: fixture.start.threadId, + binding: { ...fixture.binding, kind: "source" }, + }), + ), + ), + ); + }).pipe(Effect.provide(fixture.layer)); +}); + +it.effect( + "Codex quarantine fences queued adapter output and a suspended native logger across release", + () => + Effect.gen(function* () { + const entered = yield* Deferred.make(); + const release = yield* Deferred.make(); + const logged: string[] = []; + const fixture = makeExactAdapterFixture({ + filePath: "unused-test-logger", + close: () => Effect.void, + write: (event, _threadId, guard) => + Effect.gen(function* () { + const id = + typeof event === "object" && + event !== null && + "id" in event && + typeof event.id === "string" + ? event.id + : ""; + if (id === "blocked-old") { + yield* Deferred.succeed(entered, undefined); + yield* Deferred.await(release); + } + if (guard && !(yield* guard)) return; + logged.push(id); + }), + }); + yield* Effect.gen(function* () { + const adapter = yield* CodexAdapter; + yield* adapter.startSession(fixture.start); + yield* fixture.completed(adapter); + const runtime = fixture.factory.lastRuntime; + NodeAssert.ok(runtime); + const delta = (id: string): ProviderEvent => ({ + id: asEventId(id), + provider: ProviderDriverKind.make("codex"), + kind: "notification", + threadId: fixture.start.threadId, + createdAt: "2026-01-01T00:00:00.000Z", + method: "item/agentMessage/delta", + payload: { + threadId: "native-live", + turnId: "turn-exact", + itemId: "assistant-exact", + delta: id, + }, + }); + yield* runtime.emit(delta("queued-old")); + yield* runtime.emit(delta("blocked-old")); + yield* Deferred.await(entered); + const source = yield* adapter.absoluteConversationRollback!.captureAnchor({ + threadId: fixture.start.threadId, + binding: { ...fixture.binding, kind: "source" }, + }); + yield* adapter.absoluteConversationRollback!.releaseAnchor( + fixture.start.threadId, + source.anchor, + ); + yield* Deferred.succeed(release, undefined); + const next = yield* adapter.streamEvents.pipe( + Stream.filter((event) => event.type === "content.delta"), + Stream.runHead, + Effect.forkChild, + ); + yield* runtime.emit(delta("fresh")); + const event = yield* Fiber.join(next); + NodeAssert.ok(Option.isSome(event)); + NodeAssert.equal(event.value.eventId, "fresh"); + NodeAssert.equal(logged.includes("blocked-old"), false); + }).pipe(Effect.provide(fixture.layer)); + }), +); + +it.effect( + "Codex compaction completion persists a current proof without replacing its original checkpoint", + () => { + const fixture = makeExactAdapterFixture(); + return Effect.gen(function* () { + const adapter = yield* CodexAdapter; + yield* adapter.startSession(fixture.start); + const original = yield* fixture.completed(adapter); + const compaction = adapter.compaction; + NodeAssert.ok(compaction?.type === "native"); + yield* compaction.start(fixture.start.threadId); + NodeAssert.equal( + readCodexExactCursor((yield* adapter.listSessions())[0]?.resumeCursor), + undefined, + ); + const before = fixture.histories.get(fixture.selected); + NodeAssert.ok(before); + fixture.histories.set(fixture.selected, { + ...before, + turns: [ + ...before.turns, + { + id: "compact-turn", + status: "completed", + itemsView: "full", + items: [{ id: "compact-item", type: "contextCompaction" }], + }, + ], + }); + const receipt = yield* adapter.streamEvents.pipe( + Stream.filter( + (event) => event.type === "thread.state.changed" && event.payload.state === "compacted", + ), + Stream.runHead, + Effect.forkChild, + ); + const runtime = fixture.factory.lastRuntime; + NodeAssert.ok(runtime); + yield* runtime.emit({ + id: asEventId("compacted"), + kind: "notification", + provider: ProviderDriverKind.make("codex"), + threadId: fixture.start.threadId, + turnId: asTurnId("compact-turn"), + createdAt: "2026-01-01T00:00:00.000Z", + method: "item/completed", + payload: { + threadId: "native-live", + turnId: "compact-turn", + item: { id: "compact-item", type: "contextCompaction" }, + completedAtMs: 1, + }, + }); + yield* Fiber.join(receipt); + const cursor = readCodexExactCursor((yield* adapter.listSessions())[0]?.resumeCursor); + NodeAssert.ok(cursor); + NodeAssert.equal(cursor.exactRollback.anchor.compactionTurnId, "compact-turn"); + const source = yield* adapter.absoluteConversationRollback!.captureAnchor({ + threadId: fixture.start.threadId, + binding: { ...fixture.binding, kind: "source" }, + }); + NodeAssert.notEqual(source.digest, original.digest); + NodeAssert.equal( + (yield* adapter.absoluteConversationRollback!.captureAnchor({ + threadId: fixture.start.threadId, + binding: fixture.binding, + })).digest, + original.digest, + ); + yield* adapter.absoluteConversationRollback!.applyAnchor( + fixture.start.threadId, + original.anchor, + ); + yield* adapter.absoluteConversationRollback!.releaseAnchor( + fixture.start.threadId, + original.anchor, + ); + NodeAssert.deepEqual(fixture.histories.get(fixture.selected)?.turns, before.turns); + NodeAssert.equal(runtime.rollbackThreadImpl.mock.calls.length, 0); + }).pipe(Effect.provide(fixture.layer)); + }, +); + +it.effect("Codex initialization cannot return a stopped owner or remove its replacement", () => + Effect.gen(function* () { + const entered = yield* Deferred.make(); + const release = yield* Deferred.make(); + let block = true; + const fixture = makeExactAdapterFixture( + undefined, + Effect.suspend(() => { + if (!block) return Effect.void; + block = false; + return Deferred.succeed(entered, undefined).pipe(Effect.andThen(Deferred.await(release))); + }), + ); + yield* Effect.gen(function* () { + const adapter = yield* CodexAdapter; + const first = yield* adapter.startSession(fixture.start).pipe(Effect.exit, Effect.forkChild); + yield* Deferred.await(entered); + yield* adapter.stopSession(fixture.start.threadId); + const replacementIncarnation = RuntimeSessionId.make("replacement-incarnation"); + yield* adapter.startSession({ + ...fixture.start, + sessionIncarnationId: replacementIncarnation, + }); + yield* Deferred.succeed(release, undefined); + NodeAssert.ok(Exit.isFailure(yield* Fiber.join(first))); + const sessions = yield* adapter.listSessions(); + NodeAssert.equal(sessions.length, 1); + NodeAssert.equal(sessions[0]?.sessionIncarnationId, replacementIncarnation); + yield* adapter.sendTurn({ + threadId: fixture.start.threadId, + input: "replacement still works", + }); + }).pipe(Effect.provide(fixture.layer)); + }), +); diff --git a/apps/server/src/provider/Layers/CodexAdapter.ts b/apps/server/src/provider/Layers/CodexAdapter.ts index f0c803ee4..5d9b767d9 100644 --- a/apps/server/src/provider/Layers/CodexAdapter.ts +++ b/apps/server/src/provider/Layers/CodexAdapter.ts @@ -1,3 +1,8 @@ +import { + makeCodexAbsoluteRollback, + readCodexExactCursor, + type CodexExactCursor, +} from "./CodexAbsoluteRollback.ts"; /** * CodexAdapterLive - Scoped live implementation for the Codex provider adapter. * @@ -23,6 +28,7 @@ import { type ToolActivitySource, type ProviderUserInputAnswers, RuntimeItemId, + type RuntimeSessionId, RuntimeRequestId, RuntimeTaskId, type RuntimeTaskUsage, @@ -108,6 +114,9 @@ interface CodexAdapterSessionContext { readonly runtime: CodexSessionRuntimeShape; readonly eventFiber: Fiber.Fiber; readonly admissionSemaphore: Semaphore.Semaphore; + readonly rollback: ReturnType; + readonly sessionIncarnationId: RuntimeSessionId | undefined; + readonly ownedTurnIds: Set; readonly pendingAdmissions: Array<{ readonly admissionRequestId: NonNullable; readonly sessionIncarnationId: NonNullable; @@ -2247,10 +2256,17 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( : undefined); const managedNativeEventLogger = options?.nativeEventLogger === undefined ? nativeEventLogger : undefined; - const runtimeEventQueue = yield* Queue.unbounded(); + const runtimeEventQueue = yield* Queue.unbounded<{ + readonly event: ProviderRuntimeEvent; + readonly rollback: ReturnType; + readonly epoch: number; + }>(); const sessions = new Map(); - const startSession: CodexAdapterShape["startSession"] = (input) => + const startSessionInternal = ( + input: Parameters[0], + recovering?: CodexExactCursor, + ) => Effect.scoped( Effect.gen(function* () { if (input.provider !== undefined && input.provider !== PROVIDER) { @@ -2271,6 +2287,10 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( ? getCodexServiceTierOptionValue(input.modelSelection) : undefined; const mcpSession = McpProviderSession.readMcpProviderSession(input.threadId); + const hasPrivateResumeProof = + typeof input.resumeCursor === "object" && + input.resumeCursor !== null && + "exactRollback" in input.resumeCursor; const runtimeInput: CodexSessionRuntimeOptions = { threadId: input.threadId, providerInstanceId: boundInstanceId, @@ -2280,7 +2300,19 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( ...(options?.environment ? { environment: options.environment } : {}), ...(codexConfig.homePath ? { homePath: codexConfig.homePath } : {}), ...(isCodexResumeCursorSchema(input.resumeCursor) - ? { resumeCursor: input.resumeCursor } + ? { resumeCursor: { threadId: input.resumeCursor.threadId } } + : {}), + ...(recovering || hasPrivateResumeProof ? { strictResume: true } : {}), + ...(recovering + ? { + quarantined: true, + exactRecoverySnapshot: recovering.exactRollback.anchor.snapshot, + ...(recovering.exactRollback.anchor.compactionCheckpoint + ? { + exactRecoveryCheckpoint: recovering.exactRollback.anchor.compactionCheckpoint, + } + : {}), + } : {}), runtimeMode: input.runtimeMode, ...(input.modelSelection?.instanceId === boundInstanceId @@ -2325,6 +2357,7 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( Effect.provideService(Scope.Scope, sessionScope), Effect.provideService(ChildProcessSpawner.ChildProcessSpawner, childProcessSpawner), Effect.provideService(Crypto.Crypto, crypto), + Effect.provideService(FileSystem.FileSystem, fileSystem), Effect.mapError( (cause) => new ProviderAdapterProcessError({ @@ -2337,6 +2370,26 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( ); const pendingAdmissions: CodexAdapterSessionContext["pendingAdmissions"] = []; + let context: CodexAdapterSessionContext | undefined; + const rollback = makeCodexAbsoluteRollback({ + threadId: input.threadId, + instanceId: boundInstanceId, + incarnationId: input.sessionIncarnationId, + cwd: runtimeInput.cwd, + runtime, + recovering: recovering !== undefined, + requireCurrent: Effect.suspend(() => + context && !context.stopped && sessions.get(input.threadId) === context + ? Effect.void + : Effect.fail( + new ProviderAdapterValidationError({ + provider: PROVIDER, + operation: "absoluteConversationRollback", + issue: "The Codex session owner changed during exact recovery.", + }), + ), + ), + }); // Fork into the session scope, not the calling fiber. `forkChild` makes // this a child of `startSession`, and Effect interrupts a fiber's @@ -2344,7 +2397,50 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( // runtime event the session emitted afterwards was dropped. const eventFiber = yield* Stream.runForEach(runtime.events, (event) => Effect.gen(function* () { - yield* writeNativeEvent(event); + if (!rollback.outputAllowed()) return; + const epoch = rollback.outputEpoch(); + if (event.method === "turn/started") rollback.invalidate(); + if (event.method === "turn/completed" && event.turnId && context) { + const completedTurnId = event.turnId; + yield* context.admissionSemaphore.withPermit( + Effect.suspend(() => + context?.ownedTurnIds.has(completedTurnId) + ? rollback.recordCompleted(completedTurnId).pipe(Effect.ignore) + : Effect.void, + ), + ); + } + const compactionPayload = + typeof event.payload === "object" && event.payload !== null + ? event.payload + : undefined; + const compactionItem = + compactionPayload && "item" in compactionPayload ? compactionPayload.item : undefined; + const isCompactionReceipt = + event.method === "thread/compacted" || + (event.method === "item/completed" && + compactionItem !== null && + typeof compactionItem === "object" && + "type" in compactionItem && + compactionItem.type === "contextCompaction"); + const nativeTurnId = + event.turnId ?? + (compactionPayload && + "turnId" in compactionPayload && + typeof compactionPayload.turnId === "string" + ? compactionPayload.turnId + : undefined); + if (isCompactionReceipt && nativeTurnId) rollback.compactionReceipt(nativeTurnId); + if (context && (isCompactionReceipt || event.method === "turn/completed")) + yield* context.admissionSemaphore.withPermit( + rollback.finishCompaction().pipe(Effect.ignore), + ); + if (!rollback.outputAllowed() || epoch !== rollback.outputEpoch()) return; + yield* writeNativeEvent( + event, + Effect.sync(() => rollback.outputAllowed() && epoch === rollback.outputEpoch()), + ); + if (!rollback.outputAllowed() || epoch !== rollback.outputEpoch()) return; if (event.method === "turn/started" && event.turnId) { if (turnTokenUsage.activeTurnId !== event.turnId) { turnTokenUsage.byTurnId.clear(); @@ -2479,7 +2575,10 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( }); return; } - yield* Queue.offerAll(runtimeEventQueue, runtimeEvents); + yield* Queue.offerAll( + runtimeEventQueue, + runtimeEvents.map((event) => ({ event, rollback, epoch })), + ); }), ).pipe(Effect.forkIn(sessionScope)); @@ -2502,8 +2601,11 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( ), ); - sessions.set(input.threadId, { + context = { threadId: input.threadId, + rollback, + sessionIncarnationId: input.sessionIncarnationId, + ownedTurnIds: new Set(), scope: sessionScope, runtime, eventFiber, @@ -2511,13 +2613,58 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( pendingAdmissions, turnTokenUsage, stopped: false, - }); + }; + sessions.set(input.threadId, context); + if (recovering) { + if (!isCodexResumeCursorSchema(started.resumeCursor)) + return yield* new ProviderAdapterValidationError({ + provider: PROVIDER, + operation: "recoverSession", + issue: "The verified Codex recovery owner has no native cursor.", + }); + const selectedThreadId = started.resumeCursor.threadId; + yield* rollback + .recover({ + ...recovering, + threadId: selectedThreadId, + exactRollback: { ...recovering.exactRollback, selectedThreadId }, + }) + .pipe(Effect.onError(() => stopSessionInternal(context!))); + } else + yield* rollback.initialize(readCodexExactCursor(input.resumeCursor)).pipe(Effect.ignore); + if (context.stopped || sessions.get(input.threadId) !== context) + return yield* new ProviderAdapterValidationError({ + provider: PROVIDER, + operation: "startSession", + issue: "The Codex session owner changed before startup completed.", + }); sessionScopeTransferred = true; return started; }), ); + const startSession: CodexAdapterShape["startSession"] = (input) => startSessionInternal(input); + const recoverSession: NonNullable = Effect.fn( + "CodexAdapter.recoverSession", + )(function* (input) { + const cursor = readCodexExactCursor(input.resumeCursor); + if ( + !cursor || + input.providerInstanceId !== boundInstanceId || + cursor.exactRollback.anchor.providerInstanceId !== boundInstanceId || + cursor.exactRollback.anchor.sessionIncarnationId !== input.sessionIncarnationId || + cursor.exactRollback.anchor.pylonThreadId !== input.threadId || + cursor.exactRollback.anchor.cwd !== input.cwd + ) + return null; + const session = yield* startSessionInternal( + { ...input, provider: PROVIDER, resumeCursor: { threadId: cursor.threadId } }, + cursor, + ); + return { ...session, sessionIncarnationId: input.sessionIncarnationId }; + }); + const resolveAttachment = Effect.fn("resolveAttachment")(function* ( input: ProviderSendTurnInput, attachment: NonNullable[number], @@ -2563,6 +2710,7 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( const session = yield* requireSession(input.threadId); return yield* session.admissionSemaphore.withPermit( Effect.gen(function* () { + yield* session.rollback.beforeMutation; const reasoningEffort = input.modelSelection?.instanceId === boundInstanceId ? getModelSelectionStringOptionValue(input.modelSelection, "reasoningEffort") @@ -2606,6 +2754,7 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( }), ), ); + if (result.turnId !== undefined) session.ownedTurnIds.add(result.turnId); if (admission !== undefined && session.pendingAdmissions.includes(admission)) { admission.turnId = result.turnId; } @@ -2637,8 +2786,15 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( const compactThread = Effect.fn("compactThread")(function* (threadId: ThreadId) { const session = yield* requireSession(threadId); - yield* session.runtime.compactThread.pipe( - Effect.mapError((cause) => mapCodexRuntimeError(threadId, "thread/compact/start", cause)), + yield* session.admissionSemaphore.withPermit( + session.rollback.beforeCompaction.pipe( + Effect.andThen(session.runtime.compactThread), + Effect.mapError((cause) => + "provider" in cause + ? cause + : mapCodexRuntimeError(threadId, "thread/compact/start", cause), + ), + ), ); }); @@ -2669,20 +2825,21 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( return requireSession(threadId).pipe( Effect.flatMap((session) => - session.runtime.rollbackThread(numTurns).pipe( - Effect.tap(() => - Effect.sync(() => { - session.turnTokenUsage.baseline = undefined; - session.turnTokenUsage.activeTurnId = undefined; - session.turnTokenUsage.byTurnId.clear(); - }), + session.admissionSemaphore.withPermit( + session.rollback.beforeMutation.pipe( + Effect.andThen(session.runtime.rollbackThread(numTurns)), + Effect.tap(() => + Effect.sync(() => { + session.turnTokenUsage.baseline = undefined; + session.turnTokenUsage.activeTurnId = undefined; + session.turnTokenUsage.byTurnId.clear(); + }), + ), ), ), ), Effect.mapError((cause) => - cause._tag === "ProviderAdapterSessionNotFoundError" - ? cause - : mapCodexRuntimeError(threadId, "thread/rollback", cause), + "provider" in cause ? cause : mapCodexRuntimeError(threadId, "thread/rollback", cause), ), Effect.map((snapshot) => ({ threadId, @@ -2726,11 +2883,14 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( ), ); - const writeNativeEvent = Effect.fnUntraced(function* (event: ProviderEvent) { + const writeNativeEvent = Effect.fnUntraced(function* ( + event: ProviderEvent, + commitGuard?: Effect.Effect, + ) { if (!nativeEventLogger) { return; } - yield* nativeEventLogger.write(event, event.threadId); + yield* nativeEventLogger.write(event, event.threadId, commitGuard); }); const stopSessionInternal = Effect.fn("stopSessionInternal")(function* ( @@ -2740,7 +2900,7 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( return; } session.stopped = true; - sessions.delete(session.threadId); + if (sessions.get(session.threadId) === session) sessions.delete(session.threadId); yield* session.runtime.close.pipe(Effect.ignore); yield* Effect.ignore(Scope.close(session.scope, Exit.void)); yield* Fiber.interrupt(session.eventFiber).pipe(Effect.ignore); @@ -2758,10 +2918,49 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( const listSessions: CodexAdapterShape["listSessions"] = () => Effect.forEach( Array.from(sessions.values()).filter((session) => !session.stopped), - (session) => session.runtime.getSession, + (context) => + context.runtime.getSession.pipe( + Effect.map((session) => ({ + ...session, + ...(context.sessionIncarnationId + ? { sessionIncarnationId: context.sessionIncarnationId } + : {}), + ...(context.rollback.cursor() ? { resumeCursor: context.rollback.cursor() } : {}), + })), + ), { concurrency: 1 }, ); + const withExact = ( + threadId: ThreadId, + run: ( + rollback: ReturnType, + ) => Effect.Effect, + ) => + requireSession(threadId).pipe( + Effect.flatMap((context) => context.admissionSemaphore.withPermit(run(context.rollback))), + ); + const absoluteConversationRollback: NonNullable< + CodexAdapterShape["absoluteConversationRollback"] + > = { + isAvailable: (threadId) => + withExact(threadId, (rollback) => rollback.operations.isAvailable(threadId)).pipe( + Effect.orElseSucceed(() => false), + ), + captureAnchor: (input) => + withExact(input.threadId, (rollback) => rollback.operations.captureAnchor(input)), + inspectAnchor: (threadId) => + withExact(threadId, (rollback) => rollback.operations.inspectAnchor(threadId)), + applyAnchor: (threadId, anchor) => + withExact(threadId, (rollback) => rollback.operations.applyAnchor(threadId, anchor)), + releaseAnchor: (threadId, anchor) => + withExact(threadId, (rollback) => rollback.operations.releaseAnchor(threadId, anchor)), + prepareRecovery: (input) => + withExact(input.threadId, (rollback) => rollback.operations.prepareRecovery!(input)), + }; + const activateRecoveredSession = (threadId: ThreadId) => + withExact(threadId, (rollback) => rollback.activate); + const hasSession: CodexAdapterShape["hasSession"] = (threadId) => Effect.succeed(Boolean(sessions.get(threadId) && !sessions.get(threadId)?.stopped)); @@ -2787,6 +2986,9 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( promptlessTurnContinuation: true, }, startSession, + recoverSession, + activateRecoveredSession, + absoluteConversationRollback, sendTurn, compaction: { type: "native", start: compactThread }, interruptTurn, @@ -2800,7 +3002,12 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( hasSession, stopAll, get streamEvents() { - return Stream.fromQueue(runtimeEventQueue); + return Stream.fromQueue(runtimeEventQueue).pipe( + Stream.filter( + ({ rollback, epoch }) => rollback.outputAllowed() && rollback.outputEpoch() === epoch, + ), + Stream.map(({ event }) => event), + ); }, } satisfies CodexAdapterShape; }); diff --git a/apps/server/src/provider/Layers/CodexSessionRuntime.exact.test.ts b/apps/server/src/provider/Layers/CodexSessionRuntime.exact.test.ts new file mode 100644 index 000000000..9448f05f1 --- /dev/null +++ b/apps/server/src/provider/Layers/CodexSessionRuntime.exact.test.ts @@ -0,0 +1,641 @@ +// @effect-diagnostics nodeBuiltinImport:off +import * as NodeAssert from "node:assert/strict"; +import * as NodeFS from "node:fs"; +import * as NodeOS from "node:os"; +import * as NodePath from "node:path"; + +import * as NodeServices from "@effect/platform-node/NodeServices"; +import { describe, it } from "@effect/vitest"; +import { type ProviderEvent, ThreadId } from "@t3tools/contracts"; +import * as Crypto from "effect/Crypto"; +import * as Deferred from "effect/Deferred"; +import * as Effect from "effect/Effect"; +import * as Exit from "effect/Exit"; +import * as Fiber from "effect/Fiber"; +import * as Schema from "effect/Schema"; +import * as Stream from "effect/Stream"; +import { ChildProcess, ChildProcessSpawner } from "effect/unstable/process"; + +import { makeCodexSessionRuntime, type CodexSessionRuntimeOptions } from "./CodexSessionRuntime.ts"; + +const SOURCE = "private-source"; +const decodeLogEntry = Schema.decodeUnknownSync( + Schema.fromJsonString( + Schema.Struct({ + method: Schema.optionalKey(Schema.String), + params: Schema.optionalKey(Schema.Record(Schema.String, Schema.Unknown)), + id: Schema.optionalKey(Schema.Union([Schema.String, Schema.Number])), + result: Schema.optionalKey(Schema.Unknown), + error: Schema.optionalKey(Schema.Struct({ code: Schema.Number, message: Schema.String })), + }), + ), +); +const markerPayload = Schema.Struct({ requestId: Schema.String }); +const isMarkerPayload = Schema.is(markerPayload); + +// The real runtime owns a real scoped stdio child. Only its executable is redirected +// to this isolated peer; lifecycle, protocol decoding, queues, and approval handlers run unchanged. +const PEER = String.raw` +import * as fs from "node:fs"; +import * as readline from "node:readline"; +const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8")); +const cwd = process.cwd(); +const write = (message) => process.stdout.write(JSON.stringify(message) + "\n"); +const reply = (id, result) => write({ id, result }); +const log = (message) => fs.appendFileSync(config.logPath, JSON.stringify(message) + "\n"); +const threads = new Map([["private-source", []], ["private-immutable", []]]); +const nativeRecords = new Map(); +const loaded = new Set(); +const nativePath = (threadId) => cwd + "/" + threadId + ".jsonl"; +const nativeHeader = (threadId, sourceId) => ({ timestamp: "2026-01-01T00:00:00.000Z", type: "session_meta", payload: { + id: threadId, session_id: threadId, timestamp: "2026-01-01T00:00:00.000Z", cwd, + context_window: { window_id: "window-" + threadId }, base_instructions: { text: "native proof fixture" }, + ...(sourceId ? { forked_from_id: sourceId } : {}) +} }); +const persistNative = (threadId, records) => { + nativeRecords.set(threadId, records); + fs.writeFileSync(nativePath(threadId), records.map((record) => JSON.stringify(record)).join("\n") + "\n"); +}; +for (const threadId of threads.keys()) persistNative(threadId, [nativeHeader(threadId), + ...(config.changedNative && threadId === "private-source" ? [{ timestamp: "2026-01-01T00:00:00.000Z", type: "world_state", payload: { full: true, state: { hidden: "changed model context" } } }] : []) +]); +const metadata = (threadId) => ({ + id: threadId, cwd, path: nativePath(threadId), ephemeral: false, + status: { type: config.unloadedSnapshots && !loaded.has(threadId) ? "notLoaded" : "idle" }, turns: threads.get(threadId) ?? [], + cliVersion: "test", createdAt: 1, updatedAt: 1, modelProvider: "openai", preview: "", + sessionId: threadId, source: "appServer" +}); +const opened = (threadId) => ({ + cwd: config.wrongCwd ? cwd + "/other" : cwd, model: "codex-test", modelProvider: "openai", + approvalPolicy: "never", approvalsReviewer: "user", sandbox: { type: "dangerFullAccess" }, + thread: metadata(config.wrongIdentity ? "foreign-thread" : threadId) +}); +const notification = (method, params) => write({ method, params }); +const marker = (name, threadId = "private-source") => notification("serverRequest/resolved", { threadId, requestId: name }); +const delta = (name, threadId = "private-source") => notification("item/agentMessage/delta", { + threadId, turnId: "native-turn", itemId: "native-item", delta: name +}); +let pendingOpen; +let pendingFeedback; +let requestSequence = 500; +const serverRequest = () => { + const request = config.serverRequest; + const id = ++requestSequence; + write({ id, method: request.method, params: { threadId: "private-source", ...request.params } }); +}; +readline.createInterface({ input: process.stdin }).on("line", (line) => { + const message = JSON.parse(line); + log(message); + if (message.method === undefined) { + if (pendingOpen !== undefined) { + const id = pendingOpen; + pendingOpen = undefined; + reply(id, opened("private-source")); + } + if (pendingFeedback !== undefined) { + const id = pendingFeedback; + pendingFeedback = undefined; + marker("approval-complete"); + reply(id, { threadId: "private-source" }); + } + return; + } + const { id, method, params } = message; + if (method === "initialize") return reply(id, { userAgent: "exact-test", codexHome: cwd, platformFamily: "unix", platformOs: "linux" }); + if (method === "initialized") return; + if (method === "thread/resume" || method === "thread/start") { + if (method === "thread/resume" && config.missingThread) return write({ id, error: { code: -32603, message: "thread not found" } }); + loaded.add(method === "thread/start" ? "fresh-thread" : params.threadId); + if (config.startupTraffic) { + notification("thread/started", { thread: metadata("private-source") }); + delta("startup-private-text"); + pendingOpen = id; + serverRequest(); + return; + } + return reply(id, opened(method === "thread/start" ? "fresh-thread" : params.threadId)); + } + if (method === "thread/read") return reply(id, { thread: metadata(params.threadId) }); + if (method === "thread/goal/get") return reply(id, { goal: config.nativeGoal ? { ...config.nativeGoal, threadId: params.threadId } : null }); + if (method === "thread/fork") { + threads.set("private-fork", [...threads.get(params.threadId)]); + persistNative("private-fork", [nativeHeader("private-fork", params.threadId), ...nativeRecords.get(params.threadId)]); + loaded.add("private-fork"); + if (config.knownChildTraffic) { + const usage = { totalTokens: 9, inputTokens: 5, cachedInputTokens: 0, outputTokens: 4, reasoningOutputTokens: 0 }; + notification("thread/tokenUsage/updated", { threadId: "known-child", turnId: "child-completed-turn", tokenUsage: { total: usage, last: usage, modelContextWindow: 100 } }); + marker("known-receiver-receipt", "known-receiver"); + } + if (config.forkTraffic) { + notification("thread/started", { thread: metadata("private-fork") }); + delta("private-fork-secret", "private-fork"); + notification("turn/started", { threadId: "private-fork", turn: { id: "private-fork-turn", status: "inProgress", items: [] } }); + delta("parent-during-fork"); + } + return reply(id, { ...opened("private-fork"), thread: { ...metadata("private-fork"), forkedFromId: params.threadId } }); + } + if (method === "feedback/upload") { + if (params.reason === "register-children") { + notification("item/completed", { + threadId: "private-source", turnId: "parent-completed-turn", completedAtMs: 1, + item: { id: "child-activity", type: "subAgentActivity", agentThreadId: "known-child", agentPath: "/root/child", kind: "interacted" } + }); + notification("item/completed", { + threadId: "private-source", turnId: "parent-completed-turn", completedAtMs: 1, + item: { id: "receiver-call", type: "collabAgentToolCall", tool: "wait", status: "completed", senderThreadId: "private-source", receiverThreadIds: ["known-receiver"], agentsStates: {} } + }); + } + if (params.reason === "approval-race") { + pendingFeedback = id; + serverRequest(); + return; + } + if (params.reason === "unload-fork") loaded.delete("private-fork"); + if (params.reason === "after-fork") delta("late-private-fork-secret", "private-fork"); + delta(params.reason, params.threadId); + if (params.reason === "queued") delta("second-queued", params.threadId); + marker(params.reason + "-complete", params.threadId); + return reply(id, { threadId: params.threadId }); + } + if (method === "turn/start") return reply(id, { turn: { id: "new-native-turn", status: "inProgress", items: [] } }); + return write({ id, error: { code: -32601, message: "Unexpected method " + method } }); +}); +`; + +const commandRequest = { + method: "item/commandExecution/requestApproval", + params: { turnId: "native-turn", itemId: "native-command", startedAtMs: 1, command: "echo test" }, +}; +const approvalRequests = [ + commandRequest, + { + method: "item/fileChange/requestApproval", + params: { turnId: "native-turn", itemId: "native-file", startedAtMs: 1 }, + }, + { + method: "mcpServer/elicitation/request", + params: { + turnId: "native-turn", + serverName: "test", + mode: "form", + message: "Approve", + requestedSchema: { type: "object", properties: {} }, + }, + }, + { + method: "item/tool/requestUserInput", + params: { + turnId: "native-turn", + itemId: "native-input", + questions: [{ id: "question", header: "Choice", question: "Choose" }], + }, + }, +]; + +const makeHarness = Effect.fn("makeExactCodexRuntimeHarness")(function* ( + config: Schema.JsonObject = {}, + options: Pick = {}, +) { + const cwd = NodeFS.realpathSync( + NodeFS.mkdtempSync(NodePath.join(NodeOS.tmpdir(), "pylon-codex-exact-runtime-")), + ); + yield* Effect.addFinalizer(() => + Effect.sync(() => NodeFS.rmSync(cwd, { recursive: true, force: true })), + ); + const peerPath = NodePath.join(cwd, "peer.mjs"); + const configPath = NodePath.join(cwd, "config.json"); + const logPath = NodePath.join(cwd, "requests.jsonl"); + NodeFS.writeFileSync(peerPath, PEER); + // @effect-diagnostics-next-line preferSchemaOverJson:off + NodeFS.writeFileSync(configPath, JSON.stringify({ ...config, logPath })); + NodeFS.writeFileSync(logPath, ""); + const spawner = yield* ChildProcessSpawner.ChildProcessSpawner; + const crypto = yield* Crypto.Crypto; + const uuidEntered = yield* Deferred.make(); + const uuidReleased = yield* Deferred.make(); + let gateNextUuid = false; + const runtime = yield* makeCodexSessionRuntime({ + threadId: ThreadId.make("pylon-exact-runtime"), + binaryPath: process.execPath, + cwd, + homePath: NodePath.join(cwd, "codex-home"), + runtimeMode: "full-access", + environment: {}, + ...(config.preflightProof + ? { + exactRecoverySnapshot: { + threadId: "private-immutable", + cwd, + turns: [], + rolloutPath: NodePath.join(cwd, "private-immutable.jsonl"), + nativeRecords: [ + { + type: "session_meta", + payload: { cwd, base_instructions: { text: "native proof fixture" } }, + }, + ], + nativeGoal: null, + }, + } + : {}), + ...options, + }).pipe( + Effect.provideService(ChildProcessSpawner.ChildProcessSpawner, { + ...spawner, + spawn: () => + spawner.spawn( + ChildProcess.make(process.execPath, [peerPath, configPath], { + cwd, + extendEnv: false, + env: {}, + forceKillAfter: "1 second", + }), + ), + }), + Effect.provideService(Crypto.Crypto, { + ...crypto, + randomUUIDv4: Effect.suspend(() => { + if (!gateNextUuid) return crypto.randomUUIDv4; + gateNextUuid = false; + return Deferred.succeed(uuidEntered, undefined).pipe( + Effect.andThen(Deferred.await(uuidReleased)), + Effect.andThen(crypto.randomUUIDv4), + ); + }), + }), + ); + NodeAssert.ok(runtime.absoluteConversation); + return { + runtime, + absolute: runtime.absoluteConversation, + uuidEntered, + uuidReleased, + armUuid: Effect.sync(() => { + gateNextUuid = true; + }), + requests: () => + NodeFS.readFileSync(logPath, "utf8") + .split("\n") + .filter(Boolean) + .map((line) => decodeLogEntry(line)), + }; +}); + +const collectThroughMarker = (events: Stream.Stream, name: string) => + events.pipe( + Stream.takeUntil( + (event) => + event.method === "serverRequest/resolved" && + isMarkerPayload(event.payload) && + event.payload.requestId === name, + ), + Stream.runCollect, + ); + +describe("CodexSessionRuntime exact recovery", () => { + it.effect("preflights unloaded selected and immutable native histories before resuming", () => + Effect.gen(function* () { + const harness = yield* makeHarness( + { preflightProof: true, unloadedSnapshots: true }, + { strictResume: true, quarantined: true, resumeCursor: { threadId: SOURCE } }, + ); + yield* harness.runtime.start(); + const immutable = yield* harness.absolute.read("private-immutable"); + NodeAssert.equal(immutable.threadId, "private-immutable"); + NodeAssert.equal(immutable.nativeRecords.length, 1); + const requests = harness.requests(); + const resumeIndex = requests.findIndex((entry) => entry.method === "thread/resume"); + NodeAssert.ok(resumeIndex > 0); + NodeAssert.ok( + requests + .slice(0, resumeIndex) + .some( + (entry) => + entry.method === "thread/read" && entry.params?.threadId === "private-immutable", + ), + ); + NodeAssert.equal(requests.filter((entry) => entry.method === "thread/resume").length, 1); + NodeAssert.equal(requests[resumeIndex]?.params?.threadId, "private-fork"); + NodeAssert.equal( + requests.some( + (entry) => entry.method === "thread/resume" && entry.params?.threadId === SOURCE, + ), + false, + ); + NodeAssert.ok( + requests + .slice(0, resumeIndex) + .some( + (entry) => + entry.method === "thread/fork" && entry.params?.deferGoalContinuation === true, + ), + ); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + + for (const change of [ + { changedNative: true }, + { + nativeGoal: { + objective: "externally activated goal", + status: "active", + tokenBudget: null, + tokensUsed: 0, + timeUsedSeconds: 0, + createdAt: 1, + updatedAt: 1, + }, + }, + ]) + it.effect( + `rejects changed exact native proof before any native resume ${JSON.stringify(change)}`, + () => + Effect.gen(function* () { + const harness = yield* makeHarness( + { preflightProof: true, unloadedSnapshots: true, ...change }, + { strictResume: true, quarantined: true, resumeCursor: { threadId: SOURCE } }, + ); + yield* harness.runtime.start().pipe(Effect.flip); + NodeAssert.equal( + harness + .requests() + .some((entry) => entry.method === "thread/resume" || entry.method === "thread/start"), + false, + ); + NodeAssert.ok(harness.requests().some((entry) => entry.method === "thread/goal/get")); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + + it.effect( + "preserves settled child usage and receiver receipts arriving during a private fork", + () => + Effect.gen(function* () { + const harness = yield* makeHarness( + { knownChildTraffic: true }, + { resumeCursor: { threadId: SOURCE } }, + ); + yield* harness.runtime.start(); + const registration = yield* collectThroughMarker( + harness.runtime.events, + "register-children-complete", + ).pipe(Effect.forkScoped); + yield* harness.runtime.uploadFeedback("register-children"); + yield* Fiber.join(registration); + NodeAssert.equal(yield* harness.absolute.isIdle, true); + const collected = yield* collectThroughMarker( + harness.runtime.events, + "after-fork-complete", + ).pipe(Effect.forkScoped); + const source = yield* harness.absolute.read(); + yield* harness.absolute.fork({ source }); + yield* harness.runtime.uploadFeedback("after-fork"); + const events = Array.from(yield* Fiber.join(collected)); + const childUsage = events.find((event) => event.method === "collabAgent/tokenUsage"); + NodeAssert.ok(childUsage); + NodeAssert.ok( + typeof childUsage.payload === "object" && + childUsage.payload !== null && + "agentThreadId" in childUsage.payload, + ); + NodeAssert.equal(childUsage.payload.agentThreadId, "known-child"); + NodeAssert.equal( + events.find( + (event) => + isMarkerPayload(event.payload) && + event.payload.requestId === "known-receiver-receipt", + )?.turnId, + "parent-completed-turn", + ); + NodeAssert.equal( + events.some((event) => event.textDelta === "late-private-fork-secret"), + false, + ); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + it.effect( + "suppresses private fork lifecycle frames while preserving ordinary parent output", + () => + Effect.gen(function* () { + const harness = yield* makeHarness( + { forkTraffic: true }, + { resumeCursor: { threadId: SOURCE } }, + ); + yield* harness.runtime.start(); + const collected = yield* collectThroughMarker( + harness.runtime.events, + "after-fork-complete", + ).pipe(Effect.forkScoped); + const source = yield* harness.absolute.read(); + yield* harness.absolute.fork({ source }); + yield* harness.runtime.uploadFeedback("after-fork"); + const events = Array.from(yield* Fiber.join(collected)); + NodeAssert.deepEqual( + events.filter((event) => event.textDelta !== undefined).map((event) => event.textDelta), + ["parent-during-fork", "after-fork"], + ); + NodeAssert.equal( + events.some( + (event) => + event.method === "thread/started" || + event.method === "turn/started" || + event.method.startsWith("collabAgent/"), + ), + false, + ); + NodeAssert.equal(yield* harness.absolute.isIdle, true); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + it.effect( + "does not create a fresh native thread when strict resume cannot find the saved thread", + () => + Effect.gen(function* () { + const harness = yield* makeHarness( + { missingThread: true }, + { strictResume: true, quarantined: true, resumeCursor: { threadId: SOURCE } }, + ); + const error = yield* harness.runtime.start().pipe(Effect.flip); + NodeAssert.match(error.message, /thread not found/); + NodeAssert.deepEqual( + harness + .requests() + .filter((entry) => entry.method?.startsWith("thread/")) + .map((entry) => entry.method), + ["thread/resume"], + ); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + + it.effect("rejects a missing strict resume cursor before creating a native thread", () => + Effect.gen(function* () { + const harness = yield* makeHarness({}, { strictResume: true, quarantined: true }); + yield* harness.runtime.start().pipe(Effect.flip); + NodeAssert.equal( + harness.requests().some((entry) => entry.method === "thread/start"), + false, + ); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + + for (const config of [{ wrongIdentity: true }, { wrongCwd: true }]) { + it.effect(`rejects strict resume metadata mismatch ${JSON.stringify(config)}`, () => + Effect.gen(function* () { + const harness = yield* makeHarness(config, { + strictResume: true, + quarantined: true, + resumeCursor: { threadId: SOURCE }, + }); + const error = yield* harness.runtime.start().pipe(Effect.flip); + NodeAssert.match(error.message, /different conversation or workspace/); + NodeAssert.equal( + harness.requests().some((entry) => entry.method === "thread/start"), + false, + ); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + } + + it.effect( + "suppresses startup output and rejects startup approval before releasing recovery", + () => + Effect.gen(function* () { + const harness = yield* makeHarness( + { startupTraffic: true, serverRequest: commandRequest }, + { strictResume: true, quarantined: true, resumeCursor: { threadId: SOURCE } }, + ); + yield* harness.runtime.start(); + NodeAssert.equal(yield* harness.absolute.isIdle, true); + NodeAssert.ok(harness.requests().find((entry) => entry.id === 501)?.error); + yield* harness.absolute.quarantine(false); + const collected = yield* collectThroughMarker(harness.runtime.events, "live-complete").pipe( + Effect.forkScoped, + ); + yield* harness.runtime.uploadFeedback("live"); + const events = Array.from(yield* Fiber.join(collected)); + NodeAssert.deepEqual( + events.map((event) => event.method), + ["item/agentMessage/delta", "serverRequest/resolved"], + ); + NodeAssert.equal(events[0]?.textDelta, "live"); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + + it.effect( + "drops both queued public events and in-flight native notifications across quarantine release", + () => + Effect.gen(function* () { + const harness = yield* makeHarness({}, { resumeCursor: { threadId: SOURCE } }); + yield* harness.runtime.start(); + yield* harness.armUuid; + const queued = yield* harness.runtime.uploadFeedback("queued").pipe(Effect.forkScoped); + yield* Deferred.await(harness.uuidEntered); + yield* Fiber.join(queued); + yield* harness.absolute.quarantine(true); + yield* harness.absolute.quarantine(false); + yield* Deferred.succeed(harness.uuidReleased, undefined); + const collected = yield* collectThroughMarker(harness.runtime.events, "live-complete").pipe( + Effect.forkScoped, + ); + yield* harness.runtime.uploadFeedback("live"); + const events = Array.from(yield* Fiber.join(collected)); + NodeAssert.deepEqual( + events.map((event) => event.method), + ["item/agentMessage/delta", "serverRequest/resolved"], + ); + NodeAssert.equal(events[0]?.textDelta, "live"); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + + for (const request of approvalRequests) { + it.effect( + `rejects ${request.method} that was received before quarantine and resumed after release`, + () => + Effect.gen(function* () { + const harness = yield* makeHarness( + { serverRequest: request }, + { resumeCursor: { threadId: SOURCE } }, + ); + yield* harness.runtime.start(); + const seen: Array = []; + const collected = yield* harness.runtime.events.pipe( + Stream.tap((event) => + Effect.gen(function* () { + seen.push(event); + if (event.kind !== "request" || event.requestId === undefined) return; + // A buggy handler leaks an approval and would otherwise wait forever. + // Answer it so the peer supplies the same completion barrier on both paths. + if (event.method === "item/tool/requestUserInput") + yield* harness.runtime.respondToUserInput(event.requestId, {}); + else yield* harness.runtime.respondToRequest(event.requestId, "decline"); + }), + ), + (events) => collectThroughMarker(events, "approval-complete"), + Effect.forkScoped, + ); + yield* harness.armUuid; + const feedback = yield* harness.runtime + .uploadFeedback("approval-race") + .pipe(Effect.forkScoped); + yield* Deferred.await(harness.uuidEntered); + yield* harness.absolute.quarantine(true); + yield* harness.absolute.quarantine(false); + yield* Deferred.succeed(harness.uuidReleased, undefined); + yield* Fiber.join(feedback); + yield* Fiber.join(collected); + NodeAssert.deepEqual( + seen.filter((event) => event.kind === "request"), + [], + ); + NodeAssert.ok(harness.requests().find((entry) => entry.id === 501)?.error); + NodeAssert.equal(yield* harness.absolute.isIdle, true); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + } + + it.effect("refuses an unloaded selected target without resuming its immutable history", () => + Effect.gen(function* () { + const harness = yield* makeHarness( + { unloadedSnapshots: true }, + { strictResume: true, quarantined: true, resumeCursor: { threadId: SOURCE } }, + ); + yield* harness.runtime.start(); + const source = yield* harness.absolute.read(); + const fork = yield* harness.absolute.fork({ source }); + yield* harness.runtime.uploadFeedback("unload-fork"); + NodeAssert.ok(Exit.isFailure(yield* Effect.exit(harness.absolute.select(fork)))); + NodeAssert.deepEqual((yield* harness.runtime.getSession).resumeCursor, { threadId: SOURCE }); + NodeAssert.equal( + harness.requests().filter((entry) => entry.method === "thread/resume").length, + 1, + ); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); + + it.effect("selects a verified private fork and routes the next turn to its native identity", () => + Effect.gen(function* () { + const harness = yield* makeHarness( + {}, + { strictResume: true, quarantined: true, resumeCursor: { threadId: SOURCE } }, + ); + yield* harness.runtime.start(); + const source = yield* harness.absolute.read(); + const fork = yield* harness.absolute.fork({ source }); + yield* harness.absolute.select(fork); + NodeAssert.deepEqual((yield* harness.runtime.getSession).resumeCursor, { + threadId: "private-fork", + }); + yield* harness.absolute.quarantine(false); + const sent = yield* harness.runtime.sendTurn({ input: "Continue after rollback" }); + NodeAssert.deepEqual(sent.resumeCursor, { threadId: "private-fork" }); + NodeAssert.equal( + harness.requests().find((entry) => entry.method === "turn/start")?.params?.threadId, + "private-fork", + ); + NodeAssert.equal( + harness + .requests() + .some((entry) => entry.method === "thread/rollback" || entry.method === "thread/revert"), + false, + ); + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)), + ); +}); diff --git a/apps/server/src/provider/Layers/CodexSessionRuntime.ts b/apps/server/src/provider/Layers/CodexSessionRuntime.ts index 3286efe6f..2b3cdbdf2 100644 --- a/apps/server/src/provider/Layers/CodexSessionRuntime.ts +++ b/apps/server/src/provider/Layers/CodexSessionRuntime.ts @@ -1,3 +1,9 @@ +import { + codexConversationDigest, + forkCodexConversation, + readCodexConversation, + type CodexConversationSnapshot, +} from "./CodexAbsoluteHistory.ts"; import { ApprovalRequestId, DEFAULT_MODEL, @@ -20,10 +26,13 @@ import { import { resolveSpawnCommand } from "@t3tools/shared/shell"; import { normalizeModelSlug } from "@t3tools/shared/model"; import * as Crypto from "effect/Crypto"; +import * as Context from "effect/Context"; import * as DateTime from "effect/DateTime"; import * as Deferred from "effect/Deferred"; import * as Effect from "effect/Effect"; import * as Exit from "effect/Exit"; +import * as FileSystem from "effect/FileSystem"; +import * as Option from "effect/Option"; import * as Layer from "effect/Layer"; import * as Queue from "effect/Queue"; import * as Ref from "effect/Ref"; @@ -46,6 +55,9 @@ import { const decodeV2TurnStartResponse = Schema.decodeUnknownEffect(EffectCodexSchema.V2TurnStartResponse); const PROVIDER = ProviderDriverKind.make("codex"); +const CodexNotificationEpoch = Context.Reference("t3/CodexNotificationEpoch", { + defaultValue: () => undefined, +}); const ANSI_ESCAPE_CHAR = String.fromCharCode(27); const ANSI_ESCAPE_REGEX = new RegExp(`${ANSI_ESCAPE_CHAR}\\[[0-9;]*m`, "g"); @@ -178,6 +190,11 @@ export interface CodexSessionRuntimeOptions { readonly model?: string; readonly serviceTier?: CodexServiceTier | undefined; readonly resumeCursor?: CodexResumeCursor; + /** Private exact recovery never falls back to a different thread. */ + readonly strictResume?: boolean; + readonly quarantined?: boolean; + readonly exactRecoverySnapshot?: CodexConversationSnapshot; + readonly exactRecoveryCheckpoint?: CodexConversationSnapshot; readonly appServerArgs?: ReadonlyArray; /** Capabilities the session's `t3-code` MCP credential grants; drives the prompt blocks. */ readonly mcpCapabilities?: ReadonlySet; @@ -206,6 +223,20 @@ export interface CodexThreadSnapshot { } export interface CodexSessionRuntimeShape { + readonly absoluteConversation?: { + readonly isIdle: Effect.Effect; + readonly read: ( + threadId?: string, + ) => Effect.Effect; + readonly fork: (input: { + source: CodexConversationSnapshot; + lastTurnId?: string; + }) => Effect.Effect; + readonly select: ( + snapshot: CodexConversationSnapshot, + ) => Effect.Effect; + readonly quarantine: (enabled: boolean) => Effect.Effect; + }; readonly start: () => Effect.Effect; readonly getSession: Effect.Effect; readonly sendTurn: ( @@ -724,6 +755,7 @@ export const openCodexThread = (input: { readonly requestedModel: string | undefined; readonly serviceTier: CodexServiceTier | undefined; readonly resumeThreadId: string | undefined; + readonly strictResume?: boolean; }): Effect.Effect => { const resumeThreadId = input.resumeThreadId; const startParams = buildThreadStartParams({ @@ -734,6 +766,12 @@ export const openCodexThread = (input: { }); if (resumeThreadId === undefined) { + if (input.strictResume) + return Effect.fail( + CodexErrors.CodexAppServerRequestError.internalError( + "Strict Codex recovery requires an existing native conversation.", + ), + ); return input.client.request("thread/start", startParams); } @@ -758,14 +796,16 @@ export const openCodexThread = (input: { ), ), ), - Effect.catchIf(isRecoverableThreadResumeError, (error) => - Effect.logWarning("codex app-server thread resume fell back to fresh start", { - threadId: input.threadId, - requestedRuntimeMode: input.runtimeMode, - resumeThreadId, - recoverable: true, - cause: error, - }).pipe(Effect.andThen(input.client.request("thread/start", startParams))), + Effect.catchIf( + (error) => !input.strictResume && isRecoverableThreadResumeError(error), + (error) => + Effect.logWarning("codex app-server thread resume fell back to fresh start", { + threadId: input.threadId, + requestedRuntimeMode: input.runtimeMode, + resumeThreadId, + recoverable: true, + cause: error, + }).pipe(Effect.andThen(input.client.request("thread/start", startParams))), ), ); }; @@ -1288,13 +1328,18 @@ export const makeCodexSessionRuntime = ( ): Effect.Effect< CodexSessionRuntimeShape, CodexErrors.CodexAppServerError, - ChildProcessSpawner.ChildProcessSpawner | Crypto.Crypto | Scope.Scope + ChildProcessSpawner.ChildProcessSpawner | Crypto.Crypto | FileSystem.FileSystem | Scope.Scope > => Effect.gen(function* () { const spawner = yield* ChildProcessSpawner.ChildProcessSpawner; const runtimeScope = yield* Scope.Scope; const crypto = yield* Crypto.Crypto; - const events = yield* Queue.unbounded(); + const fileSystem = yield* FileSystem.FileSystem; + let quarantined = options.quarantined ?? false; + let eventEpoch = 0; + let privateForksInFlight = 0; + const privateSnapshotThreadIds = new Set(); + const events = yield* Queue.unbounded<{ event: ProviderEvent; epoch: number }>(); const pendingApprovalsRef = yield* Ref.make(new Map()); const approvalCorrelationsRef = yield* Ref.make(new Map()); const pendingUserInputsRef = yield* Ref.make(new Map()); @@ -1350,7 +1395,74 @@ export const makeCodexSessionRuntime = ( const client = yield* Effect.service(CodexClient.CodexAppServerClient).pipe( Effect.provide(clientContext), ); - const serverNotifications = yield* Queue.unbounded(); + const readRollout = (path: string) => + Effect.gen(function* () { + const maxBytes = 16 * 1024 * 1024; + const before = yield* fileSystem.stat(path); + if (before.type !== "File" || before.size <= 0n || before.size > BigInt(maxBytes)) + return yield* CodexErrors.CodexAppServerRequestError.internalError( + "The native Codex history is not a bounded regular file.", + ); + const file = yield* fileSystem.open(path, { flag: "r" }); + const opened = yield* file.stat; + const same = (a: FileSystem.File.Info, b: FileSystem.File.Info) => + a.type === "File" && + b.type === "File" && + a.dev === b.dev && + Option.isSome(a.ino) && + Option.isSome(b.ino) && + a.ino.value > 0 && + a.ino.value === b.ino.value && + a.size === b.size && + Option.getOrUndefined(a.mtime)?.getTime() === Option.getOrUndefined(b.mtime)?.getTime(); + if (!same(before, opened)) + return yield* CodexErrors.CodexAppServerRequestError.internalError( + "The native Codex history changed before reading.", + ); + const bytes = new Uint8Array(Number(opened.size) + 1); + let size = 0; + while (size < bytes.byteLength) { + const count = Number(yield* file.read(bytes.subarray(size))); + if (count <= 0) break; + size += count; + } + const after = yield* file.stat; + const pathAfter = yield* fileSystem.stat(path); + if (size !== Number(opened.size) || !same(opened, after) || !same(opened, pathAfter)) + return yield* CodexErrors.CodexAppServerRequestError.internalError( + "The native Codex history changed during reading.", + ); + return yield* Effect.try({ + try: () => new TextDecoder("utf-8", { fatal: true }).decode(bytes.subarray(0, size)), + catch: () => + CodexErrors.CodexAppServerRequestError.internalError( + "The native Codex history is not valid UTF-8.", + ), + }); + }).pipe( + Effect.scoped, + Effect.mapError(() => + CodexErrors.CodexAppServerRequestError.internalError( + "The complete native Codex history could not be verified.", + ), + ), + ); + const exactForkConfig = runtimeModeToThreadConfig(options.runtimeMode); + const exactHistoryClient = { + raw: client.raw, + readRollout, + forkOptions: { + approvalPolicy: exactForkConfig.approvalPolicy, + sandbox: exactForkConfig.sandbox, + approvalsReviewer: exactForkConfig.approvalsReviewer, + ...(options.model ? { model: options.model } : {}), + ...(options.serviceTier ? { serviceTier: options.serviceTier } : {}), + }, + }; + const serverNotifications = yield* Queue.unbounded<{ + notification: CodexServerNotification; + epoch: number; + }>(); const nowIso = Effect.map(DateTime.now, DateTime.formatIso); const randomUUIDv4 = (purpose: CodexErrors.CodexAppServerIdentifierPurpose) => crypto.randomUUIDv4.pipe( @@ -1377,18 +1489,30 @@ export const makeCodexSessionRuntime = ( updatedAt: sessionCreatedAt, } satisfies ProviderSession; const sessionRef = yield* Ref.make(initialSession); - const offerEvent = (event: ProviderEvent) => Queue.offer(events, event).pipe(Effect.asVoid); + const offerEvent = (event: ProviderEvent, epoch: number) => + Effect.suspend(() => + quarantined || epoch !== eventEpoch + ? Effect.void + : Queue.offer(events, { event, epoch }).pipe(Effect.asVoid), + ); const emitEvent = (event: Omit) => Effect.gen(function* () { + const epoch = (yield* CodexNotificationEpoch) ?? eventEpoch; + if (quarantined || epoch !== eventEpoch) return; const id = yield* randomUUIDv4("provider-event"); - return yield* offerEvent({ - id: EventId.make(id), - provider: PROVIDER, - ...(options.providerInstanceId ? { providerInstanceId: options.providerInstanceId } : {}), - createdAt: yield* nowIso, - ...event, - }); + return yield* offerEvent( + { + id: EventId.make(id), + provider: PROVIDER, + ...(options.providerInstanceId + ? { providerInstanceId: options.providerInstanceId } + : {}), + createdAt: yield* nowIso, + ...event, + }, + epoch, + ); }); const emitSessionEvent = (method: string, message: string) => emitEvent({ @@ -1871,6 +1995,19 @@ export const makeCodexSessionRuntime = ( const handleRawNotification = (notification: CodexServerNotification) => Effect.gen(function* () { + const notificationThreadId = readNotificationThreadId(notification); + const selectedThreadId = currentProviderThreadId(yield* Ref.get(sessionRef)); + const knownChild = + notificationThreadId !== undefined && + ((yield* Ref.get(collabChildAgentsRef)).has(notificationThreadId) || + (yield* Ref.get(collabReceiverTurnsRef)).has(notificationThreadId)); + if ( + notificationThreadId !== undefined && + notificationThreadId !== selectedThreadId && + (privateSnapshotThreadIds.has(notificationThreadId) || + (privateForksInFlight > 0 && !knownChild)) + ) + return; const isMemoryConsolidationNotification = suppressMemoryConsolidationNotification(notification); @@ -2064,11 +2201,20 @@ export const makeCodexSessionRuntime = ( yield* client.handleServerRequest("item/commandExecution/requestApproval", (payload) => Effect.gen(function* () { + const requestEpoch = eventEpoch; + if (quarantined) + return yield* CodexErrors.CodexAppServerRequestError.invalidParams( + "Codex conversation is held for exact recovery.", + ); const requestId = ApprovalRequestId.make(yield* randomUUIDv4("command-approval-request")); const turnId = TurnId.make(payload.turnId); const itemId = ProviderItemId.make(payload.itemId); const decision = yield* Deferred.make(); + if (quarantined || requestEpoch !== eventEpoch) + return yield* CodexErrors.CodexAppServerRequestError.invalidParams( + "The Codex request belongs to a retired conversation boundary.", + ); yield* Ref.update(pendingApprovalsRef, (current) => { const next = new Map(current); next.set(requestId, { @@ -2101,7 +2247,7 @@ export const makeCodexSessionRuntime = ( ...(turnId ? { turnId } : {}), ...(itemId ? { itemId } : {}), payload, - }); + }).pipe(Effect.provideService(CodexNotificationEpoch, requestEpoch)); const resolved = yield* Deferred.await(decision).pipe( Effect.ensuring( @@ -2120,6 +2266,11 @@ export const makeCodexSessionRuntime = ( yield* client.handleServerRequest("item/fileChange/requestApproval", (payload) => Effect.gen(function* () { + const requestEpoch = eventEpoch; + if (quarantined) + return yield* CodexErrors.CodexAppServerRequestError.invalidParams( + "Codex conversation is held for exact recovery.", + ); const requestId = ApprovalRequestId.make( yield* randomUUIDv4("file-change-approval-request"), ); @@ -2127,6 +2278,10 @@ export const makeCodexSessionRuntime = ( const itemId = ProviderItemId.make(payload.itemId); const decision = yield* Deferred.make(); + if (quarantined || requestEpoch !== eventEpoch) + return yield* CodexErrors.CodexAppServerRequestError.invalidParams( + "The Codex request belongs to a retired conversation boundary.", + ); yield* Ref.update(pendingApprovalsRef, (current) => { const next = new Map(current); next.set(requestId, { @@ -2159,7 +2314,7 @@ export const makeCodexSessionRuntime = ( ...(turnId ? { turnId } : {}), ...(itemId ? { itemId } : {}), payload, - }); + }).pipe(Effect.provideService(CodexNotificationEpoch, requestEpoch)); const resolved = yield* Deferred.await(decision).pipe( Effect.ensuring( @@ -2178,6 +2333,11 @@ export const makeCodexSessionRuntime = ( yield* client.handleServerRequest("mcpServer/elicitation/request", (payload) => Effect.gen(function* () { + const requestEpoch = eventEpoch; + if (quarantined) + return yield* CodexErrors.CodexAppServerRequestError.invalidParams( + "Codex conversation is held for exact recovery.", + ); if (toMcpElicitationResponse(payload, "accept").action !== "accept") { yield* Effect.logWarning("Declined an unsupported MCP elicitation.", { serverName: payload.serverName, @@ -2195,6 +2355,10 @@ export const makeCodexSessionRuntime = ( const jsonRpcId = payload.mode === "url" ? payload.elicitationId : requestId; const decision = yield* Deferred.make(); + if (quarantined || requestEpoch !== eventEpoch) + return yield* CodexErrors.CodexAppServerRequestError.invalidParams( + "The Codex request belongs to a retired conversation boundary.", + ); yield* Ref.update(pendingApprovalsRef, (current) => { const next = new Map(current); next.set(requestId, { @@ -2226,7 +2390,7 @@ export const makeCodexSessionRuntime = ( requestKind: "mcp-elicitation", ...(turnId ? { turnId } : {}), payload, - }); + }).pipe(Effect.provideService(CodexNotificationEpoch, requestEpoch)); const resolved = yield* Deferred.await(decision).pipe( Effect.ensuring( @@ -2243,11 +2407,20 @@ export const makeCodexSessionRuntime = ( yield* client.handleServerRequest("item/tool/requestUserInput", (payload) => Effect.gen(function* () { + const requestEpoch = eventEpoch; + if (quarantined) + return yield* CodexErrors.CodexAppServerRequestError.invalidParams( + "Codex conversation is held for exact recovery.", + ); const requestId = ApprovalRequestId.make(yield* randomUUIDv4("user-input-request")); const turnId = TurnId.make(payload.turnId); const itemId = ProviderItemId.make(payload.itemId); const answers = yield* Deferred.make(); + if (quarantined || requestEpoch !== eventEpoch) + return yield* CodexErrors.CodexAppServerRequestError.invalidParams( + "The Codex request belongs to a retired conversation boundary.", + ); yield* Ref.update(pendingUserInputsRef, (current) => { const next = new Map(current); next.set(requestId, { @@ -2267,7 +2440,7 @@ export const makeCodexSessionRuntime = ( ...(turnId ? { turnId } : {}), ...(itemId ? { itemId } : {}), payload, - }); + }).pipe(Effect.provideService(CodexNotificationEpoch, requestEpoch)); const resolvedAnswers = yield* Deferred.await(answers).pipe( Effect.ensuring( @@ -2297,9 +2470,28 @@ export const makeCodexSessionRuntime = ( const registerServerNotification = (method: M) => client.handleServerNotification(method, (params) => - Queue.offer(serverNotifications, makeCodexServerNotification(method, params)).pipe( - Effect.asVoid, - ), + Effect.gen(function* () { + if (quarantined) return; + const epoch = eventEpoch; + const notification = makeCodexServerNotification(method, params); + const notificationThreadId = readNotificationThreadId(notification); + const selectedThreadId = currentProviderThreadId(yield* Ref.get(sessionRef)); + const knownChild = + notificationThreadId !== undefined && + ((yield* Ref.get(collabChildAgentsRef)).has(notificationThreadId) || + (yield* Ref.get(collabReceiverTurnsRef)).has(notificationThreadId)); + if ( + notificationThreadId !== undefined && + notificationThreadId !== selectedThreadId && + (privateSnapshotThreadIds.has(notificationThreadId) || + (privateForksInFlight > 0 && !knownChild)) + ) { + privateSnapshotThreadIds.add(notificationThreadId); + return; + } + if (quarantined || epoch !== eventEpoch) return; + yield* Queue.offer(serverNotifications, { notification, epoch }); + }), ); yield* Effect.forEach( @@ -2311,7 +2503,15 @@ export const makeCodexSessionRuntime = ( ); yield* Stream.fromQueue(serverNotifications).pipe( - Stream.runForEach(handleRawNotification), + Stream.runForEach(({ notification, epoch }) => + Effect.suspend(() => + quarantined || epoch !== eventEpoch + ? Effect.void + : handleRawNotification(notification).pipe( + Effect.provideService(CodexNotificationEpoch, epoch), + ), + ), + ), Effect.forkIn(runtimeScope), ); @@ -2329,6 +2529,7 @@ export const makeCodexSessionRuntime = ( Effect.forEach( lines, (line) => { + if (privateForksInFlight > 0) return Effect.void; const classified = classifyCodexStderrLine(line); if (!classified) { return Effect.void; @@ -2337,7 +2538,11 @@ export const makeCodexSessionRuntime = ( kind: "notification", threadId: options.threadId, method: "process/stderr", - message: classified.message, + message: Array.from(privateSnapshotThreadIds).some((id) => + classified.message.includes(id), + ) + ? "Codex reported a diagnostic for a private conversation snapshot." + : classified.message, }); }, { discard: true }, @@ -2382,6 +2587,49 @@ export const makeCodexSessionRuntime = ( const requestedModel = normalizeCodexModelSlug(options.model); + let resumeThreadId = readResumeCursorThreadId(options.resumeCursor); + if (options.exactRecoverySnapshot) { + const selectedId = readResumeCursorThreadId(options.resumeCursor); + if (!selectedId) + return yield* CodexErrors.CodexAppServerRequestError.internalError( + "Exact Codex recovery requires a selected native conversation.", + ); + const expected = options.exactRecoverySnapshot; + const selected = yield* readCodexConversation(exactHistoryClient, selectedId, options.cwd); + const immutable = yield* readCodexConversation( + exactHistoryClient, + expected.threadId, + options.cwd, + ); + if ( + codexConversationDigest(selected) !== codexConversationDigest(expected) || + codexConversationDigest(immutable) !== codexConversationDigest(expected) + ) + return yield* CodexErrors.CodexAppServerRequestError.internalError( + "The private Codex recovery proof changed before native resume.", + ); + if (options.exactRecoveryCheckpoint) { + const checkpoint = yield* readCodexConversation( + exactHistoryClient, + options.exactRecoveryCheckpoint.threadId, + options.cwd, + ); + if ( + codexConversationDigest(checkpoint) !== + codexConversationDigest(options.exactRecoveryCheckpoint) + ) + return yield* CodexErrors.CodexAppServerRequestError.internalError( + "The original Codex checkpoint changed before native resume.", + ); + } + // A fresh fork persists goal-continuation deferral atomically. Resume that verified + // private owner, so activating an old native goal between reads cannot start recovery input. + const recoveredFork = yield* forkCodexConversation(exactHistoryClient, { + source: selected, + }); + privateSnapshotThreadIds.add(recoveredFork.threadId); + resumeThreadId = recoveredFork.threadId; + } const opened = yield* openCodexThread({ client, threadId: options.threadId, @@ -2389,8 +2637,17 @@ export const makeCodexSessionRuntime = ( cwd: options.cwd, requestedModel, serviceTier: options.serviceTier, - resumeThreadId: readResumeCursorThreadId(options.resumeCursor), + resumeThreadId, + ...(options.strictResume ? { strictResume: true } : {}), }); + if ( + options.strictResume && + (opened.thread.id !== resumeThreadId || opened.cwd !== options.cwd) + ) { + return yield* CodexErrors.CodexAppServerRequestError.internalError( + "Exact Codex recovery resumed a different conversation or workspace.", + ); + } const providerThreadId = opened.thread.id; const session = { @@ -2437,7 +2694,78 @@ export const makeCodexSessionRuntime = ( yield* Queue.shutdown(events); }); + const isExactIdle = Effect.gen(function* () { + const session = yield* Ref.get(sessionRef); + return ( + !(yield* Ref.get(closedRef)) && + session.status === "ready" && + !session.activeTurnId && + (yield* Ref.get(pendingApprovalsRef)).size === 0 && + (yield* Ref.get(pendingUserInputsRef)).size === 0 && + (yield* Ref.get(collabChildLiveTurnsRef)).size === 0 + ); + }); + const requireExactIdle = Effect.gen(function* () { + if (!(yield* isExactIdle)) + return yield* CodexErrors.CodexAppServerRequestError.internalError( + "Exact Codex recovery requires an idle conversation with no pending input or agents.", + ); + }); return { + absoluteConversation: { + isIdle: isExactIdle, + read: (threadId) => + Effect.gen(function* () { + yield* requireExactIdle; + return yield* readCodexConversation( + exactHistoryClient, + threadId ?? (yield* readProviderThreadId), + options.cwd, + { requireLoaded: threadId === undefined }, + ); + }), + fork: (input) => + Effect.gen(function* () { + yield* requireExactIdle; + privateForksInFlight += 1; + return yield* forkCodexConversation(exactHistoryClient, input).pipe( + Effect.tap((snapshot) => + Effect.sync(() => privateSnapshotThreadIds.add(snapshot.threadId)), + ), + Effect.ensuring( + Effect.sync(() => { + privateForksInFlight -= 1; + }), + ), + ); + }), + select: (snapshot) => + Effect.gen(function* () { + yield* requireExactIdle; + const verified = yield* readCodexConversation( + exactHistoryClient, + snapshot.threadId, + options.cwd, + { requireLoaded: true }, + ); + if (codexConversationDigest(verified) !== codexConversationDigest(snapshot)) { + return yield* CodexErrors.CodexAppServerRequestError.internalError( + "The private Codex conversation changed before selection.", + ); + } + yield* requireExactIdle; + yield* updateSession(sessionRef, { + resumeCursor: { threadId: verified.threadId }, + activeTurnId: undefined, + status: "ready", + }); + }), + quarantine: (enabled) => + Effect.sync(() => { + quarantined = enabled; + eventEpoch += 1; + }), + }, start, getSession: Ref.get(sessionRef), compactThread: Effect.gen(function* () { @@ -2617,7 +2945,10 @@ export const makeCodexSessionRuntime = ( }, }); }), - events: Stream.fromQueue(events), + events: Stream.fromQueue(events).pipe( + Stream.filter(({ epoch }) => !quarantined && epoch === eventEpoch), + Stream.map(({ event }) => event), + ), close, } satisfies CodexSessionRuntimeShape; }); diff --git a/apps/server/src/provider/Services/ProviderAdapter.test.ts b/apps/server/src/provider/Services/ProviderAdapter.test.ts index 3fc80b0f6..724cd2102 100644 --- a/apps/server/src/provider/Services/ProviderAdapter.test.ts +++ b/apps/server/src/provider/Services/ProviderAdapter.test.ts @@ -6,7 +6,7 @@ describe("built-in provider conversation rollback modes", () => { it("classifies production adapters by their verified rollback support", () => { expect(BUILT_IN_ADAPTER_CONVERSATION_ROLLBACK_MODES).toEqual({ antigravity: "unsupported", - codex: "relative", + codex: "absolute", claude: "relative", cursor: "unsupported", grok: "unsupported", diff --git a/apps/server/src/provider/Services/ProviderAdapter.ts b/apps/server/src/provider/Services/ProviderAdapter.ts index b38276737..5f6ecd013 100644 --- a/apps/server/src/provider/Services/ProviderAdapter.ts +++ b/apps/server/src/provider/Services/ProviderAdapter.ts @@ -66,7 +66,7 @@ export type ProviderConversationRollbackMode = "absolute" | "relative" | "unsupp export const BUILT_IN_ADAPTER_CONVERSATION_ROLLBACK_MODES = { antigravity: "unsupported", - codex: "relative", + codex: "absolute", claude: "relative", cursor: "unsupported", grok: "unsupported", diff --git a/docs/internals/rollback-recovery.md b/docs/internals/rollback-recovery.md index 13ae29da7..2b73b39c9 100644 --- a/docs/internals/rollback-recovery.md +++ b/docs/internals/rollback-recovery.md @@ -41,7 +41,7 @@ A target is published as available only after checkpoint capture proves all of t 1. the checkpoint is immutable and ready; 2. the provider adapter advertises the absolute rollback gate; -3. the live session is the managed native Prime or exact-capable OpenCode session for the projected incarnation; +3. the live session is the managed native Prime or exact-capable OpenCode or Codex session for the projected incarnation; 4. a matching exact provider anchor was stored for that checkpoint. Admission repeats the proof, checks the exact source revision, requires an idle thread and empty provider queues, and acquires the canonical workspace lease. Published availability is never admission authority. @@ -50,6 +50,8 @@ OpenCode snapshots native history into private immutable forks and verifies full OpenCode's optional experimental plan mode stores a plan file under a name derived from the native session's creation time and slug. Native forks regenerate that identity without copying the file, so exact rewind is unavailable when `OPENCODE_EXPERIMENTAL_PLAN_MODE` is enabled, including through `OPENCODE_EXPERIMENTAL` unless the plan-mode flag explicitly disables it. Externally managed OpenCode servers also remain ineligible because their API does not expose that runtime flag. Ordinary plan mode, conversation history and resume remain usable; Pylon neither changes these flags nor copies plan files during rewind. +Codex captures immutable full native forks for owned completed turns. Exact proof includes the complete bounded native JSONL history and inactive native goal state; fresh forks defer goal continuation until the next explicit send. Original checkpoints survive compaction separately from current source proofs. Recovery verifies the same account, workspace and incarnation before selecting a fresh deferred fork. Active or uninspectable goals, paginated forks, external history bases, non-regular files, histories over 16 MiB or 100,000 records, and lines over 1 MiB are ineligible. Imported history has no guessed root or old Pylon turn bindings. Ordinary resume remains available. + ## Multi-client behavior The engine serializes admission. Requests for the same source, target and file choice join the active operation. A different target or file choice is rejected. Status is projected and streamed, so refresh, reconnect, remote clients, and multiple devices converge on the same fence and result. diff --git a/docs/user/providers-codex.md b/docs/user/providers-codex.md index a4b80ecfd..95b0c2d69 100644 --- a/docs/user/providers-codex.md +++ b/docs/user/providers-codex.md @@ -90,3 +90,7 @@ In an existing Codex thread, send `/feedback` with an optional description, for `/feedback The agent stopped before finishing the tests`. This uploads the conversation and Codex logs to OpenAI without adding messages to the thread. When the upload succeeds, choose **Copy ID** to share the thread ID with OpenAI support. + +## Rewind a conversation + +Eligible completed turns support rewinding the conversation with either restored files or your current files. Pylon keeps private native snapshots and verifies the selected history before applying a rewind. Older or imported turns without a captured snapshot remain unavailable. Active native goals and histories that cannot be verified also prevent rewind; ordinary conversation resume remains usable.