-
Notifications
You must be signed in to change notification settings - Fork 3.1k
refactor(onboard): add versioned resume checkpoint with tri-state decisions #7022
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
41 commits
Select commit
Hold shift + click to select a range
c02a9c3
refactor(onboard): add versioned resume checkpoint with tri-state dec…
laitingsheng 5142e9a
fix(onboard): wire checkpoint replay into the live sandbox executor
laitingsheng baa0e11
Merge remote-tracking branch 'origin/main' into refactor/onboard-vers…
laitingsheng b7053f3
test(onboard): keep changed checkpoint test bodies linear
laitingsheng 395d850
Merge remote-tracking branch 'origin/main' into refactor/onboard-vers…
laitingsheng 26de066
fix(onboard): close checkpoint crash-recovery and validation gaps
laitingsheng cfb0934
Merge remote-tracking branch 'origin/main' into refactor/onboard-vers…
laitingsheng 8c5cbc3
fix(onboard): keep the checkpoint resume wiring net-neutral for onboa…
laitingsheng cae7157
Merge remote-tracking branch 'origin/main' into refactor/onboard-vers…
laitingsheng e43a56b
fix(onboard): make the checkpoint the sole resume authority
laitingsheng 05c9903
fix(onboard): keep the checkpoint machine state live and record every…
laitingsheng 62daa7f
Merge remote-tracking branch 'origin/main' into refactor/onboard-vers…
laitingsheng c614f38
fix(onboard): wire effect-group replay and revalidate build/policy dr…
laitingsheng f4ded27
fix(onboard): revalidate provider bindings by type and credential env…
laitingsheng 940bf40
Merge remote-tracking branch 'origin/main' into refactor/onboard-vers…
laitingsheng 62c2ac7
fix(onboard): make the checkpoint the resume authority for sandbox de…
laitingsheng ab9d509
fix(onboard): address CodeRabbit findings on checkpoint resume authority
laitingsheng 69c0081
test(onboard): keep crash-recovery test bodies linear for the growth …
laitingsheng c084f53
fix(onboard): recover the checkpointed sandbox name for non-OpenClaw …
laitingsheng 4ef84bd
Merge remote-tracking branch 'origin/main' into refactor/onboard-vers…
laitingsheng 069f87e
test(state): wire the resume checkpoint resolver into the merged Stat…
laitingsheng 235e1a2
Merge remote-tracking branch 'origin/main' into refactor/onboard-vers…
laitingsheng 18b50f2
fix(onboard): revalidate checkpoint bindings at the locked destructiv…
laitingsheng aa168a3
fix(onboard): fingerprint the full resolved package and policy input …
laitingsheng ae1278c
fix(onboard): revalidate the full resolved create intent at the locke…
laitingsheng 6ceafc5
test(onboard): exercise real checkpoint resolution in the non-OpenCla…
laitingsheng cadd549
Merge remote-tracking branch 'origin/main' into refactor/onboard-vers…
laitingsheng 6b2a592
fix(onboard): backfill sandbox checkpoint receipts
cv 05176f0
merge(main): refresh PR 7022
cv cfdf1e8
test(onboard): match checkpoint update contract
cv c0dad7e
test(onboard): keep crash recovery setup linear
cv 5e8654d
fix(onboard): accept scrubbed registered credentials
cv cba7d67
Merge remote-tracking branch 'origin/main' into codex/salvage-7022-re…
cv 2621de7
docs(onboard): update live replay contract
cv 07faa70
merge: refresh PR #7022 from main
cv f8837e3
test(onboard): cover provider receipt loss by mode
cv 0f2d39c
fix(onboard): scope checkpoint credential bindings
cv 38fd79e
test(onboard): complete checkpoint crash matrix
cv 728f9a7
docs(onboard): clarify checkpoint credential ownership
cv 78a19cf
fix(onboard): reconcile extra providers on resume
cv e267a74
test(onboard): keep checkpoint setup linear
cv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,130 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import type { WebSearchConfig } from "../inference/web-search"; | ||
| import type { SandboxMessagingPlan } from "../messaging/manifest"; | ||
| import { | ||
| getActiveChannelIdsFromPlan, | ||
| getDisabledChannelIdsFromPlan, | ||
| } from "../messaging/plan-validation"; | ||
| import { decisionDeclined, decisionSelected } from "../state/onboard-checkpoint-decision"; | ||
| import { deriveCheckpointFromSession } from "../state/onboard-checkpoint-migrate"; | ||
| import type { | ||
| CheckpointEffectGroupName, | ||
| CheckpointProviderBinding, | ||
| CheckpointResourceProfile, | ||
| OnboardCheckpoint, | ||
| } from "../state/onboard-checkpoint-types"; | ||
| import type { Session } from "../state/onboard-session"; | ||
|
|
||
| function baseCheckpoint(session: Session): OnboardCheckpoint { | ||
| return session.checkpoint ?? deriveCheckpointFromSession(session); | ||
| } | ||
|
|
||
| export function recordCheckpointSandboxIdentity( | ||
| session: Session, | ||
| name: string, | ||
| agent: string, | ||
| ): void { | ||
| const base = baseCheckpoint(session); | ||
| session.checkpoint = { | ||
| ...base, | ||
| machineState: session.machine.state, | ||
| updatedAt: new Date().toISOString(), | ||
| sandboxIdentity: decisionSelected({ name, agent }), | ||
| }; | ||
| } | ||
|
|
||
| export function recordCheckpointEffectGroup( | ||
| session: Session, | ||
| group: CheckpointEffectGroupName, | ||
| fingerprint: string, | ||
| ): void { | ||
| const base = baseCheckpoint(session); | ||
| const now = new Date().toISOString(); | ||
| session.checkpoint = { | ||
| ...base, | ||
| machineState: session.machine.state, | ||
| updatedAt: now, | ||
| effectGroups: { | ||
| ...base.effectGroups, | ||
| [group]: { completedAt: now, fingerprint }, | ||
| }, | ||
| }; | ||
| } | ||
|
|
||
| export function recordCheckpointWebSearch( | ||
| session: Session, | ||
| webSearchConfig: WebSearchConfig | null, | ||
| ): void { | ||
| const base = baseCheckpoint(session); | ||
| session.checkpoint = { | ||
| ...base, | ||
| machineState: session.machine.state, | ||
| updatedAt: new Date().toISOString(), | ||
| webSearch: webSearchConfig ? decisionSelected(webSearchConfig) : decisionDeclined(), | ||
| }; | ||
| } | ||
|
|
||
| export function recordCheckpointMessaging( | ||
| session: Session, | ||
| messagingPlan: SandboxMessagingPlan | null, | ||
| ): void { | ||
| const base = baseCheckpoint(session); | ||
| session.checkpoint = { | ||
| ...base, | ||
| machineState: session.machine.state, | ||
| updatedAt: new Date().toISOString(), | ||
| messaging: messagingPlan | ||
| ? decisionSelected({ | ||
| selectedChannels: getActiveChannelIdsFromPlan(messagingPlan), | ||
| disabledChannels: getDisabledChannelIdsFromPlan(messagingPlan), | ||
| }) | ||
| : decisionDeclined(), | ||
| }; | ||
| } | ||
|
|
||
| export function recordCheckpointResourceProfile( | ||
| session: Session, | ||
| resourceProfile: CheckpointResourceProfile | null, | ||
| ): void { | ||
| const base = baseCheckpoint(session); | ||
| session.checkpoint = { | ||
| ...base, | ||
| machineState: session.machine.state, | ||
| updatedAt: new Date().toISOString(), | ||
| resourceProfile: resourceProfile ? decisionSelected(resourceProfile) : decisionDeclined(), | ||
| }; | ||
| } | ||
|
|
||
| export function recordCheckpointBindings( | ||
| session: Session, | ||
| additions: { | ||
| registeredProviders?: readonly CheckpointProviderBinding[]; | ||
| }, | ||
| ): void { | ||
| const base = baseCheckpoint(session); | ||
| const credentialEnvs = additions.registeredProviders | ||
| ? [ | ||
| ...new Set([ | ||
| ...base.bindings.credentialEnvs, | ||
| ...additions.registeredProviders.map((binding) => binding.credentialEnv), | ||
| ]), | ||
| ] | ||
| : base.bindings.credentialEnvs; | ||
| const registeredProviders = additions.registeredProviders | ||
| ? [ | ||
| ...new Map( | ||
| [...base.bindings.registeredProviders, ...additions.registeredProviders].map( | ||
| (binding) => [binding.name, binding], | ||
| ), | ||
| ).values(), | ||
| ] | ||
| : base.bindings.registeredProviders; | ||
| session.checkpoint = { | ||
| ...base, | ||
| machineState: session.machine.state, | ||
| updatedAt: new Date().toISOString(), | ||
| bindings: { credentialEnvs, registeredProviders }, | ||
| }; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import { describe, expect, it } from "vitest"; | ||
|
|
||
| import { decisionSelected, decisionUnset } from "../state/onboard-checkpoint-decision"; | ||
| import { | ||
| CHECKPOINT_SCHEMA_VERSION, | ||
| type OnboardCheckpoint, | ||
| } from "../state/onboard-checkpoint-types"; | ||
| import { planEffectGroupReplay, planSandboxCreateReplay } from "./checkpoint-replay"; | ||
| import { bindingRevalidationGuidance, revalidateCheckpointBindings } from "./checkpoint-revalidate"; | ||
|
|
||
| const ISO = "2026-01-01T00:00:00.000Z"; | ||
|
|
||
| function checkpoint(overrides: Partial<OnboardCheckpoint> = {}): OnboardCheckpoint { | ||
| return { | ||
| schemaVersion: CHECKPOINT_SCHEMA_VERSION, | ||
| sessionId: "s1", | ||
| machineState: "sandbox", | ||
| updatedAt: ISO, | ||
| sandboxIdentity: decisionSelected({ name: "my-sandbox", agent: "openclaw" }), | ||
| webSearch: decisionUnset(), | ||
| messaging: decisionUnset(), | ||
| resourceProfile: decisionUnset(), | ||
| effectGroups: {}, | ||
| bindings: { credentialEnvs: [], registeredProviders: [] }, | ||
| ...overrides, | ||
| }; | ||
| } | ||
|
|
||
| describe("planEffectGroupReplay", () => { | ||
| it("runs an unrecorded effect group", () => { | ||
| expect(planEffectGroupReplay(checkpoint(), "messaging_providers", true).action).toBe("run"); | ||
| }); | ||
|
|
||
| it("re-runs a recorded group whose postcondition no longer holds (never blind skip)", () => { | ||
| const cp = checkpoint({ | ||
| effectGroups: { messaging_providers: { completedAt: ISO, fingerprint: "fp" } }, | ||
| }); | ||
| const decision = planEffectGroupReplay(cp, "messaging_providers", false); | ||
| expect(decision).toEqual({ | ||
| group: "messaging_providers", | ||
| action: "run", | ||
| reason: "postcondition_failed", | ||
| }); | ||
| }); | ||
|
|
||
| it("skips a recorded group only after its postcondition is revalidated", () => { | ||
| const cp = checkpoint({ | ||
| effectGroups: { messaging_providers: { completedAt: ISO, fingerprint: "fp" } }, | ||
| }); | ||
| expect(planEffectGroupReplay(cp, "messaging_providers", true).action).toBe("skip"); | ||
| }); | ||
| }); | ||
|
|
||
| describe("planSandboxCreateReplay never opens a second sandbox (#5961)", () => { | ||
| it("requires identity capture before any create when identity is not durable", () => { | ||
| const cp = checkpoint({ sandboxIdentity: decisionUnset() }); | ||
| expect(planSandboxCreateReplay(cp, { liveSandboxExists: false })).toEqual({ | ||
| action: "capture_identity_first", | ||
| }); | ||
| }); | ||
|
|
||
| it("reuses the live sandbox when create is recorded and it still exists", () => { | ||
| const cp = checkpoint({ | ||
| effectGroups: { sandbox_create: { completedAt: ISO, fingerprint: "fp" } }, | ||
| }); | ||
| expect(planSandboxCreateReplay(cp, { liveSandboxExists: true })).toEqual({ | ||
| action: "reuse", | ||
| identity: { name: "my-sandbox", agent: "openclaw" }, | ||
| }); | ||
| }); | ||
|
|
||
| it("recreates under the SAME durable identity when the sandbox is gone, never a new name", () => { | ||
| const cp = checkpoint({ | ||
| effectGroups: { sandbox_create: { completedAt: ISO, fingerprint: "fp" } }, | ||
| }); | ||
| expect(planSandboxCreateReplay(cp, { liveSandboxExists: false })).toEqual({ | ||
| action: "create", | ||
| identity: { name: "my-sandbox", agent: "openclaw" }, | ||
| }); | ||
| }); | ||
|
|
||
| it("creates under the durable identity when create was never recorded", () => { | ||
| expect(planSandboxCreateReplay(checkpoint(), { liveSandboxExists: false })).toEqual({ | ||
| action: "create", | ||
| identity: { name: "my-sandbox", agent: "openclaw" }, | ||
| }); | ||
| }); | ||
|
|
||
| it("reuses a live sandbox even when the create receipt was lost to a mid-create crash (#7022)", () => { | ||
| expect(planSandboxCreateReplay(checkpoint(), { liveSandboxExists: true })).toEqual({ | ||
| action: "reuse", | ||
| identity: { name: "my-sandbox", agent: "openclaw" }, | ||
| }); | ||
| }); | ||
| }); | ||
|
|
||
| describe("crash-then-resume matrix proves at-most-once destructive create (#6228)", () => { | ||
| const states = [ | ||
| "sandbox", | ||
| "openclaw", | ||
| "agent_setup", | ||
| "policies", | ||
| "finalizing", | ||
| "post_verify", | ||
| ] as const; | ||
|
|
||
| it.each( | ||
| states, | ||
| )("crash at %s: reuse a surviving sandbox, recreate under the same identity when it is gone", (state) => { | ||
| const cp = checkpoint({ | ||
| machineState: state, | ||
| effectGroups: { sandbox_create: { completedAt: ISO, fingerprint: "fp" } }, | ||
| }); | ||
| expect(planSandboxCreateReplay(cp, { liveSandboxExists: true }).action).toBe("reuse"); | ||
| expect(planSandboxCreateReplay(cp, { liveSandboxExists: false })).toEqual({ | ||
| action: "create", | ||
| identity: { name: "my-sandbox", agent: "openclaw" }, | ||
| }); | ||
| }); | ||
| }); | ||
|
|
||
| describe("revalidateCheckpointBindings fails closed without leaking values (#6228)", () => { | ||
| it("passes when every binding is currently available", () => { | ||
| const cp = checkpoint({ | ||
| bindings: { | ||
| credentialEnvs: ["OPENAI_API_KEY"], | ||
| registeredProviders: [{ name: "p1", type: "generic", credentialEnv: "P1_API_KEY" }], | ||
| }, | ||
| }); | ||
| const result = revalidateCheckpointBindings(cp, { | ||
| availableCredentialEnvs: new Set(["OPENAI_API_KEY"]), | ||
| liveRegisteredProviders: new Set(["p1"]), | ||
| }); | ||
| expect(result).toEqual({ status: "ok" }); | ||
| expect(bindingRevalidationGuidance(result)).toBeNull(); | ||
| }); | ||
|
|
||
| it("fails closed on a stale binding and reports only names, never values", () => { | ||
| const cp = checkpoint({ | ||
| bindings: { | ||
| credentialEnvs: ["OPENAI_API_KEY"], | ||
| registeredProviders: [{ name: "p1", type: "generic", credentialEnv: "P1_API_KEY" }], | ||
| }, | ||
| }); | ||
| const result = revalidateCheckpointBindings(cp, { | ||
| availableCredentialEnvs: new Set(), | ||
| liveRegisteredProviders: new Set(), | ||
| }); | ||
| expect(result).toEqual({ | ||
| status: "stale", | ||
| missingCredentialEnvs: ["OPENAI_API_KEY"], | ||
| missingProviders: ["p1"], | ||
| }); | ||
| const guidance = bindingRevalidationGuidance(result); | ||
| expect(guidance).toContain("OPENAI_API_KEY"); | ||
| expect(guidance).toContain("p1"); | ||
| }); | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import { ONBOARD_MACHINE_STATES } from "./machine/types"; | ||
| import type { OnboardMachineState } from "./machine/types"; | ||
| import { isDecisionSelected } from "../state/onboard-checkpoint-decision"; | ||
| import type { | ||
| CheckpointEffectGroupName, | ||
| CheckpointSandboxIdentity, | ||
| OnboardCheckpoint, | ||
| } from "../state/onboard-checkpoint-types"; | ||
|
|
||
| export interface CheckpointedMachineSession { | ||
| readonly checkpoint: OnboardCheckpoint | null; | ||
| readonly machine: { readonly state: OnboardMachineState }; | ||
| } | ||
|
|
||
| export function checkpointProvesSandboxStepComplete( | ||
| session: CheckpointedMachineSession | null | undefined, | ||
| ): boolean { | ||
| if (!session?.checkpoint) return false; | ||
| const sandboxIndex = ONBOARD_MACHINE_STATES.indexOf("sandbox"); | ||
| const stateIndex = ONBOARD_MACHINE_STATES.indexOf(session.machine.state); | ||
| return stateIndex > sandboxIndex; | ||
| } | ||
|
|
||
| export type EffectGroupReplayReason = | ||
| | "not_recorded" | ||
| | "postcondition_failed" | ||
| | "already_complete_revalidated"; | ||
|
|
||
| export interface EffectGroupReplayDecision { | ||
| readonly group: CheckpointEffectGroupName; | ||
| readonly action: "skip" | "run"; | ||
| readonly reason: EffectGroupReplayReason; | ||
| } | ||
|
|
||
| export function planEffectGroupReplay( | ||
| checkpoint: OnboardCheckpoint, | ||
| group: CheckpointEffectGroupName, | ||
| postconditionHolds: boolean, | ||
| ): EffectGroupReplayDecision { | ||
| const record = checkpoint.effectGroups[group]; | ||
| if (!record) return { group, action: "run", reason: "not_recorded" }; | ||
| if (!postconditionHolds) return { group, action: "run", reason: "postcondition_failed" }; | ||
| return { group, action: "skip", reason: "already_complete_revalidated" }; | ||
| } | ||
|
|
||
| export interface SandboxCreateObservation { | ||
| readonly liveSandboxExists: boolean; | ||
| } | ||
|
|
||
| export type SandboxCreateReplayDecision = | ||
| | { readonly action: "reuse"; readonly identity: CheckpointSandboxIdentity } | ||
| | { readonly action: "create"; readonly identity: CheckpointSandboxIdentity } | ||
| | { readonly action: "capture_identity_first" }; | ||
|
|
||
| export function planSandboxCreateReplay( | ||
| checkpoint: OnboardCheckpoint, | ||
| observed: SandboxCreateObservation, | ||
| ): SandboxCreateReplayDecision { | ||
| if (!isDecisionSelected(checkpoint.sandboxIdentity)) { | ||
| return { action: "capture_identity_first" }; | ||
| } | ||
| const identity = checkpoint.sandboxIdentity.value; | ||
| if (observed.liveSandboxExists) { | ||
| return { action: "reuse", identity }; | ||
| } | ||
| return { action: "create", identity }; | ||
| } | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.