Skip to content

test(e2e): add cross-runtime execution foundation - #7988

Merged
jyaunches merged 15 commits into
feat/buildless-managed-contract-hardeningfrom
feat/buildless-runtime-e2e-foundation
Jul 31, 2026
Merged

test(e2e): add cross-runtime execution foundation#7988
jyaunches merged 15 commits into
feat/buildless-managed-contract-hardeningfrom
feat/buildless-runtime-e2e-foundation

Conversation

@ericksoa

@ericksoa ericksoa commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the inert, runtime-parameterized E2E contract needed to qualify buildless and future native runtimes without introducing Podman switches into central orchestration. The catalog compiles Docker and a fixture-only MXC-style provider through the same open provider identity, scenario, obligation, execution, and parity-evidence contracts. It keeps OpenClaw, Hermes, and DCode together and models amd64/arm64 plus CPU/GPU execution dimensions. Local-inference, recovery, installer, user-facing documentation, and protected-E2E qualification remain deferred; this slice does not activate or advertise runtime support.

Related Issue

Part of #7744

Changes

  • Add an open branded execution-provider identity and immutable execution-profile registry rather than a closed Docker/Podman union.
  • Add runtime-neutral scenarios, support obligations, bounded host lanes, exact preparation ownership, and normalized parity evidence.
  • Add a fixture-local provider adapter seam that owns environment, lifecycle, state observation, and exact scenario assertions.
  • Compile all-agent OpenClaw, Hermes, and DCode scenarios for Docker and a fixture-only MXC-style provider with exact adapter/provider/scenario ownership checks.
  • Validate the inspected workload identity before obligations, preserve execution and cleanup failures, require non-empty provider receipts, and allowlist persisted receipt and scenario fields.
  • Prove profile construction does not register either fixture provider through the canonical runtime-profile query and public live-matrix boundary.
  • Preserve the canonical live matrix, workflow plan, risk plan, and existing Docker fixture behavior.
  • Keep the slice inert: no production provider registration, Podman command, workflow target, support claim, or runtime activation.
  • Document the cross-runtime foundation and the deferred activation obligations.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: This inert slice changes no user-visible behavior. test/e2e/docs/README.md documents the contributor-facing E2E contract.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent exact-diff review found no P0/P1/P2. Provider identity stays open and runtime-neutral, every matrix case has exact provider/scenario/obligation ownership, preparation is bounded and atomic, inspected identity is validated before obligations, and observed lifecycle/state must conform exactly before parity evidence is accepted.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: test/e2e/docs/README.md accurately documents the inert cross-runtime foundation, provider-neutral contracts, public non-registration boundary, and deferred activation. No additional user-facing documentation is required.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: The 26-test focused hardening suite, 36-test changed-file suite, 12-test registry/foundation suite, typecheck, typecheck:cli, source-shape policy, test-size policy, git diff --check, and full npm run validate:pr passed on 4788d287b.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Exact-head required CI, advisors, CodeRabbit, multiarch, and protected E2E are running for 4788d287b8672be1b44999e78e094b2221303bd1.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Stack

  • Base: PR3.5b fix(runtime): close prior review debt #7976 branch feat/buildless-coderabbit-debt at 771f48c47c0aaa1d51511d172fffc691e6c9ac76.
  • This slice: PR3.5c branch feat/buildless-runtime-e2e-foundation at 4788d287b8672be1b44999e78e094b2221303bd1.
  • Next: PR3.6 adds the driver-neutral lifecycle bundle registry and sandbox-action parity; it is not part of this review diff.
  • Production buildless and Podman support remain disabled until the full all-agent, amd64/arm64, GPU, local Ollama/NIM/vLLM, recovery, installer, documentation, and protected-E2E gates pass.

Signed-off-by: Aaron Erickson aerickson@nvidia.com

Summary by CodeRabbit

Release Notes

  • Documentation

    • Added comprehensive documentation for cross-runtime end-to-end test foundation.
  • New Features

    • Introduced cross-runtime test execution planning with support for multiple execution profiles and platforms.
    • Added execution profile system with validation for platforms, architectures, and capabilities.
    • Implemented runtime matrix compilation and resolution for deterministic test case distribution.
    • Added test fixture infrastructure for provider-neutral scenario definitions.
  • Tests

    • Added cross-runtime compatibility and parity verification tests.
    • Added comprehensive runtime matrix validation test suite.

ericksoa added 7 commits July 30, 2026 23:11
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
(cherry picked from commit f99197b)
@ericksoa ericksoa added area: e2e End-to-end tests, nightly failures, or validation infrastructure platform: container Affects Docker, containerd, Podman, or images labels Jul 31, 2026
@ericksoa

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@github-code-quality

github-code-quality Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 4788d28 in the feat/buildless-runti... branch remains at 96%, unchanged from commit 387cb08 in the feat/buildless-coder... branch.

TypeScript / code-coverage/cli

The overall coverage in commit 4788d28 in the feat/buildless-runti... branch remains at 81%, unchanged from commit 387cb08 in the feat/buildless-coder... branch.

Show a code coverage summary of the most impacted files.
File feat/buildless-coder... 387cb08 feat/buildless-runti... 4788d28 +/-
src/lib/messagi.../plan-filter.ts 100% 0% -100%
src/lib/onboard...mage-runtime.ts 93% 50% -43%
src/lib/onboard.../application.ts 86% 84% -2%
src/lib/onboard...file-builder.ts 90% 90% 0%
src/lib/onboard...-environment.ts 92% 96% +4%
src/lib/onboard...-transaction.ts 0% 78% +78%
src/lib/onboard...p/root-apply.ts 0% 84% +84%
src/lib/messagi...ll-selection.ts 0% 84% +84%
src/lib/onboard...shared-state.ts 0% 87% +87%
src/lib/onboard...r-root-apply.ts 0% 94% +94%

Updated July 31, 2026 08:57 UTC

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The PR adds a cross-runtime E2E foundation. It defines validated execution profiles, compiles runtime matrices into deterministic shards, provides provider-neutral fixtures, integrates resolved cases into live plans, and adds parity and evidence validation tests.

Changes

Cross-runtime E2E foundation

Layer / File(s) Summary
Execution profile contracts
test/e2e/registry/execution-profile.ts
Adds typed execution profiles, capability sets, runner definitions, branded provider IDs, normalization, and validation.
Runtime matrix compilation
test/e2e/registry/runtime-matrix.ts
Adds binding and adapter validation, deterministic resource identities, preparation keys, host sharding, matrix compilation, and runtime-case resolution.
Provider-neutral fixtures and execution
test/e2e/support/cross-runtime-foundation-fixtures.ts, test/e2e/support/e2e-runtime-matrix.test.ts
Defines runtime-neutral scenarios, Docker and test-MXC profiles, adapter bindings, fake provider behavior, scheduling checks, and provider-neutral execution tests.
Live plan and parity evidence
test/e2e/live/run-plan.ts, test/e2e/support/e2e-parity-evidence.test.ts, test/e2e/support/e2e-cross-runtime-compatibility.test.ts, test/e2e/docs/README.md
Adds optional runtime-case resolution to live plans and documents and tests deterministic outputs, parity evidence, validation failures, secret redaction, and integration behavior.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related issues

Suggested labels: feature, area: architecture

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 4.17% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the added cross-runtime E2E execution foundation.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/buildless-runtime-e2e-foundation

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 1 warning · 1 suggestion
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, the same number of warnings, 1 more suggestion.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, security-posture, channels-add-remove, channels-stop-start, onboard-repair, onboard-resume

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Remove the unconsumed cross-runtime foundation

  • Location: test/e2e/registry/runtime-matrix.ts:1
  • Category: architecture
  • Problem: The patch adds a registry, adapter catalog, host-shard scheduler, provider fixture seam, and parity-evidence model, but no canonical target, workflow selector, live scenario, or production registration consumes them.
  • Impact: The repository gains a large unowned execution-model abstraction that can drift from the live E2E path and increases future security-sensitive maintenance surface.
  • Recommendation: Remove the inert cross-runtime foundation from this PR. Add a focused layer only with a named live consumer and its contract test.
  • Verification: Read test/e2e/docs/README.md and search canonical target definitions and workflow selection code for runtimeCase or a runtime-matrix consumer.
  • Test coverage: The existing support tests prove only the inert foundation. A future consumer must add a contract test that a named canonical target dispatches the registered runtime case through the trusted execution and evidence path.
  • Simplification (delete): Remove Delete the unconsumed cross-runtime registry, provider fixture seam, parity-evidence model, and support-only tests.; use Keep the existing canonical Docker E2E registry and fixtures until a named live consumer requires a focused extension.. Net: 3000 lines.
  • Keep: Do not remove existing E2E evidence redaction, trusted target selection, or lifecycle coverage.
  • Evidence: test/e2e/registry/runtime-matrix.ts:1 introduces the matrix compiler, adapter catalog, shard scheduler, and runtime-case resolver. test/e2e/fixtures/e2e-test.ts:61 adds executionProfile and runtimeProvider fixtures that always resolve to undefined. test/e2e/docs/README.md:75 states that no canonical target, workflow selector, live scenario, or production runtime registration consumes the metadata yet.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🧹 Nitpick comments (2)
test/e2e/registry/runtime-matrix.ts (1)

146-218: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider extracting the per-binding validation loop.

compileObligationBindings combines many checks in one function: pattern validation, catalog lookup, provider/scenario/obligation identity matching, duplicate detection, and, after the loop, missing/unknown/capability checks. This raises the branching complexity of a single function.

Extract the per-binding checks (lines 153-187) into a small helper, for example validateObligationBinding(scenario, profile, binding, adapterCatalog, declared), called once per binding inside the loop. This keeps each function focused on one responsibility and reduces cyclomatic complexity in the main function.

As per coding guidelines, **/*.{js,ts,tsx} should "keep function complexity low."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/registry/runtime-matrix.ts` around lines 146 - 218, The per-binding
validation in compileObligationBindings is making the function overly complex.
Extract the adapter ID, catalog, provider, scenario, obligation identity, and
duplicate checks into a helper such as validateObligationBinding(scenario,
profile, binding, adapterCatalog, declared), then call it for each binding while
leaving the missing, unknown, capability, and compilation-order logic in
compileObligationBindings.

Source: Coding guidelines

test/e2e/fixtures/runtime-provider.ts (1)

98-132: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Add bounded cancellation before enabling a real runtime provider.

runtimeProvider is currently undefined, and only the immediate fake provider exists. The contract has no deadline or AbortSignal, so a future provider can leave prepare(), adapter execution, state inspection, observation, or cleanup pending indefinitely.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/fixtures/runtime-provider.ts` around lines 98 - 132, Add a bounded
deadline and AbortSignal to the runtime provider contract used by the fixture,
and propagate it through prepare, inspectWorkload, adapter.execute, observe, and
cleanup. Update the flow around runtimeProvider and the shown lifecycle
try/finally block so every provider operation is cancelled when the deadline
expires, including cleanup, rather than allowing a real provider to remain
pending indefinitely.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/fixtures/runtime-provider.ts`:
- Around line 118-132: Update the try/finally flow around the obligation
execution and provider.state.observe calls to preserve the primary rejection
when provider.lifecycle.cleanup also fails. Capture the execution error, run
cleanup, and report both failures while ensuring the original execution error
remains the primary cause; retain cleanupReceipts assignment on successful
cleanup.
- Around line 113-125: Validate workload.logicalId against
runtimeCase.identities.sandbox immediately inside the existing try/finally,
before iterating through obligationBindings or invoking any adapter. Reject or
throw on a mismatch, and preserve cleanup using the inspected workload identity.

In `@test/e2e/registry/parity-evidence.ts`:
- Around line 173-189: Update the receipt normalization callback in the receipts
map to explicitly require receipt.kind and receipt.operationId to be strings
before applying RECEIPT_ID_PATTERN, while preserving duplicate operationId
detection. Replace the raw receipt spread in the frozen result with an
allowlisted object containing only kind, operationId, and the normalized value,
so arbitrary provider fields cannot reach persisted evidence.
- Around line 165-170: Align the provider-receipt cardinality contract across
the provider interfaces and buildExecutionEvidence flow. If at least one receipt
is required, enforce that constraint in both interfaces and add contract
coverage; otherwise allow empty arrays and remove the rejection in
normalizeProviderReceipts and its corresponding test.

In `@test/e2e/registry/scenario.ts`:
- Around line 235-237: Update defineRuntimeScenario to allowlist fields when
normalizing the journey steps, support obligations, and top-level scenario
input. Replace raw-object spreads with explicit objects containing only each
structure’s declared fields, including the existing normalized action/foundation
values. Ensure unknown properties such as provider are omitted from the returned
registry scenario.

In `@test/e2e/support/e2e-runtime-foundation-types.test.ts`:
- Around line 17-35: Update the test around foundationProfiles() to verify the
public registry remains unchanged after creating the Docker and test-MXC
profiles, proving neither provider is registered; use the existing public
registry inspection API and assert both providers are absent. If that boundary
cannot be inspected, rename the test to describe only profile shape and
capability validation.

---

Nitpick comments:
In `@test/e2e/fixtures/runtime-provider.ts`:
- Around line 98-132: Add a bounded deadline and AbortSignal to the runtime
provider contract used by the fixture, and propagate it through prepare,
inspectWorkload, adapter.execute, observe, and cleanup. Update the flow around
runtimeProvider and the shown lifecycle try/finally block so every provider
operation is cancelled when the deadline expires, including cleanup, rather than
allowing a real provider to remain pending indefinitely.

In `@test/e2e/registry/runtime-matrix.ts`:
- Around line 146-218: The per-binding validation in compileObligationBindings
is making the function overly complex. Extract the adapter ID, catalog,
provider, scenario, obligation identity, and duplicate checks into a helper such
as validateObligationBinding(scenario, profile, binding, adapterCatalog,
declared), then call it for each binding while leaving the missing, unknown,
capability, and compilation-order logic in compileObligationBindings.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3cfc6fa4-9ac1-41a5-81b4-c4b4d958347c

📥 Commits

Reviewing files that changed from the base of the PR and between 7cf2a8d and abf45b3.

📒 Files selected for processing (16)
  • test/e2e/docs/README.md
  • test/e2e/fixtures/artifacts.ts
  • test/e2e/fixtures/e2e-test.ts
  • test/e2e/fixtures/runtime-provider.ts
  • test/e2e/live/run-plan.ts
  • test/e2e/registry/builder.ts
  • test/e2e/registry/execution-profile.ts
  • test/e2e/registry/parity-evidence.ts
  • test/e2e/registry/runtime-matrix.ts
  • test/e2e/registry/scenario.ts
  • test/e2e/registry/types.ts
  • test/e2e/support/cross-runtime-foundation-fixtures.ts
  • test/e2e/support/e2e-cross-runtime-compatibility.test.ts
  • test/e2e/support/e2e-parity-evidence.test.ts
  • test/e2e/support/e2e-runtime-foundation-types.test.ts
  • test/e2e/support/e2e-runtime-matrix.test.ts

Comment thread test/e2e/fixtures/runtime-provider.ts
Comment thread test/e2e/fixtures/runtime-provider.ts
Comment thread test/e2e/registry/parity-evidence.ts Outdated
Comment thread test/e2e/registry/parity-evidence.ts Outdated
Comment thread test/e2e/registry/scenario.ts Outdated
Comment thread test/e2e/support/e2e-runtime-foundation-types.test.ts
ericksoa added 2 commits July 31, 2026 01:06
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
ericksoa added 5 commits July 31, 2026 01:31
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa

Copy link
Copy Markdown
Contributor Author

Advisor PRA-1 disposition for exact head 4788d28: the inert foundation remains intentional under #7744. Adding a canonical live consumer in this slice would cross the activation boundary before all-agent qualification. The final contract now proves that Docker and fixture MXC profiles are not registered and that the public live matrix is unchanged. Later slices own live buildless and native-runtime consumers only after their all-agent, multiarch, GPU, local-inference, recovery, installer, documentation, and protected-E2E gates pass. Full validate:pr and the focused registry/foundation suites pass on this head.

Base automatically changed from feat/buildless-coderabbit-debt to feat/buildless-managed-contract-hardening July 31, 2026 15:00
@jyaunches
jyaunches merged commit 24290d1 into feat/buildless-managed-contract-hardening Jul 31, 2026
96 of 100 checks passed
@jyaunches
jyaunches deleted the feat/buildless-runtime-e2e-foundation branch July 31, 2026 15:00
senthilr-nv pushed a commit that referenced this pull request Jul 31, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Introduces the driver-neutral runtime-provider lifecycle and mutation
contract used by the incremental buildless/runtime stack, and closes the
destructive-cleanup authority boundary identified during exact-head
review. Destroy, rebuild, and snapshot force-restore must now prove
provider and workload cleanup authority through a side-effect-free
provider plan before deleting or stopping anything.

Production selection remains limited to the existing Docker and
Kubernetes providers. This slice does not activate another runtime or
expand supported lifecycle platforms.

## Related Issue

Part of #7744

## Changes

- Add one versioned, immutable provider bundle registry covering plan,
capability, preflight, gateway, workload, lifecycle, mutation-authority,
bootstrap, snapshot, recovery, cleanup, and container-engine surfaces.
- Route sandbox registration, start, provider-owned post-start
verification, stop, inference-set authority, live destroy
preparation/deletion, and owned-workload cleanup through the selected
bundle.
- Add `planOwnedWorkloadCleanup` to the cleanup contract and require
every supported provider to prove cleanup intent without side effects
before a destructive action.
- Apply one complete authority check before destructive side effects in
normal destroy, rebuild, and snapshot force-restore.
- Recheck rebuild authority at the exact delete edge after MCP
preparation; on failure, restore MCP attachment, relock shields, retain
ownership state, and skip deletion.
- Keep actual cleanup independently fail closed, preserving a residual
post-delete guard for raw-writer or TOCTOU changes outside NemoClaw's
lifecycle lock.
- Preserve shared managed images and rows with no owned image without
turning a missing or legacy receipt into a deletion candidate.
- Replace nonexistent "repair the receipt" guidance with the real
`nemoclaw <sandbox> doctor --json` diagnostic path. Operators must
restore trusted ownership metadata or resolve the runtime conflict and
must not rewrite a receipt to match a mutable sandbox name.
- Make the Kubernetes compatibility boundary explicit: the shipped
Kubernetes gateway path's legacy per-sandbox image remains owned by the
host Docker engine until a separately registered CRI-native provider
exists.
- Preserve the existing Kubernetes lifecycle gate. Its bundle remains
`lifecycle.supported: false`, and this PR does not claim Kubernetes
lifecycle activation.
- Add a socket-free MXC-style contract provider and exercise OpenClaw,
Hermes, and LangChain Deep Agents Code without Podman- or MXC-specific
central switches.
- Fail closed for unknown provider identities, unsupported mutation
surfaces, malformed ownership receipts, and unresolved cleanup
authority.

Direct connect, status, logs, authenticated reconciliation, and durable
crash recovery remain owned by later slices. No future provider is
production-selectable or advertised by this PR.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: The provider contract remains
inert for future providers, production selection is unchanged, and no
supported CLI/runtime behavior is advertised or activated.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Exact-head review
covers registry identity binding, provider-owned lifecycle verification,
mutation authority, all-agent MXC-style action proof, pre-delete and
exact-edge cleanup authority, rollback, and source architecture.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: The exact 45-file diff (`+4,073/-508`) tightens an inert
internal provider and destructive-authority contract and replaces
misleading failure text with an existing diagnostic command. It does not
activate or advertise a new provider, platform, or runtime.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 75730cf -->
<!-- docs-review-agents-blob-sha: c669f7c -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- Exact locally validated head/base:
`75730cf09bf1a1aa901cc3b275052250f8e7d85d` /
`4788d287b8672be1b44999e78e094b2221303bd1`
- Review budget: 45 files, `+4,073/-508`; five files above the soft file
guide to apply and prove one complete cross-cutting
destructive-authority boundary, while remaining within the 2–5k line
guide.
- Stable exact-slice patch ID:
`eef1fddf8138d6e8a3ef4efb443aa2adc9f74fe3`.
- [x] PR description includes a `Signed-off-by:` line and every new
commit contains an SSH signature and DCO trailer.
- [x] `npm run validate:pr` passed on the exact clean head.
- [x] 320 focused changed-surface provider, lifecycle, snapshot, and
image-cleanup tests passed on the exact head; CLI typecheck, repository
architecture, source-shape and test-size budgets, Biome, secret
scanning, and `git diff --check` passed.
- [x] Unknown-provider and mismatched-receipt tests prove snapshot
force-restore performs no NIM stop, OpenShell delete, provider deletion,
shields cleanup, replacement creation, or registry registration.
- [x] Rebuild tests prove authority is checked before MCP preparation
and again at the exact delete edge; a changed receipt restores MCP
attachment, relocks shields, and performs no sandbox delete.
- [x] Cleanup-contract tests prove planning is side-effect free,
registration rejects providers without it, and actual cleanup
revalidates authority before mutation.
- [x] Error-path tests prove ownership state is retained, the command
exits nonzero, `doctor --json` is named, unsafe receipt rewriting is
rejected, and no false success is emitted.
- [x] Kubernetes cleanup and lifecycle findings are dispositioned
against the shipped compatibility contract: legacy images remain
host-Docker-owned, while Kubernetes lifecycle remains explicitly
unsupported and fail closed.
- [x] CodeRabbit's inline findings, including inert planner-state and
boolean-removal fixtures, are resolved; exact-head incremental review is
active.
- [ ] Applicable broad gate passed — exact-head required CI, advisors,
CodeRabbit, multiarch, and protected E2E are the broad remote gates.

## Stack

- Base: PR3.5c #7988 branch `feat/buildless-runtime-e2e-foundation` at
`4788d287b8672be1b44999e78e094b2221303bd1`.
- This slice: PR3.6 branch `feat/runtime-provider-lifecycle-parity` at
`75730cf09bf1a1aa901cc3b275052250f8e7d85d`.
- Later slices own snapshot/clone/rebuild/restore parity, transactional
bootstrap, authenticated reconciliation, durable recovery, and final
all-agent multiarch activation.
- Buildless support remains disabled until every supported agent and
required qualification gate passes.

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added provider-neutral runtime support for Docker and Kubernetes
environments.
* Sandbox start, stop, recovery, diagnostics, inference updates,
snapshots, and rebuilds now use the selected runtime provider.
* Added workload ownership records to improve managed-image tracking and
portability.
* **Bug Fixes**
* Destructive cleanup now fails safely when ownership or runtime
authority cannot be verified.
* Added recovery guidance while preserving registry and session state
after blocked cleanup.
* Improved diagnostics for unsupported or unregistered runtime
providers.
* **Tests**
* Expanded coverage for lifecycle operations, cleanup safeguards,
workload validation, and provider portability.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
ericksoa added a commit that referenced this pull request Aug 1, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the inert, provider-neutral managed-workload rebuild transaction
for the incremental buildless stack. The exact old workload and registry
row remain authoritative through replacement preparation, readiness,
state restore, and provider rebind. Only one exact compare-and-swap
publishes the replacement, and old-runtime retirement happens afterward
through provider-owned opaque handles.

This slice does not wire a production rebuild caller or activate
buildless onboarding. Snapshot/backup and durable recovery ownership
remain tracked in #7744 and are required before activation.

## Related Issue

Part of #7744

## Changes

- Capture a deep-frozen rebuild plan bound to the exact provider,
shipped agent, platform, prior managed receipt, full durable-row
revision, lifecycle generation, and live identity fingerprint.
- Pre-render and validate the exact replacement image, startup profile,
receipt, and safe metadata before provider mutation.
- Define provider-neutral prepare, create, readiness, restore,
provider-rebind, rollback, abort-preparation, and retire-previous phases
using opaque exact handles rather than sandbox-name deletion.
- Keep partial prepare/create cleanup transaction-idempotent and run
abort cleanup even when post-prepare registry revalidation throws.
- Publish only through exact old-authority CAS; reconcile ambiguous
persistence against either the exact replacement or exact old row.
- Preserve the staged replacement and return an immutable recovery task
when publication is indeterminate, avoiding rollback of a replacement
that may already be durable.
- Retire the exact old runtime only after publication; return a
durable-owner recovery task if retirement remains pending.
- Bind replacement contracts and startup profiles to OpenClaw, Hermes,
or DCode authority and reject provider, agent, platform, receipt,
generation, or identity drift.
- Reject malformed provider artifacts at every transition, stop before
later phases, and prove exact transaction abort or exact staged-handle
rollback.
- Document the shared backup boundary and the durable recovery ownership
tracked in #7744 before activation.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: The transaction is inert with
no production caller or support claim; the internal README records
ownership boundaries for later slices.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Exact-head audit covers
immutable authority, pre-mutation validation, CAS ambiguity, abort
cleanup, exact-handle rollback, and deferred recovery ownership.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: The reviewed 23-file, `+4,272/-0` slice remains
byte-identical after the append-only current-main refresh to
`0de2789608a86e580d787991e81c03c5f0b14dbf` through
`e97ecce48c7fcc1dfb398e1cfae8c81a859b7dcd`; stable patch ID remains
`dd1c4a899fd9a62954a00d4e2e61da445a306e03`. The only documentation path
is `src/lib/onboard/managed-workload/rebuild/README.md`. It accurately
states that the transaction is dormant, has no CLI command or
production-action importer, and does not activate buildless rebuilds. It
assigns ambiguous publication and pending retirement to durable
recovery, links recovery and snapshot/backup ownership to the live
accepted epic #7744, and requires normalized backup manifests, restore
validation, durable reconciliation, and protected qualification for
OpenClaw, Hermes, and LangChain Deep Agents Code before activation.
Production-import and command/action diff scans found no activation
caller. Markdownlint passed with zero issues on the exact refreshed
head. The append-only parent refresh to `362a70cda` preserves the exact
reviewed slice diff and changes no reviewed documentation.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 362a70c -->
<!-- docs-review-agents-blob-sha: 3dd7c24 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- Exact locally validated head/base:
`e97ecce48c7fcc1dfb398e1cfae8c81a859b7dcd` /
`0de2789608a86e580d787991e81c03c5f0b14dbf`
- Review budget: 23 files, `+4,272/-0`.
- Stable exact-slice patch ID:
`dd1c4a899fd9a62954a00d4e2e61da445a306e03`.
- [x] The six implementation/review commits and both maintainer refresh
commits are SSH-signed and contain DCO trailers; GitHub-generated
conflict-resolution merge commits preserve append-only branch history.
- [x] `npm run validate:pr` passed on the exact clean head with Node
22.16.0.
- [x] 132 focused rebuild transaction, workload authority, registry CAS,
and source-boundary tests passed again on the exact refreshed head; CLI
typecheck and repository checks also passed; changed test files add zero
`if` statements.
- [x] `npm run build:cli`, CLI typecheck through `validate:pr`,
exact-base pre-commit, commitlint, and pre-push gates passed.
- [x] Failure tests prove prepare/create ambiguity aborts exact
transaction resources, staged failures roll back only exact staging
authority, and indeterminate CAS never rolls back.
- [x] Agent-binding tests reject cross-agent image/profile drift for all
shipped managed-image agents.
- [x] No snapshot manifest dependency, production rebuild callsite,
runtime selection change, or public activation exists in this slice.
- [ ] Applicable broad gate passed — exact-head required CI, advisors,
CodeRabbit, multiarch, and protected E2E are the broad remote gates.

## Stack

- Base: live `main` at `0de2789608a86e580d787991e81c03c5f0b14dbf`; PR3.1
through PR3.6 content is already landed, with #7976, #7988, and #7990
carried once through the final #7973 aggregate tree.
- This slice: PR3.7 branch `feat/managed-workload-rebuild-parity` at
`e97ecce48c7fcc1dfb398e1cfae8c81a859b7dcd`.
- Epic #7744 tracks shared snapshot, backup, restore, and durable
recovery ownership before activation.
- Buildless support remains disabled until OpenClaw, Hermes, and DCode
plus required multiarch and protected qualification pass together.

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added managed workload rebuild workflows with staged replacement,
validation, rollback, recovery, and atomic commit handling.
* Added authority validation for managed workloads, including receipt,
image, platform, and startup configuration checks.
  * Added safe cloning and deep-freezing for supported immutable data.
* Added safeguards against stale, conflicting, or incomplete workload
state during rebuilds.

* **Documentation**
  * Documented rebuild recovery behavior and activation requirements.

* **Tests**
* Added comprehensive coverage for rebuild transactions, authority
validation, rollback, persistence reconciliation, and immutable data
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure platform: container Affects Docker, containerd, Podman, or images

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants