-
Notifications
You must be signed in to change notification settings - Fork 3.1k
test(e2e): define protected inference qualification #8079
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
ericksoa
wants to merge
14
commits into
feat/host-local-inference-lifecycle-recovery
from
feat/host-local-inference-protected-qualification
Closed
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
6d4389d
test(e2e): define protected inference qualification
ericksoa da403b0
test(e2e): bind qualification to workflow revision
ericksoa 350f075
test(e2e): authenticate qualification evidence
ericksoa adf2791
merge(stack): restack protected qualification on lifecycle fixes
ericksoa b1b9110
test(e2e): connect trusted qualification controller
ericksoa a3812f5
fix(e2e): stabilize qualification artifact trust
ericksoa a6a04de
Merge PR4.5f receipt-preservation cascade into PR4.5g
ericksoa b3eeed4
test(e2e): keep qualification race harness linear
ericksoa 0e3cf12
Merge PR4.5f snapshot authority coverage into PR4.5g
ericksoa 5b6c95b
Merge PR4.5f recovery fixes into PR4.5g
ericksoa 2984428
refactor: defer native qualification controller wrapper
ericksoa c8436e2
test: cover single-job qualification evidence
ericksoa 8514315
refactor: defer qualification evidence reporter
ericksoa 326edb5
test: reject malformed qualification evidence kinds
ericksoa File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> | ||
| <!-- SPDX-License-Identifier: Apache-2.0 --> | ||
|
|
||
| # Native Runtime Activation Qualification | ||
|
|
||
| `registry/activation-qualification.ts` is a dormant release gate for a native | ||
| container-runtime candidate. It does not register a runtime, add a live target, | ||
| alter the production runtime registry, or advertise support. | ||
|
|
||
| The compiler has one provider-neutral contract. A candidate supplies an open | ||
| provider ID and fixture bindings; the compiler does not branch on Podman, MXC, | ||
| or any other engine name. The inert Podman candidate in | ||
| `support/native-runtime-qualification-fixtures.ts` proves the intended scope, | ||
| while the fake MXC-style test proves that the same contract accepts another | ||
| provider without a central switch. | ||
|
|
||
| ## Required protected matrix | ||
|
|
||
| Compilation requires 24 exact cases: | ||
|
|
||
| - OpenClaw, Hermes, and DCode; | ||
| - Linux `amd64` and `arm64`; | ||
| - rootless CPU with host-local Ollama; | ||
| - rootless NVIDIA GPU with CDI-backed Ollama, NIM, and vLLM; | ||
| - the release installer with Docker unavailable; and | ||
| - protected E2E for every case. | ||
|
|
||
| Every case must declare installation, Docker-unavailable proof, onboarding, an | ||
| agent turn, stop/start, snapshot/restore, rebuild, restart/reconciliation, and | ||
| exact cleanup. Removing one case or obligation is a compile error, not a skip. | ||
|
|
||
| ## Exact evidence | ||
|
|
||
| Every compiled case declares the evidence categories that the protected | ||
| collector must eventually produce. Those categories preserve the complete | ||
| activation target: | ||
|
|
||
| - the exact protected workflow revision, run, job, attempt, head SHA, and base | ||
| SHA; | ||
| - hashed installer script and invocation artifacts with a successful result; | ||
| - an exact provider/profile/architecture/acceleration identity and persisted | ||
| host-local engine authority; | ||
| - immutable agent and probe image references, plus an immutable inference image | ||
| reference for provider-managed NIM and vLLM; | ||
| - the exact provider-native host, port, network, gateway provider URL, and the | ||
| canonical `https://inference.local/v1` application route; | ||
| - the serialized host-local inference authority digest, including the exact | ||
| provider-owned runtime/container identity and specification digest for NIM | ||
| and vLLM; | ||
| - the exercised model ID and a hashed inference-result artifact; | ||
| - hashed artifacts for every lifecycle obligation, all bound to that same | ||
| durable authority; | ||
| - a reconciliation receipt proving recovery retained the same authority; | ||
| - an NVIDIA CDI `nvidia.com/gpu=all` receipt for GPU cases; and | ||
| - exact cleanup proving external Ollama was retained or provider-owned NIM and | ||
| vLLM were removed, with no provider-owned runtime IDs remaining. | ||
|
|
||
| These are compile-time requirements, not a receipt schema or generated | ||
| evidence. This slice intentionally does not export a reporter, artifact | ||
| verifier, or process-local evidence brand before a protected workflow consumes | ||
| that API. The protected-collector slice must add those pieces together: obtain | ||
| authenticated GitHub run/job state independently of worker receipts, verify | ||
| every artifact below its downloaded job root, and fail closed on incomplete, | ||
| inexact, linked, escaping, conflicting, changing, or oversized evidence. All | ||
| cases must bind to one exact protected workflow/head/base source before the | ||
| runtime can activate. | ||
|
|
||
| ## Activation boundary | ||
|
|
||
| Keep this contract inert until an implementation PR supplies executable | ||
| provider adapters and the entire protected matrix passes on one exact head/base | ||
| pair. Public support, installer selection, production registry wiring, and | ||
| workflow dispatch are separate activation work and must not infer support from | ||
| the existence of this contract. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,292 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import { | ||
| defineExecutionProfile, | ||
| type ExecutionAcceleration, | ||
| type ExecutionArchitecture, | ||
| type ExecutionCapability, | ||
| type ExecutionProfile, | ||
| type ExecutionProviderId, | ||
| } from "./execution-profile.ts"; | ||
| import { compareCodeUnits, type RuntimeAgent } from "./scenario.ts"; | ||
|
|
||
| export const NATIVE_RUNTIME_QUALIFICATION_AGENTS = ["openclaw", "hermes", "dcode"] as const; | ||
| export const NATIVE_RUNTIME_QUALIFICATION_ARCHITECTURES = ["amd64", "arm64"] as const; | ||
| export const NATIVE_RUNTIME_QUALIFICATION_ACCELERATIONS = ["cpu", "nvidia-gpu"] as const; | ||
| export const NATIVE_RUNTIME_QUALIFICATION_INFERENCE = { | ||
| cpu: ["ollama"], | ||
| "nvidia-gpu": ["ollama", "nim", "vllm"], | ||
| } as const satisfies Readonly<Record<ExecutionAcceleration, readonly LocalInferenceProvider[]>>; | ||
|
|
||
| export type LocalInferenceProvider = "ollama" | "nim" | "vllm"; | ||
| export type QualificationObligation = | ||
| | "installer.install" | ||
| | "runtime.docker-unavailable" | ||
| | "agent.onboard" | ||
| | "agent.turn" | ||
| | "sandbox.stop-start" | ||
| | "sandbox.snapshot-restore" | ||
| | "sandbox.rebuild" | ||
| | "runtime.restart-reconcile" | ||
| | "cleanup.exact"; | ||
| export type QualificationEvidenceKind = | ||
| | "protected-run" | ||
| | "source-identity" | ||
| | "installer-result" | ||
| | "docker-unavailable-guard" | ||
| | "managed-images" | ||
| | "agent-turn" | ||
| | "local-inference" | ||
| | "lifecycle" | ||
| | "recovery" | ||
| | "cleanup" | ||
| | "nvidia-cdi"; | ||
|
|
||
| export const NATIVE_RUNTIME_QUALIFICATION_OBLIGATIONS = [ | ||
| "installer.install", | ||
| "runtime.docker-unavailable", | ||
| "agent.onboard", | ||
| "agent.turn", | ||
| "sandbox.stop-start", | ||
| "sandbox.snapshot-restore", | ||
| "sandbox.rebuild", | ||
| "runtime.restart-reconcile", | ||
| "cleanup.exact", | ||
| ] as const satisfies readonly QualificationObligation[]; | ||
|
|
||
| const BASE_EVIDENCE_KINDS = [ | ||
| "protected-run", | ||
| "source-identity", | ||
| "installer-result", | ||
| "docker-unavailable-guard", | ||
| "managed-images", | ||
| "agent-turn", | ||
| "local-inference", | ||
| "lifecycle", | ||
| "recovery", | ||
| "cleanup", | ||
| ] as const satisfies readonly QualificationEvidenceKind[]; | ||
|
|
||
| const REQUIRED_CAPABILITIES = [ | ||
| "agent.configure", | ||
| "agent.turn", | ||
| "evidence.collect", | ||
| "sandbox.lifecycle", | ||
| "state.observe", | ||
| "transport.socket-free", | ||
| ] as const satisfies readonly ExecutionCapability[]; | ||
|
|
||
| export interface NativeRuntimeQualificationCase { | ||
| id: string; | ||
| agent: RuntimeAgent; | ||
| profile: ExecutionProfile; | ||
| inference: LocalInferenceProvider; | ||
| gate: "protected-e2e"; | ||
| install: "release-installer"; | ||
| dockerAvailability: "unavailable"; | ||
| obligations: readonly QualificationObligation[]; | ||
| evidenceKinds: readonly QualificationEvidenceKind[]; | ||
| } | ||
|
|
||
| export interface NativeRuntimeQualificationDefinition { | ||
| id: string; | ||
| repository: string; | ||
| protectedWorkflow: string; | ||
| provider: ExecutionProviderId; | ||
| engineName: string; | ||
| cases: readonly NativeRuntimeQualificationCase[]; | ||
| } | ||
|
|
||
| export interface CompiledNativeRuntimeQualification { | ||
| id: string; | ||
| repository: string; | ||
| protectedWorkflow: string; | ||
| provider: ExecutionProviderId; | ||
| engineName: string; | ||
| cases: readonly Readonly<NativeRuntimeQualificationCase>[]; | ||
| } | ||
|
|
||
| const REPOSITORY_PATTERN = /^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/u; | ||
|
|
||
| function assertSingleLine(value: string, label: string): string { | ||
| const normalized = value.trim(); | ||
| if (!normalized || /[\r\n]/u.test(normalized)) { | ||
| throw new Error(`${label} must be a non-empty single-line string`); | ||
| } | ||
| return normalized; | ||
| } | ||
|
|
||
| function assertExactSet<T extends string>( | ||
| actual: readonly T[], | ||
| expected: readonly T[], | ||
| label: string, | ||
| ): void { | ||
| const actualSet = new Set(actual); | ||
| if (actualSet.size !== actual.length) { | ||
| throw new Error(`${label} contains duplicate values`); | ||
| } | ||
| const missing = expected.filter((value) => !actualSet.has(value)); | ||
| const unknown = actual.filter((value) => !expected.includes(value)); | ||
| if (missing.length > 0 || unknown.length > 0) { | ||
| throw new Error( | ||
| `${label} is incomplete (missing: ${missing.join(", ") || "none"}; unknown: ${unknown.join(", ") || "none"})`, | ||
| ); | ||
| } | ||
| } | ||
|
|
||
| export function requiredQualificationEvidenceKinds( | ||
| acceleration: ExecutionAcceleration, | ||
| ): readonly QualificationEvidenceKind[] { | ||
| return acceleration === "nvidia-gpu" | ||
| ? Object.freeze([...BASE_EVIDENCE_KINDS, "nvidia-cdi"]) | ||
| : BASE_EVIDENCE_KINDS; | ||
| } | ||
|
|
||
| export function qualificationCaseId(input: { | ||
| provider: ExecutionProviderId; | ||
| agent: RuntimeAgent; | ||
| architecture: ExecutionArchitecture; | ||
| acceleration: ExecutionAcceleration; | ||
| inference: LocalInferenceProvider; | ||
| }): string { | ||
| const acceleration = input.acceleration === "nvidia-gpu" ? "gpu" : input.acceleration; | ||
| return [ | ||
| input.provider, | ||
| input.agent, | ||
| "linux", | ||
| input.architecture, | ||
| acceleration, | ||
| input.inference, | ||
| ].join("-"); | ||
| } | ||
|
|
||
| function coverageKey(input: { | ||
| agent: RuntimeAgent; | ||
| architecture: ExecutionArchitecture; | ||
| acceleration: ExecutionAcceleration; | ||
| inference: LocalInferenceProvider; | ||
| }): string { | ||
| return [input.agent, input.architecture, input.acceleration, input.inference].join("|"); | ||
| } | ||
|
|
||
| function requiredCoverageKeys(): string[] { | ||
| return NATIVE_RUNTIME_QUALIFICATION_AGENTS.flatMap((agent) => | ||
| NATIVE_RUNTIME_QUALIFICATION_ARCHITECTURES.flatMap((architecture) => | ||
| NATIVE_RUNTIME_QUALIFICATION_ACCELERATIONS.flatMap((acceleration) => | ||
| NATIVE_RUNTIME_QUALIFICATION_INFERENCE[acceleration].map((inference) => | ||
| coverageKey({ agent, architecture, acceleration, inference }), | ||
| ), | ||
| ), | ||
| ), | ||
| ).sort(compareCodeUnits); | ||
| } | ||
|
|
||
| function compileCase( | ||
| definition: NativeRuntimeQualificationDefinition, | ||
| input: NativeRuntimeQualificationCase, | ||
| ): Readonly<NativeRuntimeQualificationCase> { | ||
| const profile = defineExecutionProfile(input.profile); | ||
| if (profile.provider !== definition.provider) { | ||
| throw new Error( | ||
| `Qualification case '${input.id}' profile provider '${profile.provider}' does not match '${definition.provider}'`, | ||
| ); | ||
| } | ||
| if (profile.platform !== "linux" || profile.rootMode !== "rootless") { | ||
| throw new Error(`Qualification case '${input.id}' must use a rootless Linux profile`); | ||
| } | ||
| const capabilities = new Set(profile.capabilities); | ||
| const missingCapabilities = REQUIRED_CAPABILITIES.filter((value) => !capabilities.has(value)); | ||
| if (missingCapabilities.length > 0 || capabilities.has("transport.docker-socket")) { | ||
| throw new Error( | ||
| `Qualification case '${input.id}' must be socket-free and declares invalid capabilities (missing: ${missingCapabilities.join(", ") || "none"})`, | ||
| ); | ||
| } | ||
| if (input.gate !== "protected-e2e") { | ||
| throw new Error(`Qualification case '${input.id}' must run through protected E2E`); | ||
| } | ||
| if (input.install !== "release-installer") { | ||
| throw new Error(`Qualification case '${input.id}' must exercise the release installer`); | ||
| } | ||
| if (input.dockerAvailability !== "unavailable") { | ||
| throw new Error(`Qualification case '${input.id}' must prove Docker is unavailable`); | ||
| } | ||
| const allowedInference = NATIVE_RUNTIME_QUALIFICATION_INFERENCE[profile.acceleration]; | ||
| if (!(allowedInference as readonly string[]).includes(input.inference)) { | ||
| throw new Error( | ||
| `Qualification case '${input.id}' cannot use ${input.inference} with ${profile.acceleration}`, | ||
| ); | ||
| } | ||
| assertExactSet( | ||
| input.obligations, | ||
| NATIVE_RUNTIME_QUALIFICATION_OBLIGATIONS, | ||
| `Qualification case '${input.id}' obligations`, | ||
| ); | ||
| assertExactSet( | ||
| input.evidenceKinds, | ||
| requiredQualificationEvidenceKinds(profile.acceleration), | ||
| `Qualification case '${input.id}' evidence kinds`, | ||
| ); | ||
| const expectedId = qualificationCaseId({ | ||
| provider: definition.provider, | ||
| agent: input.agent, | ||
| architecture: profile.architecture, | ||
| acceleration: profile.acceleration, | ||
| inference: input.inference, | ||
| }); | ||
| if (input.id !== expectedId) { | ||
| throw new Error(`Qualification case id '${input.id}' must be '${expectedId}'`); | ||
| } | ||
| return Object.freeze({ | ||
| ...input, | ||
| profile, | ||
| obligations: Object.freeze([...input.obligations]), | ||
| evidenceKinds: Object.freeze([...input.evidenceKinds]), | ||
| }); | ||
| } | ||
|
|
||
| export function compileNativeRuntimeQualification( | ||
| input: NativeRuntimeQualificationDefinition, | ||
| ): CompiledNativeRuntimeQualification { | ||
| const id = assertSingleLine(input.id, "Native runtime qualification id"); | ||
| const repository = assertSingleLine(input.repository, "Native runtime qualification repository"); | ||
| if (!REPOSITORY_PATTERN.test(repository)) { | ||
| throw new Error(`Native runtime qualification repository '${repository}' must be owner/name`); | ||
| } | ||
| const protectedWorkflow = assertSingleLine( | ||
| input.protectedWorkflow, | ||
| "Native runtime qualification protected workflow", | ||
| ); | ||
| const engineName = assertSingleLine(input.engineName, "Native runtime engine name"); | ||
| const cases = input.cases.map((entry) => compileCase(input, entry)); | ||
| const casesByCoverage = new Map<string, Readonly<NativeRuntimeQualificationCase>>(); | ||
| for (const entry of cases) { | ||
| const key = coverageKey({ | ||
| agent: entry.agent, | ||
| architecture: entry.profile.architecture, | ||
| acceleration: entry.profile.acceleration, | ||
| inference: entry.inference, | ||
| }); | ||
| if (casesByCoverage.has(key)) { | ||
| throw new Error(`Native runtime qualification repeats case coverage '${key}'`); | ||
| } | ||
| casesByCoverage.set(key, entry); | ||
| } | ||
| const expectedCoverage = requiredCoverageKeys(); | ||
| const missing = expectedCoverage.filter((key) => !casesByCoverage.has(key)); | ||
| const unknown = [...casesByCoverage.keys()].filter((key) => !expectedCoverage.includes(key)); | ||
| if (missing.length > 0 || unknown.length > 0) { | ||
| throw new Error( | ||
| `Native runtime qualification coverage is incomplete (missing: ${missing.join(", ") || "none"}; unknown: ${unknown.join(", ") || "none"})`, | ||
| ); | ||
| } | ||
| const compiled = Object.freeze({ | ||
| id, | ||
| repository, | ||
| protectedWorkflow, | ||
| provider: input.provider, | ||
| engineName, | ||
| cases: Object.freeze([...cases].sort((left, right) => compareCodeUnits(left.id, right.id))), | ||
| }); | ||
| return compiled; | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.