From 5148f261d5abace436bf80ad4aad24fcf9817b27 Mon Sep 17 00:00:00 2001 From: Deepak Jain Date: Fri, 14 Aug 2026 13:07:48 -0700 Subject: [PATCH 01/23] test(e2e): bound downstream retries by evidence Fixes #9166 Signed-off-by: Deepak Jain --- .github/workflows/e2e-main-retry.yaml | 2 +- test/e2e/README.md | 27 +-- test/e2e/RETRY_INVENTORY.md | 53 +++++ test/e2e/fixtures/inference-switch-retry.ts | 40 ++-- test/e2e/fixtures/retry-policy.ts | 185 ++++++++++++++++ test/e2e/lib/inference-switch-retry.sh | 18 +- test/e2e/live/agent-turn-latency-helpers.ts | 8 +- .../e2e/live/cloud-inference-provider-skip.ts | 34 +++ test/e2e/live/cloud-inference.test.ts | 149 ++++++++----- test/e2e/live/common-egress-agent-helpers.ts | 11 + test/e2e/live/common-egress-agent.test.ts | 12 +- test/e2e/live/hermes-e2e.test.ts | 102 +++------ .../cloud-inference-provider-skip.test.ts | 30 +++ .../common-egress-agent-helpers.test.ts | 9 + ...mes-inference-switch-command-shape.test.ts | 19 +- .../support/inference-switch-retry.test.ts | 83 +++++++- test/e2e/support/main-run-retry.test.ts | 54 ++--- test/e2e/support/retry-policy.test.ts | 197 ++++++++++++++++++ tools/e2e/main-run-retry.mts | 43 ++-- 19 files changed, 828 insertions(+), 248 deletions(-) create mode 100644 test/e2e/RETRY_INVENTORY.md create mode 100644 test/e2e/fixtures/retry-policy.ts create mode 100644 test/e2e/support/retry-policy.test.ts diff --git a/.github/workflows/e2e-main-retry.yaml b/.github/workflows/e2e-main-retry.yaml index a9eaef56f6b..9ea9161edb5 100644 --- a/.github/workflows/e2e-main-retry.yaml +++ b/.github/workflows/e2e-main-retry.yaml @@ -35,7 +35,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 10 permissions: - actions: write + actions: read contents: read steps: - name: Checkout trusted retry controller diff --git a/test/e2e/README.md b/test/e2e/README.md index 2fa460eea4a..6cf67ec24c0 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -19,7 +19,8 @@ before those targets run; local runners must provide it themselves. failures from approved `main` workflows and requests one full rerun only when every non-passing job has authenticated GitHub-hosted runner-loss evidence. - `.github/workflows/e2e-main-retry.yaml` evaluates eligible `E2E main` push - attempts, requests at most two failed-job reruns, and uploads attempt evidence. + attempts and uploads attempt evidence. It never authorizes a broad failed-job + rerun; retry decisions belong to bounded operation-level policies. - The `staging-brev-launchable` job in `.github/workflows/e2e.yaml` validates the baked candidate without installing or copying NemoClaw source. - `.github/workflows/platform-vitest-main.yaml` publishes `CI / Platform Evidence` for Ubuntu 26.04, macOS, and WSL. @@ -667,18 +668,18 @@ It does not handle `E2E main`. The complete non-passing job listing must contain only authenticated hosted-runner-loss evidence for the workflow's approved runner labels. An ordinary assertion failure, mixed failure set, incomplete listing, custom or self-hosted label, changed evidence, or ambiguous pagination prevents recovery. -For eligible `E2E main` push runs, `E2E / Main Retry` asks GitHub Actions to rerun failed jobs and their dependent jobs. -A successful CLI artifact producer is not rerun. -The workflow retains its CLI artifact for 3 days. -During that period, consumers can reuse the immutable, content-addressed artifact from an earlier producer attempt in the same workflow run. -If the artifact is unavailable when a consumer downloads it, restoration fails because the failed-job rerun does not rerun the successful producer. -Restore validation binds the producer provenance to the workflow run, workflow SHA, and candidate checkout. -It downloads by immutable artifact ID and verifies the manifest and the payload digest. -It rejects a producer attempt newer than the consumer attempt. -The controller can request two reruns. -It does not verify that GitHub schedules a different runner, so do not treat a rerun as evidence of a fresh host. -It ignores manual runs and source runs superseded by a newer `main` push. -The controller checks out only trusted default-branch code and receives no repository secrets. +For eligible `E2E main` push runs, `E2E / Main Retry` records first-attempt, +manual-retry, and exhausted-attempt outcomes without requesting a workflow +rerun. A failed job can represent a deterministic product assertion, +authentication or authorization failure, policy denial, malformed input, +ambiguous mutation, cleanup failure, or an external transient. GitHub job +conclusions do not distinguish those classes, so a broad failed-job rerun is +not authorized evidence. External operations use the checked-in retry inventory +and an explicit bounded policy; new shared paths use the bounded operation +helper. Their artifacts retain each attempt. +Hosted runner loss remains owned by Hosted Runner Recovery. The observer ignores +manual source runs and source runs superseded by a newer `main` push, checks out +only trusted default-branch code, and receives no repository secrets. The runner-allocation and internal-error failures handled by Hosted Runner Recovery originate in GitHub Actions, outside repository-controlled workflow diff --git a/test/e2e/RETRY_INVENTORY.md b/test/e2e/RETRY_INVENTORY.md new file mode 100644 index 00000000000..01cb4f5e704 --- /dev/null +++ b/test/e2e/RETRY_INVENTORY.md @@ -0,0 +1,53 @@ +# E2E retry inventory + +This inventory governs operation retries, workflow reruns, and transient skips in +the E2E suite. Readiness and eventual-consistency polling are observations, not +operation retries: they do not repeat a mutation and retain their final probe or +per-attempt artifacts through the existing E2E fixture APIs. + +Only a `transient-external` classification may retry. Authentication, +authorization, policy denial, malformed input, deterministic product failure, +and cleanup failure are terminal. An ambiguous mutation must reconcile external +state before another attempt. Exhaustion remains failed. + +| ID | Operation and source | Failure signature | Bound and backoff | Idempotence basis | Owner | Result classification | Retained evidence | Disposition | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `workflow-main-observer` | Completed `E2E main` run; `tools/e2e/main-run-retry.mts`, `.github/workflows/e2e-main-retry.yaml` | Any failed job | 0 automatic reruns | Not established at workflow scope | Test owner | `failed-no-retry`, `passed-first-attempt`, or `passed-after-retry` for a manual rerun | Per-attempt jobs, conclusions, duration, source SHA, and owner | Broad rerun removed; operation-level evidence is required | +| `hosted-runner-recovery` | Confirmed GitHub-hosted runner loss; `tools/e2e/hosted-runner-recovery.mts`, `tools/e2e/hosted-runner-loss*.mts` | Authenticated runner-allocation or internal-runner evidence only | 1 recovery request; controller-defined delay | GitHub reruns a workflow attempt | GitHub Actions | Dedicated runner-loss classifications | Source and recovery run links plus authenticated job evidence | External owner; governed by #7146, not this policy | +| `pr-rerun-reconciliation` | PR E2E dispatch reconciliation; `tools/e2e/pr-e2e-dispatch-reconciliation.mts`, `tools/e2e/pr-e2e-retry-receipt.mts` | Trusted dispatch receipt state | Contract-defined single reconciliation | Reconciles workflow and commit identity before action | GitHub Actions | Receipt-specific terminal states | Signed workflow identity and receipt | External scope; governed by #7206 | +| `github-publication-read` | GitHub API reads; `tools/e2e/base-image-publication.mts` | Fetch error, 408, rate limit, or 5xx | 3 attempts; Retry-After/rate-limit reset or linear delay capped at 10s | Read-only | GitHub API | Returned parsed selection on success; thrown terminal HTTP/fetch error on failure or exhaustion | Caller artifact records the returned publication selection; terminal errors identify exhausted fetch or HTTP status without response content | Eligible bounded read; existing implementation retained | +| `inference-switch-ts` | Verified inference route update; `test/e2e/fixtures/inference-switch-retry.ts` | Timeout, reset, DNS/connectivity/connect error, request transport error, or exact 502/503/504 status | 1-10 attempts; linear 5s | Setting the same desired provider/model is idempotent | Inference provider | Shared `RetryEvidence` classifications | Every attempt classification and aggregate outcome; command artifacts remain separate and redacted | Uses `runBoundedRetry`; deterministic verification mismatches stop; no `--no-verify` exhaustion bypass | +| `inference-switch-shell` | Verified shell inference route update; `test/e2e/lib/inference-switch-retry.sh` | Same bounded transient signature as TypeScript helper | 1-10 attempts; linear 5s | Setting the same desired provider/model is idempotent | Inference provider | Exit status remains failed on exhaustion | Existing command output and retry progress | Bounded compatibility helper; no `--no-verify` exhaustion bypass | +| `provider-install-standard` | Provider validation during Brave, cron, device-auth, Hermes-switch, network-policy, and restricted onboarding | `isTransientProviderValidationFailure` allowlist only | 1 local or 3 CI attempts; linear 10s backoff | Repeats the same desired onboarding state; restricted paths destroy the prior sandbox before retry | Inference provider | Transient allowlist versus terminal install failure | Per-attempt command artifacts; restricted paths add a terminal skip artifact | Existing bounded paths; no deterministic install retry | +| `provider-install-agent-turn` | Agent-turn latency install; `test/e2e/live/agent-turn-latency-helpers.ts` | Same transient provider allowlist | 1-10 attempts (default 2); linear 10s backoff | Cleanup completes before repeating the same install request | Inference provider | Transient allowlist versus terminal install failure | Per-attempt install and cleanup artifacts plus progress events | Existing bounded path; no deterministic install retry | +| `provider-install-launchable` | Launchable smoke onboarding; `test/e2e/live/launchable-smoke.test.ts` | Same transient provider allowlist | 3 attempts; linear 30s backoff | Repeats the same desired non-interactive onboarding state | Inference provider | Transient allowlist versus terminal install failure | Per-attempt onboarding artifacts and terminal CI skip evidence | Existing bounded path; no deterministic install retry | +| `provider-ci-skip` | Cloud-provider validation skip; `test/e2e/live/cloud-inference-provider-skip.ts` | Same transient provider allowlist | No retry; one CI-only skip | Read-only provider validation | Inference provider | Explicit transient skip versus terminal failure | Skip reason and command artifact | Intentional transient skip; release conclusion unchanged | +| `hosted-inference-budget` | Hosted inference response-budget escalation; `test/e2e/live/hermes-e2e.test.ts` | Reasoning consumed a smaller response budget | No retry; one request with the sufficient bounded budget | Read-only inference request | Test owner | Deterministic response assertion | Direct and sandbox chat artifacts | Ineligible deterministic retry removed | +| `cloud-inference-probe` | Hosted cloud inference request; `test/e2e/live/cloud-inference.test.ts` | Curl transport timeout/reset/DNS/connect error, HTTP 429, or HTTP 5xx | 1-10 attempts (default 3); fixed 5s delay | Read-only inference request | Inference provider | Explicit external transient versus terminal response/semantic assertion | Per-request artifacts and aggregate attempt evidence | Eligible bounded read; HTTP status is separated from provider content and deterministic assertions do not retry | +| `agent-turn-probe` | OpenClaw and Hermes agent turns; `test/e2e/live/common-egress-agent.test.ts`, `test/e2e/live/common-egress-agent-helpers.ts` | Exact OpenClaw scope-upgrade signal followed by successful recovery, or non-200 transport/provider timeout, reset, DNS, connection, 408/429/5xx signature | 3 attempts; OpenClaw linear 15s, Hermes fixed 5s | Read-only prompt; scope-upgrade retry requires successful recovery reconciliation | Agent runtime or inference provider | Explicit transient/reconciled convergence versus terminal product response | Per-attempt result, stdout, stderr, recovery artifact, and progress | Existing bounded path; HTTP 200 product replies and unconditional fallback retry remain terminal | +| `mcp-tool-discovery` | MCP status tool discovery; `test/e2e/live/mcp-bridge-tool-discovery.ts` | Transport failure before the fixture receives a request | 2 attempts; 1s | Read-only status request | OpenShell transport | Transport-only retry versus terminal discovery assertion | Per-attempt status artifact and fixture request log | Eligible existing bounded read | +| `mcp-restart-recovery` | MCP command after Hermes restart; `test/e2e/live/mcp-bridge-reliability.ts`, `test/e2e/live/mcp-bridge-hermes-lifecycle.ts` | Exact gateway-draining or restart transport signature after committed bridge proof | 1 restart recovery or 3 draining retries; 5s | Read-only command after committed-state verification | Hermes gateway | Exact transient signature versus terminal result | Initial and retry command artifacts | Eligible existing bounded read with precondition proof | +| `mcp-cleanup` | MCP bridge and sandbox cleanup; `test/e2e/live/mcp-bridge-cleanup.ts` | Exact concurrent-modification delete response | 2 attempts; immediate | Delete is reconciled through registry absence/presence checks | OpenShell | Cleanup always remains visible | Cleanup ledger and final registry evidence | Reconciled cleanup retry; failure is terminal | +| `windows-sandbox-delete` | Windows process-container cleanup; `test/e2e/live/windows-mxc-openclaw-process-container-helpers.ts` | Delete accepted but exact registry entry remains | 2 delete attempts; immediate | Registry read reconciles mutation result | OpenShell | Cleanup failure remains terminal | Delete result and exact registry evidence | Reconciled mutation retry | +| `state-backup-destroy` | Backup/restore sandbox destruction; `test/e2e/live/state-backup-restore.test.ts` | Nonzero destroy followed by state inspection | 2 attempts; 10s | State inspection reconciles sandbox existence | OpenShell | Cleanup failure remains terminal | Per-attempt destroy artifacts | Reconciled cleanup retry | +| `tunnel-lifecycle-observation` | Tunnel CLI URL and public reachability observation; `test/e2e/live/tunnel-lifecycle-helpers.ts` | Expected missing URL or not-yet-ready HTTP response | URL: 15 probes at 1s; public reachability: 15 probes with capped caller backoff; post-stop: 10 probes at 1s | Read-only observation after one start/stop mutation | Cloudflare tunnel | Ready URL/response or exhausted observation | Per-probe command artifact and final parsed response | No start mutation retry | +| `mcp-quick-tunnel-establishment` | Public MCP tunnel; `test/e2e/live/mcp-bridge-servers.ts` | Child exit/error, 45s readiness deadline, or missing public origin | 3 attempts; 500ms readiness probes; linear 1s inter-attempt delay | Failed child is stopped before replacement | Cloudflare tunnel | Ready public origin or exhausted setup | Bounded diagnostic, progress events, and returned origin; raw child output excluded | Eligible bounded external setup | +| `eventual-consistency-polling` | Generic readiness observation; `test/e2e/fixtures/polling.ts` and callers | Caller-defined expected not-ready state | Helper requires a finite `attempts` or `deadlineMs`; caller supplies explicit `delayMs` | Read-only observation | Component named by caller | Passed before bound, terminal observation, aborted, or exhausted | Last attempt and deterministic per-attempt artifact name | Not an operation retry; each caller owns and inventories its concrete bound | +| `filesystem-cleanup` | Node `fs.rmSync` retry options; `tools/e2e/openshell-gateway-auth-artifact-safety.mts` | OS-level transient remove failure | 4 total filesystem attempts; 50ms | Idempotent removal of quarantined local artifacts | Host filesystem | Cleanup failure remains terminal | Quarantine path state without artifact content | Bounded cleanup; credentials are never serialized | + +## Evidence contract + +`test/e2e/fixtures/retry-policy.ts` emits schema version 1. The aggregate +`outcome` is exactly one of `passed-first-attempt`, `passed-after-retry`, +`failed-no-retry`, or `exhausted`. Cleanup failures use `failed-no-retry` with a +final attempt whose `failureClass` is `cleanup`. Each attempt also records its +number, failure class, reconciliation result when applicable, and whether +another attempt was scheduled. The record deliberately excludes command output, +errors, request bodies, headers, and environment values. Callers retain their +normal redacted artifacts separately and may write the aggregate record through +`onEvidence`. + +Adding or changing a retry, rerun, transient skip, or polling family requires an +inventory update in the same pull request. A path is not eligible merely because +it sometimes passes later; it needs a narrow transient signature and a stated +idempotence or reconciliation basis. diff --git a/test/e2e/fixtures/inference-switch-retry.ts b/test/e2e/fixtures/inference-switch-retry.ts index 38f901a37ad..c5d2772c9af 100644 --- a/test/e2e/fixtures/inference-switch-retry.ts +++ b/test/e2e/fixtures/inference-switch-retry.ts @@ -2,15 +2,16 @@ // SPDX-License-Identifier: Apache-2.0 import type { ShellProbeResult } from "./shell-probe.ts"; +import { runBoundedRetry, type RetryEvidence } from "./retry-policy.ts"; const TRANSIENT_INFERENCE_SET_FAILURE = - /timed? out|timeout|ETIMEDOUT|ECONNRESET|EAI_AGAIN|ENOTFOUND|failed to connect|error sending request|failed to verify inference endpoint|502|503|504|temporar/iu; + /timed? out|timeout|ETIMEDOUT|ECONNRESET|EAI_AGAIN|ENOTFOUND|failed to connect|error sending request|\b50[234]\b/iu; export function inferenceSetAttemptCount(raw: string | undefined, fallback = 3): number { if (raw === undefined) return fallback; const parsed = Number(raw); - if (!Number.isInteger(parsed) || parsed < 1) { - throw new Error(`NEMOCLAW_SWITCH_SET_ATTEMPTS must be a positive integer; got ${raw}`); + if (!Number.isInteger(parsed) || parsed < 1 || parsed > 10) { + throw new Error(`NEMOCLAW_SWITCH_SET_ATTEMPTS must be between 1 and 10; got ${raw}`); } return parsed; } @@ -28,18 +29,25 @@ export async function runInferenceSetWithRetry(options: { attempts: number; delay?: (milliseconds: number) => Promise; run: (attempt: number, verify: boolean) => Promise; + onEvidence?: (evidence: RetryEvidence) => Promise | void; }): Promise { - const delay = - options.delay ?? - ((milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds))); - for (let attempt = 1; attempt <= options.attempts; attempt += 1) { - const result = await options.run(attempt, true); - if (result.exitCode === 0 || !isTransientInferenceSetFailure(result)) return result; - if (attempt < options.attempts) { - await delay(attempt * 5_000); - continue; - } - return options.run(attempt, false); - } - throw new Error("Inference switch retry loop completed without running an attempt."); + const execution = await runBoundedRetry({ + operation: "inference.switch.verify", + owner: "inference-provider", + idempotence: "idempotent", + maxAttempts: options.attempts, + run: (attempt) => options.run(attempt, true), + classify: (result) => { + if (result?.exitCode === 0) return { outcome: "passed" }; + return { + outcome: "failed", + failureClass: + result && isTransientInferenceSetFailure(result) ? "transient-external" : "deterministic", + }; + }, + delayMs: (attempt) => attempt * 5_000, + sleep: options.delay, + onEvidence: options.onEvidence, + }); + return execution.value!; } diff --git a/test/e2e/fixtures/retry-policy.ts b/test/e2e/fixtures/retry-policy.ts new file mode 100644 index 00000000000..56c099cc413 --- /dev/null +++ b/test/e2e/fixtures/retry-policy.ts @@ -0,0 +1,185 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +export type RetryFailureClass = + | "authentication" + | "authorization" + | "cleanup" + | "deterministic" + | "malformed-input" + | "policy-denial" + | "transient-external" + | "ambiguous-mutation"; + +export type RetryIdempotence = "read-only" | "idempotent" | "reconciled-mutation"; + +export interface RetryAttemptEvidence { + attempt: number; + outcome: "failed" | "passed"; + failureClass?: RetryFailureClass; + reconciled?: boolean; + retryScheduled: boolean; +} + +export interface RetryEvidence { + schemaVersion: 1; + operation: string; + owner: string; + idempotence: RetryIdempotence; + maxAttempts: number; + outcome: "failed-no-retry" | "exhausted" | "passed-after-retry" | "passed-first-attempt"; + attempts: RetryAttemptEvidence[]; +} + +export class RetryPolicyError extends Error { + constructor( + message: string, + readonly evidence: RetryEvidence, + ) { + super(message); + } +} + +type AttemptClassification = + | { outcome: "passed" } + | { outcome: "failed"; failureClass: RetryFailureClass }; + +export interface BoundedRetryOptions { + operation: string; + owner: string; + idempotence: RetryIdempotence; + maxAttempts: number; + run: (attempt: number) => Promise; + classify: (value: T | undefined, error: unknown) => AttemptClassification; + reconcile?: (value: T | undefined, error: unknown, attempt: number) => Promise; + delayMs?: number | ((attempt: number) => number); + sleep?: (milliseconds: number) => Promise; + onEvidence?: (evidence: RetryEvidence) => Promise | void; +} + +/** Reject unbounded or artifact-unsafe retry metadata before an operation runs. */ +function validateOptions(options: BoundedRetryOptions): void { + if (!/^[a-z0-9][a-z0-9._-]{0,127}$/u.test(options.operation)) { + throw new Error("retry operation must be a bounded identifier"); + } + if (!/^[a-z0-9][a-z0-9._-]{0,127}$/u.test(options.owner)) { + throw new Error("retry owner must be a bounded identifier"); + } + if ( + !Number.isSafeInteger(options.maxAttempts) || + options.maxAttempts < 1 || + options.maxAttempts > 10 + ) { + throw new Error("retry maxAttempts must be between 1 and 10"); + } +} + +/** Build an immutable aggregate record from the retained per-attempt facts. */ +function finalEvidence( + options: Pick< + BoundedRetryOptions, + "operation" | "owner" | "idempotence" | "maxAttempts" + >, + attempts: RetryAttemptEvidence[], + outcome: RetryEvidence["outcome"], +): RetryEvidence { + return { + schemaVersion: 1, + operation: options.operation, + owner: options.owner, + idempotence: options.idempotence, + maxAttempts: options.maxAttempts, + outcome, + attempts: attempts.map((attempt) => ({ ...attempt })), + }; +} + +/** Publish final evidence through the caller-owned artifact boundary. */ +async function emit( + options: Pick, "onEvidence">, + evidence: RetryEvidence, +): Promise { + await options.onEvidence?.(evidence); +} + +/** + * Execute an operation with a bounded, fail-closed retry policy. + * + * Only externally transient failures are retryable. Mutations additionally + * require a successful reconciliation before another attempt is authorized. + * Evidence deliberately contains no command output, exception text, or request + * data, so credential-bearing values cannot enter retained retry artifacts. + * + * A resolved operation returns its value with a terminal `evidence.outcome`, + * including failed outcomes. A thrown operation raises `RetryPolicyError` with + * the same evidence. Callers must inspect the outcome after a resolved promise. + */ +export async function runBoundedRetry( + options: BoundedRetryOptions, +): Promise<{ value: T | undefined; evidence: RetryEvidence }> { + validateOptions(options); + const sleep = + options.sleep ?? + ((milliseconds: number) => new Promise((resolve) => setTimeout(resolve, milliseconds))); + const attempts: RetryAttemptEvidence[] = []; + + for (let attempt = 1; attempt <= options.maxAttempts; attempt += 1) { + let value: T | undefined; + let error: unknown; + try { + value = await options.run(attempt); + } catch (caught) { + error = caught; + } + + const classification = options.classify(value, error); + if (classification.outcome === "passed") { + if (error !== undefined) throw new Error("retry classifier reported success after an error"); + attempts.push({ attempt, outcome: "passed", retryScheduled: false }); + const evidence = finalEvidence( + options, + attempts, + attempt === 1 ? "passed-first-attempt" : "passed-after-retry", + ); + await emit(options, evidence); + return { value, evidence }; + } + + const isTransient = classification.failureClass === "transient-external"; + const hasBudget = attempt < options.maxAttempts; + let reconciled = options.idempotence !== "reconciled-mutation"; + if (isTransient && hasBudget && options.idempotence === "reconciled-mutation") { + reconciled = (await options.reconcile?.(value, error, attempt)) === true; + } + const retryScheduled = isTransient && hasBudget && reconciled; + attempts.push({ + attempt, + outcome: "failed", + failureClass: classification.failureClass, + ...(options.idempotence === "reconciled-mutation" && isTransient && hasBudget + ? { reconciled } + : {}), + retryScheduled, + }); + + if (retryScheduled) { + const delay = + typeof options.delayMs === "function" ? options.delayMs(attempt) : (options.delayMs ?? 0); + if (!Number.isSafeInteger(delay) || delay < 0 || delay > 300_000) { + throw new Error("retry delay must be between 0 and 300000 milliseconds"); + } + if (delay > 0) await sleep(delay); + continue; + } + + const outcome = isTransient && !hasBudget ? "exhausted" : "failed-no-retry"; + const evidence = finalEvidence(options, attempts, outcome); + await emit(options, evidence); + if (error !== undefined) { + throw new RetryPolicyError(`${options.operation} ${outcome}`, evidence); + } + return { value, evidence }; + } + + throw new Error("bounded retry loop completed without an attempt"); +} diff --git a/test/e2e/lib/inference-switch-retry.sh b/test/e2e/lib/inference-switch-retry.sh index 753971314e5..f2367394da8 100755 --- a/test/e2e/lib/inference-switch-retry.sh +++ b/test/e2e/lib/inference-switch-retry.sh @@ -4,11 +4,10 @@ # Shared retry helpers for inference-switch E2Es. These tests still verify the # final OpenShell route, sandbox config, and live inference after this helper -# returns. The --no-verify fallback is only used after verified route-setting -# attempts fail with transient upstream/network symptoms. +# returns. Exhausting the transient retry budget remains a verified failure. is_transient_inference_set_failure() { - grep -qiE 'timed? out|timeout|ETIMEDOUT|ECONNRESET|EAI_AGAIN|ENOTFOUND|failed to connect|error sending request|failed to verify inference endpoint|502|503|504|temporar' <<<"$1" + grep -qiE 'timed? out|timeout|ETIMEDOUT|ECONNRESET|EAI_AGAIN|ENOTFOUND|failed to connect|error sending request|(^|[^0-9])50[234]([^0-9]|$)' <<<"$1" } log_inference_switch_retry_info() { @@ -21,8 +20,8 @@ log_inference_switch_retry_info() { run_inference_set_with_retry() { local attempts="${NEMOCLAW_SWITCH_SET_ATTEMPTS:-3}" - if ! [[ "$attempts" =~ ^[1-9][0-9]*$ ]]; then - printf 'Invalid NEMOCLAW_SWITCH_SET_ATTEMPTS=%s; expected a positive integer.\n' "$attempts" >&2 + if ! [[ "$attempts" =~ ^[1-9][0-9]*$ ]] || [ "$attempts" -gt 10 ]; then + printf 'Invalid NEMOCLAW_SWITCH_SET_ATTEMPTS=%s; expected an integer between 1 and 10.\n' "$attempts" >&2 return 2 fi if [ "$#" -eq 0 ]; then @@ -30,7 +29,7 @@ run_inference_set_with_retry() { return 2 fi - local attempt rc output fallback_output + local attempt rc output local -a command=("$@") for ((attempt = 1; attempt <= attempts; attempt++)); do output=$("${command[@]}" 2>&1) @@ -41,13 +40,6 @@ run_inference_set_with_retry() { fi if ! is_transient_inference_set_failure "$output" || [ "$attempt" -ge "$attempts" ]; then - if is_transient_inference_set_failure "$output"; then - log_inference_switch_retry_info "Verified inference switch failed after ${attempts} transient attempt(s); retrying with --no-verify before live route checks..." - fallback_output=$("${command[@]}" --no-verify 2>&1) - rc=$? - printf '%s\n%s\n' "$output" "$fallback_output" - return "$rc" - fi printf '%s\n' "$output" return "$rc" fi diff --git a/test/e2e/live/agent-turn-latency-helpers.ts b/test/e2e/live/agent-turn-latency-helpers.ts index e01ab3464d3..a69c7a140b7 100644 --- a/test/e2e/live/agent-turn-latency-helpers.ts +++ b/test/e2e/live/agent-turn-latency-helpers.ts @@ -36,7 +36,7 @@ export const HERMES_SANDBOX = validateSandboxName(OPENCLAW_SANDBOX); validateSandboxName(HERMES_SANDBOX); export const MAX_TURN_SECONDS = positiveInt(process.env.NEMOCLAW_TURN_LATENCY_MAX_SECONDS, 300); -const INSTALL_ATTEMPTS = positiveInt(process.env.NEMOCLAW_TURN_LATENCY_INSTALL_ATTEMPTS, 2); +const INSTALL_ATTEMPTS = boundedAttempts(process.env.NEMOCLAW_TURN_LATENCY_INSTALL_ATTEMPTS, 2); const INSTALL_TIMEOUT_MS = 30 * 60_000; type AgentTurnProgress = Pick; @@ -45,6 +45,12 @@ function positiveInt(value: string | undefined, fallback: number): number { return value && /^[1-9][0-9]*$/u.test(value) ? Number.parseInt(value, 10) : fallback; } +function boundedAttempts(value: string | undefined, fallback: number): number { + const attempts = positiveInt(value, fallback); + if (attempts > 10) throw new Error("install attempts must be between 1 and 10"); + return attempts; +} + export function env( sandboxName: string, agent: "openclaw" | "hermes", diff --git a/test/e2e/live/cloud-inference-provider-skip.ts b/test/e2e/live/cloud-inference-provider-skip.ts index c720019c684..c7017d94286 100644 --- a/test/e2e/live/cloud-inference-provider-skip.ts +++ b/test/e2e/live/cloud-inference-provider-skip.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { resultText } from "../fixtures/clients/command.ts"; +import type { RetryFailureClass } from "../fixtures/retry-policy.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; import { isTransientProviderValidationFailure } from "./network-policy-transient-provider.ts"; @@ -11,6 +12,9 @@ const RATE_LIMIT_OR_SANITIZED_EXTERNAL_RE = /HTTP 429|\b429\b|rate[- ]?limit|too many requests|quota|temporar|timed? out|timeout|ETIMEDOUT|ECONNRESET|EAI_AGAIN|ENOTFOUND|failed to connect|error sending request|\b(redacted|sanitized)\b/i; const CREDENTIAL_OR_AUTH_RE = /invalid.*(api[_-]?key|credential)|unauthorized|forbidden|HTTP 40[13]\b|\b40[13]\b/i; +const TRANSIENT_CHAT_FAILURE = + /timed? out|timeout|ETIMEDOUT|ECONNRESET|EAI_AGAIN|ENOTFOUND|failed to connect|rate[- ]?limit/iu; +const CLOUD_CHAT_HTTP_STATUS_MARKER = "__NEMOCLAW_HTTP_STATUS__:"; export const PRE_CONTRACT_EXTERNAL_PROVIDER_SKIP_REASON = "external-provider-validation-unavailable-before-legacy-contract"; @@ -39,6 +43,36 @@ export interface PreContractExternalProviderSkipEvidence { removalCondition: typeof PRE_CONTRACT_EXTERNAL_PROVIDER_REMOVAL_CONDITION; } +/** Classify a cloud chat failure without retaining provider output in evidence. */ +export function classifyCloudChatFailure( + httpStatus: string, + transportOutput: string, + failure: string, + error: unknown, +): RetryFailureClass { + const thrownDetail = error instanceof Error ? error.message : ""; + if (httpStatus === "429" || /^5\d{2}$/u.test(httpStatus)) return "transient-external"; + if (TRANSIENT_CHAT_FAILURE.test(`${transportOutput}\n${thrownDetail}`)) { + return "transient-external"; + } + return failure === "response was not parseable JSON" ? "malformed-input" : "deterministic"; +} + +/** Separate curl's transport-status trailer from the provider response body. */ +export function parseCloudChatResponse(stdout: string): { body: string; httpStatus: string } { + const marker = `\n${CLOUD_CHAT_HTTP_STATUS_MARKER}`; + const markerIndex = stdout.lastIndexOf(marker); + if (markerIndex < 0) return { body: stdout, httpStatus: "" }; + return { + body: stdout.slice(0, markerIndex), + httpStatus: stdout.slice(markerIndex + marker.length).trim(), + }; +} + +export function cloudChatWriteOutArg(): string { + return `\n${CLOUD_CHAT_HTTP_STATUS_MARKER}%{http_code}\n`; +} + function tailForEvidence(text: string, maxLength = 1600): string { return text.length > maxLength ? text.slice(-maxLength) : text; } diff --git a/test/e2e/live/cloud-inference.test.ts b/test/e2e/live/cloud-inference.test.ts index 7b8f075e7dc..64a5ccee850 100644 --- a/test/e2e/live/cloud-inference.test.ts +++ b/test/e2e/live/cloud-inference.test.ts @@ -22,10 +22,14 @@ import { expect, test } from "../fixtures/e2e-test.ts"; import { testHomeEnvironment } from "../fixtures/environment-profiles.ts"; import { requireHostedInferenceConfig } from "../fixtures/hosted-inference.ts"; import { CLI_ENTRYPOINT, REPO_ROOT } from "../fixtures/paths.ts"; +import { runBoundedRetry } from "../fixtures/retry-policy.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; import { buildPreContractExternalProviderSkipEvidence, + classifyCloudChatFailure, classifyPreContractExternalProviderFailure, + cloudChatWriteOutArg, + parseCloudChatResponse, type PreContractExternalProviderFailure, } from "./cloud-inference-provider-skip.ts"; @@ -59,18 +63,27 @@ const INSTALL_TIMEOUT_MS = 25 * 60_000; const CHAT_TIMEOUT_MS = 120_000; const SANDBOX_PROBE_TIMEOUT_MS = 120_000; const TEST_TIMEOUT_MS = 40 * 60_000; -const MAX_ATTEMPTS = positiveInteger(process.env.E2E_PHASE_5B_MAX_ATTEMPTS, 3); -const RETRY_SLEEP_MS = positiveInteger(process.env.E2E_PHASE_5B_RETRY_SLEEP_SEC, 5) * 1_000; +const MAX_ATTEMPTS = boundedPositiveInteger( + "E2E_PHASE_5B_MAX_ATTEMPTS", + process.env.E2E_PHASE_5B_MAX_ATTEMPTS, + 3, +); +const RETRY_SLEEP_MS = + boundedPositiveInteger( + "E2E_PHASE_5B_RETRY_SLEEP_SEC", + process.env.E2E_PHASE_5B_RETRY_SLEEP_SEC, + 5, + ) * 1_000; validateSandboxName(SANDBOX_NAME); -function positiveInteger(value: string | undefined, fallback: number): number { - if (!value || !/^[1-9][0-9]*$/.test(value)) return fallback; - return Number.parseInt(value, 10); -} - -function sleep(ms: number): Promise { - return new Promise((resolve) => setTimeout(resolve, ms)); +/** Read a bounded retry setting and name invalid configuration in CI output. */ +function boundedPositiveInteger(name: string, value: string | undefined, fallback: number): number { + const parsed = value === undefined ? fallback : Number(value); + if (!Number.isSafeInteger(parsed) || parsed < 1 || parsed > 10) { + throw new Error(`${name} must be an integer between 1 and 10; got ${value}`); + } + return parsed; } async function writePreContractExternalProviderSkip( @@ -172,58 +185,92 @@ async function expectCliOnPath(host: HostCliClient, home: string): Promise async function expectLiveChatPong( sandbox: SandboxClient, + artifacts: ArtifactSink, home: string, apiKey: string, ): Promise<{ attempt: number; content: string }> { + type ChatAttempt = { + content: string; + failure: string; + httpStatus: string; + response: ShellProbeResult; + }; const payload = JSON.stringify({ model: CLOUD_MODEL, messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], max_tokens: 100, }); - let lastFailure = "chat completion was not attempted"; - - for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt += 1) { - const response = await sandbox.exec( - SANDBOX_NAME, - [ - "curl", - "-sS", - "--max-time", - "90", - "https://inference.local/v1/chat/completions", - "-H", - "Content-Type: application/json", - "--data-raw", - payload, - ], - { - artifactName: `phase-2-inference-local-chat-attempt-${attempt}`, - env: testEnv(home), - redactionValues: [apiKey], - timeoutMs: CHAT_TIMEOUT_MS, - }, - ); - - if (response.exitCode !== 0) { - lastFailure = `ssh/curl failed (exit ${response.exitCode}): ${resultText(response).slice(0, 500)}`; - } else if (!response.stdout.trim()) { - lastFailure = "empty response from inference.local"; - } else { + const execution = await runBoundedRetry({ + operation: "cloud-inference.chat", + owner: "inference-provider", + idempotence: "read-only", + maxAttempts: MAX_ATTEMPTS, + delayMs: RETRY_SLEEP_MS, + run: async (attempt) => { + const response = await sandbox.exec( + SANDBOX_NAME, + [ + "curl", + "-sS", + "--max-time", + "90", + "--write-out", + cloudChatWriteOutArg(), + "https://inference.local/v1/chat/completions", + "-H", + "Content-Type: application/json", + "--data-raw", + payload, + ], + { + artifactName: `phase-2-inference-local-chat-attempt-${attempt}`, + env: testEnv(home), + redactionValues: [apiKey], + timeoutMs: CHAT_TIMEOUT_MS, + }, + ); + const parsed = parseCloudChatResponse(response.stdout); + let content = ""; + let failure = parsed.body.trim() ? "" : "empty response from inference.local"; try { - const content = openAiChatContent(response.stdout); - if (/pong/i.test(content)) return { attempt, content }; - lastFailure = `expected PONG, got: ${content.slice(0, 300)}`; - } catch (error) { - lastFailure = `response was not parseable JSON: ${ - error instanceof Error ? error.message : String(error) - }; body: ${response.stdout.slice(0, 500)}`; + content = parsed.body.trim() ? openAiChatContent(parsed.body) : ""; + } catch { + failure = "response was not parseable JSON"; } - } - - if (attempt < MAX_ATTEMPTS) await sleep(RETRY_SLEEP_MS); + if (!failure && !/pong/iu.test(content)) { + failure = `expected PONG, got: ${content.slice(0, 300)}`; + } + return { content, failure, httpStatus: parsed.httpStatus, response }; + }, + classify: (value, error) => { + if (!value) { + return { + outcome: "failed", + failureClass: classifyCloudChatFailure("", "", "", error), + }; + } + const { content, failure, httpStatus, response } = value; + if (response.exitCode === 0 && /^2\d{2}$/u.test(httpStatus) && /pong/iu.test(content)) { + return { outcome: "passed" }; + } + return { + outcome: "failed", + failureClass: classifyCloudChatFailure(httpStatus, response.stderr, failure, error), + }; + }, + onEvidence: async (evidence) => { + await artifacts.writeJson("phase-2-inference-local-chat-retry.json", evidence); + }, + }); + const value = execution.value; + if (value && /pong/iu.test(value.content)) { + return { attempt: execution.evidence.attempts.length, content: value.content }; } - - throw new Error(`Live chat failed after ${MAX_ATTEMPTS} attempt(s): ${lastFailure}`); + throw new Error( + `Live chat failed after ${execution.evidence.attempts.length} attempt(s): ${ + value?.failure || `exit ${value?.response.exitCode ?? "unknown"}` + }`, + ); } async function expectSandboxCredentialBoundary( @@ -412,7 +459,7 @@ test( await expectCliOnPath(host, home); progress.phase("exercise managed inference.local chat"); - const chat = await expectLiveChatPong(sandbox, home, apiKey); + const chat = await expectLiveChatPong(sandbox, artifacts, home, apiKey); await artifacts.writeJson("phase-2-chat-result.json", { model: CLOUD_MODEL, attempt: chat.attempt, diff --git a/test/e2e/live/common-egress-agent-helpers.ts b/test/e2e/live/common-egress-agent-helpers.ts index 870fc9c266e..b862f7c7df4 100644 --- a/test/e2e/live/common-egress-agent-helpers.ts +++ b/test/e2e/live/common-egress-agent-helpers.ts @@ -94,6 +94,17 @@ export function agentReplyContainsToken(reply: string, expected: string): boolea return compactExpected.length > 0 && compactAgentReply(reply).includes(compactExpected); } +/** Recognize transport/provider failures without retrying a successful product response. */ +export function isHermesTransientAgentFailure(httpStatus: string, output: string): boolean { + if (httpStatus === "200") return false; + return ( + /^(408|429|5[0-9]{2})$/u.test(httpStatus) || + /ECONNREFUSED|EAI_AGAIN|ECONNRESET|ETIMEDOUT|gateway unavailable|network connection error|DNS error|fetch failed|inference service unavailable/iu.test( + output, + ) + ); +} + export function classifyPreContractProviderValidationSkip( result: Pick, ): CommonEgressProviderValidationSkip { diff --git a/test/e2e/live/common-egress-agent.test.ts b/test/e2e/live/common-egress-agent.test.ts index 122da41be39..63eee5dabaa 100644 --- a/test/e2e/live/common-egress-agent.test.ts +++ b/test/e2e/live/common-egress-agent.test.ts @@ -30,6 +30,7 @@ import { agentReplyContainsToken, classifyPreContractProviderValidationSkip, COMMON_EGRESS_TEST_TIMEOUT_MS, + isHermesTransientAgentFailure, parseChatContent, parseOpenClawAgentText, } from "./common-egress-agent-helpers.ts"; @@ -495,11 +496,12 @@ async function runOpenClawAgentAssertion( )}' stderr='${agent.stderr.slice(0, 240)}'`; if (attempt < OPENCLAW_AGENT_ATTEMPTS && isOpenClawScopeUpgradePending(combined)) { - await host.command("node", [CLI_ENTRYPOINT, args.sandboxName, "recover"], { + const recover = await host.command("node", [CLI_ENTRYPOINT, args.sandboxName, "recover"], { artifactName: `${args.label}-recover-after-attempt-${attempt}`, env: commandEnv(), timeoutMs: 120_000, }); + if (recover.exitCode !== 0) break; await sleep(attempt * 15_000); continue; } @@ -509,7 +511,7 @@ async function runOpenClawAgentAssertion( continue; } - if (attempt < OPENCLAW_AGENT_ATTEMPTS) await sleep(5_000); + break; } throw new Error(`${args.label}: expected ${args.expected}, got ${lastFailure}`); @@ -588,7 +590,11 @@ async function runHermesAgentAssertion( 0, 240, )}' body='${body.slice(0, 240)}'`; - if (attempt < HERMES_AGENT_ATTEMPTS) await sleep(5_000); + if (attempt < HERMES_AGENT_ATTEMPTS && isHermesTransientAgentFailure(httpStatus, response)) { + await sleep(attempt * 5_000); + continue; + } + break; } throw new Error(`${args.label}: expected ${args.expected}, got ${lastFailure}`); diff --git a/test/e2e/live/hermes-e2e.test.ts b/test/e2e/live/hermes-e2e.test.ts index 2c4e6a1102d..34449573f5b 100644 --- a/test/e2e/live/hermes-e2e.test.ts +++ b/test/e2e/live/hermes-e2e.test.ts @@ -128,7 +128,8 @@ function firstChoice(response: unknown): OpenAiChoiceLike | undefined { return choices.find((choice) => choice && typeof choice === "object"); } -function shouldRetryForReasoningBudget(response: unknown): boolean { +/** Report whether a completed response spent its full budget on reasoning. */ +function exhaustedReasoningBudget(response: unknown): boolean { const content = chatContent(response); if (/PONG/i.test(content)) return false; const choice = firstChoice(response); @@ -219,26 +220,6 @@ async function postDestroyGatewayBestEffort(run: () => Promise): Promis } } -async function retryHostedInference( - label: string, - run: (attempt: number) => Promise, -): Promise { - let lastError: unknown; - for (let attempt = 1; attempt <= 3; attempt += 1) { - try { - return await run(attempt); - } catch (error) { - lastError = error; - if (attempt < 3) await sleep(5_000 * attempt); - } - } - throw new Error( - `${label} failed after retries: ${ - lastError instanceof Error ? lastError.message : String(lastError) - }`, - ); -} - // source-shape-contract: security -- Live execution proves the shipped Hermes manifest remains healthy and credential-safe test("hermes-e2e: install.sh onboards Hermes and proves health plus live inference", { timeout: HERMES_E2E_TEST_TIMEOUT_MS, @@ -1270,64 +1251,35 @@ test("hermes-e2e: install.sh onboards Hermes and proves health plus live inferen progress.phase("exercise hosted and inference.local routes"); // Phase 6: live inference through both the external provider and the // sandbox's inference.local route. - const directChat = await retryHostedInference( - `${inference.mode} direct chat`, - async (attempt) => { - const response = await inference.directChat("Reply with exactly one word: PONG", { - artifactName: `phase-6-direct-inference-chat-attempt-${attempt}`, - maxTokens: attempt === 1 ? 256 : 1024, - }); - if (shouldRetryForReasoningBudget(response)) { - throw new Error("direct chat exhausted response budget while reasoning before PONG"); - } - return response; - }, - ); + const directChat = await inference.directChat("Reply with exactly one word: PONG", { + artifactName: "phase-6-direct-inference-chat", + maxTokens: 1024, + }); + expect(exhaustedReasoningBudget(directChat)).toBe(false); expectPong(`${inference.mode} direct chat`, directChat); - const sandboxChatJson = await retryHostedInference( - "Hermes sandbox inference.local chat", - async (attempt) => { - const result = await sandbox.exec( - SANDBOX_NAME, - [ - "curl", - "-fsS", - "--max-time", - "90", - "-H", - "Content-Type: application/json", - "--data-raw", - chatPayload( - inference.model, - "Reply with exactly one word: PONG", - attempt === 1 ? 256 : 1024, - ), - "https://inference.local/v1/chat/completions", - ], - { - artifactName: `phase-6-inference-local-chat-attempt-${attempt}`, - env: commandEnv(), - timeoutMs: 120_000, - }, - ); - if (result.exitCode !== 0) throw new Error(resultText(result)); - let parsed: unknown; - try { - parsed = JSON.parse(result.stdout) as unknown; - } catch (error) { - throw new Error( - `Hermes sandbox inference.local chat response was not JSON: ${ - error instanceof Error ? error.message : String(error) - }; body=${result.stdout.slice(0, 500)}`, - ); - } - if (shouldRetryForReasoningBudget(parsed)) { - throw new Error("sandbox chat exhausted response budget while reasoning before PONG"); - } - return parsed; + const sandboxChat = await sandbox.exec( + SANDBOX_NAME, + [ + "curl", + "-fsS", + "--max-time", + "90", + "-H", + "Content-Type: application/json", + "--data-raw", + chatPayload(inference.model, "Reply with exactly one word: PONG", 1024), + "https://inference.local/v1/chat/completions", + ], + { + artifactName: "phase-6-inference-local-chat", + env: commandEnv(), + timeoutMs: 120_000, }, ); + expect(sandboxChat.exitCode, resultText(sandboxChat)).toBe(0); + const sandboxChatJson = JSON.parse(sandboxChat.stdout) as unknown; + expect(exhaustedReasoningBudget(sandboxChatJson)).toBe(false); expectPong("Hermes sandbox inference.local chat", sandboxChatJson); progress.phase("validate CLI manifest and locked-config behavior"); diff --git a/test/e2e/support/cloud-inference-provider-skip.test.ts b/test/e2e/support/cloud-inference-provider-skip.test.ts index dab649a8543..221b394ce61 100644 --- a/test/e2e/support/cloud-inference-provider-skip.test.ts +++ b/test/e2e/support/cloud-inference-provider-skip.test.ts @@ -4,7 +4,10 @@ import { describe, expect, it } from "vitest"; import { buildPreContractExternalProviderSkipEvidence, + classifyCloudChatFailure, classifyPreContractExternalProviderFailure, + cloudChatWriteOutArg, + parseCloudChatResponse, PRE_CONTRACT_EXTERNAL_PROVIDER_REMOVAL_CONDITION, PRE_CONTRACT_EXTERNAL_PROVIDER_SKIP_REASON, PRE_CONTRACT_EXTERNAL_PROVIDER_SOURCE_BOUNDARY, @@ -15,6 +18,33 @@ function probeOutput(output: string): { stdout: string; stderr: string } { } describe("cloud inference pre-contract provider skip classifier", () => { + it("classifies chat retries independently of curl exit status", () => { + expect(classifyCloudChatFailure("503", "", "expected PONG", undefined)).toBe( + "transient-external", + ); + expect(classifyCloudChatFailure("", "", "", new Error("request ETIMEDOUT"))).toBe( + "transient-external", + ); + expect(classifyCloudChatFailure("", "", "", new Error("request timeout"))).toBe( + "transient-external", + ); + expect(classifyCloudChatFailure("200", "", "response was not parseable JSON", undefined)).toBe( + "malformed-input", + ); + expect(classifyCloudChatFailure("200", "", "expected PONG", undefined)).toBe("deterministic"); + expect( + classifyCloudChatFailure("200", "", "body said rate limit and HTTP 503", undefined), + ).toBe("deterministic"); + }); + + it("separates the curl status trailer from provider content", () => { + const output = `{"message":"rate limit and HTTP 503"}${cloudChatWriteOutArg().replace("%{http_code}", "200")}`; + expect(parseCloudChatResponse(output)).toEqual({ + body: '{"message":"rate limit and HTTP 503"}', + httpStatus: "200", + }); + }); + it("skips only endpoint validation failures with transient provider evidence", () => { expect( classifyPreContractExternalProviderFailure( diff --git a/test/e2e/support/common-egress-agent-helpers.test.ts b/test/e2e/support/common-egress-agent-helpers.test.ts index 4f33e6141af..560359e3734 100644 --- a/test/e2e/support/common-egress-agent-helpers.test.ts +++ b/test/e2e/support/common-egress-agent-helpers.test.ts @@ -6,6 +6,7 @@ import { describe, expect, it } from "vitest"; import { agentReplyContainsToken, classifyPreContractProviderValidationSkip, + isHermesTransientAgentFailure, parseChatContent, parseOpenClawAgentText, } from "../live/common-egress-agent-helpers.ts"; @@ -46,6 +47,14 @@ describe("common-egress agent parsing and classification helpers", () => { ).toBe(true); }); + it("retries Hermes agent turns only for explicit transient failures", () => { + expect(isHermesTransientAgentFailure("503", "service unavailable")).toBe(true); + expect(isHermesTransientAgentFailure("000", "request failed: ECONNRESET")).toBe(true); + expect(isHermesTransientAgentFailure("401", "unauthorized")).toBe(false); + expect(isHermesTransientAgentFailure("200", "wrong deterministic answer")).toBe(false); + expect(isHermesTransientAgentFailure("200", "reply mentions fetch failed")).toBe(false); + }); + it("classifies pre-contract provider validation skips", () => { expect( classifyPreContractProviderValidationSkip({ diff --git a/test/e2e/support/hermes-inference-switch-command-shape.test.ts b/test/e2e/support/hermes-inference-switch-command-shape.test.ts index cbae3f7f3d9..af2d2011d21 100644 --- a/test/e2e/support/hermes-inference-switch-command-shape.test.ts +++ b/test/e2e/support/hermes-inference-switch-command-shape.test.ts @@ -370,15 +370,12 @@ describe("Hermes inference switch command shape", () => { }); }); - it("falls back to no-verify only after transient route verification fails", async () => { - const command = vi - .fn() - .mockResolvedValueOnce({ - exitCode: 1, - stderr: "failed to verify inference endpoint: failed to connect", - stdout: "", - }) - .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "" }); + it("keeps a transient route verification exhaustion failed", async () => { + const command = vi.fn().mockResolvedValueOnce({ + exitCode: 1, + stderr: "failed to verify inference endpoint: failed to connect", + stdout: "", + }); const compatibleBinding = compatibleAnthropicSwitchBinding( "http://host.openshell.internal:18766/v1", { COMPATIBLE_ANTHROPIC_API_KEY: "switch-key" }, @@ -391,10 +388,10 @@ describe("Hermes inference switch command shape", () => { compatibleAnthropicMetadataArgs(compatibleBinding.endpointUrl), { attempts: 1, compatibleBinding, delay: async () => {} }, ), - ).resolves.toMatchObject({ exitCode: 0 }); + ).resolves.toMatchObject({ exitCode: 1 }); expect(command.mock.calls[0]?.[1]).not.toContain("--no-verify"); - expect(command.mock.calls[1]?.[1]).toContain("--no-verify"); + expect(command).toHaveBeenCalledOnce(); expect(command.mock.calls[0]?.[2]).toMatchObject({ env: { COMPATIBLE_ANTHROPIC_API_KEY: "switch-key" }, redactionValues: ["hosted-key", "switch-key"], diff --git a/test/e2e/support/inference-switch-retry.test.ts b/test/e2e/support/inference-switch-retry.test.ts index e842cdf2628..24ade16a8ed 100644 --- a/test/e2e/support/inference-switch-retry.test.ts +++ b/test/e2e/support/inference-switch-retry.test.ts @@ -1,6 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + import { describe, expect, it, vi } from "vitest"; import { @@ -46,24 +51,78 @@ describe("inference switch retry", () => { expect(delay).toHaveBeenCalledWith(5_000); }); - it("uses no-verify only after the transient verification budget is exhausted", async () => { - const transient = result(1, "failed to connect to endpoint"); + it("retains degraded evidence when a transient verification passes after retry", async () => { const run = vi .fn() - .mockResolvedValueOnce(transient) - .mockResolvedValueOnce(transient) + .mockResolvedValueOnce(result(1, "failed to verify inference endpoint: timeout")) .mockResolvedValueOnce(result(0)); + const evidence = vi.fn(); + + await runInferenceSetWithRetry({ + attempts: 2, + delay: async () => {}, + run, + onEvidence: evidence, + }); + + expect(evidence).toHaveBeenCalledWith( + expect.objectContaining({ + outcome: "passed-after-retry", + attempts: [ + expect.objectContaining({ failureClass: "transient-external", retryScheduled: true }), + expect.objectContaining({ outcome: "passed", retryScheduled: false }), + ], + }), + ); + }); + + it("keeps exhausted verified attempts failed without bypassing verification", async () => { + const transient = result(1, "failed to connect to endpoint"); + const run = vi.fn().mockResolvedValueOnce(transient).mockResolvedValueOnce(transient); await expect( runInferenceSetWithRetry({ attempts: 2, delay: async () => {}, run }), - ).resolves.toMatchObject({ exitCode: 0 }); + ).resolves.toMatchObject({ exitCode: 1 }); expect(run.mock.calls).toEqual([ [1, true], [2, true], - [2, false], ]); }); + it("keeps the shell helper failed on exhaustion without adding a verification bypass", () => { + const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-inference-retry-")); + const invocationLog = path.join(tempDir, "invocations.log"); + const helper = path.resolve("test/e2e/lib/inference-switch-retry.sh"); + const harness = String.raw` +source "$1" +sleep() { :; } +fake_inference_set() { + printf '%s\n' "$*" >> "$INVOCATION_LOG" + printf 'failed to verify inference endpoint: timeout\n' >&2 + return 17 +} +rc=0 +NEMOCLAW_SWITCH_SET_ATTEMPTS=2 run_inference_set_with_retry fake_inference_set provider set --model target || rc=$? +printf 'terminal_rc=%s\n' "$rc" +`; + + try { + const result = spawnSync("bash", ["-s", "--", helper], { + encoding: "utf8", + env: { ...process.env, INVOCATION_LOG: invocationLog }, + input: harness, + }); + const invocations = fs.readFileSync(invocationLog, "utf8").trim().split("\n"); + + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toContain("terminal_rc=17"); + expect(invocations).toEqual(["provider set --model target", "provider set --model target"]); + expect(invocations.join(" ")).not.toContain("--no-verify"); + } finally { + fs.rmSync(tempDir, { force: true, recursive: true }); + } + }); + it("does not bypass non-transient verification failures", async () => { const run = vi.fn().mockResolvedValue(result(1, "invalid provider")); @@ -74,9 +133,19 @@ describe("inference switch retry", () => { expect(run).toHaveBeenCalledWith(1, true); }); + it("does not retry a deterministic verification mismatch", async () => { + const run = vi + .fn() + .mockResolvedValue(result(1, "failed to verify inference endpoint: model mismatch")); + + await runInferenceSetWithRetry({ attempts: 3, delay: async () => {}, run }); + expect(run).toHaveBeenCalledOnce(); + }); + it("validates the configured attempt count", () => { expect(inferenceSetAttemptCount(undefined)).toBe(3); expect(inferenceSetAttemptCount("2")).toBe(2); - expect(() => inferenceSetAttemptCount("0")).toThrow(/positive integer/u); + expect(() => inferenceSetAttemptCount("0")).toThrow(/between 1 and 10/u); + expect(() => inferenceSetAttemptCount("11")).toThrow(/between 1 and 10/u); }); }); diff --git a/test/e2e/support/main-run-retry.test.ts b/test/e2e/support/main-run-retry.test.ts index 76ab324830d..410d013f265 100644 --- a/test/e2e/support/main-run-retry.test.ts +++ b/test/e2e/support/main-run-retry.test.ts @@ -48,12 +48,10 @@ function job(attempt: number, conclusion = "success") { function setup(options: { attempt: number; conclusion: string; latestRunId?: number }) { const requests: Array<{ method: string; path: string }> = []; const run = sourceRun(options.attempt, options.conclusion); - const request = async (path: string, init?: { method?: "GET" | "POST" }) => { + const request = async (path: string, init?: { method?: "GET" }) => { const method = init?.method ?? "GET"; requests.push({ method, path }); switch (true) { - case method === "POST": - return undefined; case path.endsWith(`/actions/runs/${RUN_ID}`): return run; case path.includes(`/actions/workflows/${WORKFLOW_ID}/runs?`): @@ -92,33 +90,21 @@ async function evaluate(options: { attempt: number; conclusion: string; latestRu } describe("main E2E retry controller", () => { - it.each([1, 2])("requests failed-job rerun after attempt %s fails", async (attempt) => { - const { evidence, requests } = await evaluate({ attempt, conclusion: "failure" }); - - expect(E2E_MAX_RETRIES).toBe(2); - expect(E2E_MAX_ATTEMPTS).toBe(3); - expect(evidence).toMatchObject({ - action: "retry-requested", - flaky: false, - sourceAttempt: attempt, - }); - expect(requests).toContainEqual({ - method: "POST", - path: `repos/${REPOSITORY}/actions/runs/${RUN_ID}/rerun-failed-jobs`, - }); - expect( - requests.some( - (request) => request.path === `repos/${REPOSITORY}/actions/runs/${RUN_ID}/rerun`, - ), - ).toBe(false); - }); - - it("stops after the third failed attempt", async () => { - const { evidence, requests } = await evaluate({ attempt: 3, conclusion: "failure" }); - - expect(evidence.action).toBe("failed-after-retries"); - expect(requests.some((request) => request.method === "POST")).toBe(false); - }); + it.each([1, 2, 3])( + "retains failed attempt %s without requesting a broad rerun", + async (attempt) => { + const { evidence, requests } = await evaluate({ attempt, conclusion: "failure" }); + + expect(E2E_MAX_RETRIES).toBe(0); + expect(E2E_MAX_ATTEMPTS).toBe(3); + expect(evidence).toMatchObject({ + action: "failed-no-retry", + flaky: false, + sourceAttempt: attempt, + }); + expect(requests.some((request) => request.method === "POST")).toBe(false); + }, + ); it("reports a successful retry as flaky and sums runner minutes across attempts", async () => { const { evidence, requests } = await evaluate({ attempt: 2, conclusion: "success" }); @@ -166,7 +152,7 @@ describe("main E2E retry controller", () => { it("rejects manual workflow runs", async () => { const fixture = setup({ attempt: 1, conclusion: "failure" }); const originalRequest = fixture.request; - const request = async (path: string, init?: { method?: "GET" | "POST" }) => { + const request = async (path: string, init?: { method?: "GET" }) => { const response = await originalRequest(path, init); return path.endsWith(`/actions/runs/${RUN_ID}`) ? { ...(response as Record), event: "workflow_dispatch" } @@ -185,7 +171,7 @@ describe("main E2E retry controller", () => { }); it("rejects a truncated attempt job listing", async () => { const fixture = setup({ attempt: 1, conclusion: "failure" }); - const request = async (path: string, init?: { method?: "GET" | "POST" }) => + const request = async (path: string, init?: { method?: "GET" }) => path.includes("/attempts/1/jobs") ? { total_count: 2, jobs: [job(1, "failure")] } : fixture.request(path, init); @@ -203,7 +189,7 @@ describe("main E2E retry controller", () => { it("rejects an unbounded job name before evidence serialization", async () => { const fixture = setup({ attempt: 1, conclusion: "failure" }); - const request = async (path: string, init?: { method?: "GET" | "POST" }) => + const request = async (path: string, init?: { method?: "GET" }) => path.includes("/attempts/1/jobs") ? { total_count: 1, jobs: [{ ...job(1, "failure"), name: "x".repeat(257) }] } : fixture.request(path, init); @@ -221,7 +207,7 @@ describe("main E2E retry controller", () => { it("rejects an empty successful attempt job listing", async () => { const fixture = setup({ attempt: 1, conclusion: "success" }); - const request = async (path: string, init?: { method?: "GET" | "POST" }) => + const request = async (path: string, init?: { method?: "GET" }) => path.includes("/attempts/1/jobs") ? { total_count: 0, jobs: [] } : fixture.request(path, init); diff --git a/test/e2e/support/retry-policy.test.ts b/test/e2e/support/retry-policy.test.ts new file mode 100644 index 00000000000..a9122b510f5 --- /dev/null +++ b/test/e2e/support/retry-policy.test.ts @@ -0,0 +1,197 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; + +import { RetryPolicyError, runBoundedRetry } from "../fixtures/retry-policy.ts"; + +describe("bounded E2E operation retry policy", () => { + it("reports a first-attempt pass", async () => { + const result = await runBoundedRetry({ + operation: "provider.probe", + owner: "external-provider", + idempotence: "read-only", + maxAttempts: 3, + run: async () => "ok", + classify: (value) => + value === "ok" + ? { outcome: "passed" } + : { outcome: "failed", failureClass: "deterministic" }, + }); + + expect(result.evidence).toMatchObject({ + outcome: "passed-first-attempt", + attempts: [{ attempt: 1, outcome: "passed" }], + }); + }); + + it("records transient recovery without hiding the first failure", async () => { + const run = vi.fn().mockResolvedValueOnce("timeout").mockResolvedValueOnce("ok"); + const evidence: unknown[] = []; + const result = await runBoundedRetry({ + operation: "provider.probe", + owner: "external-provider", + idempotence: "read-only", + maxAttempts: 3, + run, + classify: (value) => + value === "ok" + ? { outcome: "passed" } + : { outcome: "failed", failureClass: "transient-external" }, + onEvidence: (record) => { + evidence.push(record); + }, + }); + + expect(result.evidence.outcome).toBe("passed-after-retry"); + expect(result.evidence.attempts).toEqual([ + { attempt: 1, outcome: "failed", failureClass: "transient-external", retryScheduled: true }, + { attempt: 2, outcome: "passed", retryScheduled: false }, + ]); + expect(evidence).toEqual([result.evidence]); + }); + + it.each([ + "authentication", + "authorization", + "policy-denial", + "malformed-input", + "deterministic", + ] as const)("does not retry %s failures", async (failureClass) => { + const run = vi.fn().mockResolvedValue("failed"); + const result = await runBoundedRetry({ + operation: "provider.probe", + owner: "nemoclaw", + idempotence: "read-only", + maxAttempts: 3, + run, + classify: () => ({ outcome: "failed", failureClass }), + }); + + expect(result.evidence.outcome).toBe("failed-no-retry"); + expect(run).toHaveBeenCalledOnce(); + }); + + it("keeps an exhausted transient retry failed with complete history", async () => { + const result = await runBoundedRetry({ + operation: "provider.probe", + owner: "external-provider", + idempotence: "read-only", + maxAttempts: 2, + run: async () => "timeout", + classify: () => ({ outcome: "failed", failureClass: "transient-external" }), + }); + + expect(result.evidence.outcome).toBe("exhausted"); + expect(result.evidence.attempts).toHaveLength(2); + expect(result.evidence.attempts[1]?.retryScheduled).toBe(false); + }); + + it("uses the computed bounded delay between transient attempts", async () => { + const sleep = vi.fn().mockResolvedValue(undefined); + await runBoundedRetry({ + operation: "provider.probe", + owner: "external-provider", + idempotence: "read-only", + maxAttempts: 3, + delayMs: (attempt) => attempt * 1_000, + sleep, + run: async () => "timeout", + classify: () => ({ outcome: "failed", failureClass: "transient-external" }), + }); + + expect(sleep.mock.calls).toEqual([[1_000], [2_000]]); + }); + + it("rejects a retry delay outside the bounded range", async () => { + await expect( + runBoundedRetry({ + operation: "provider.probe", + owner: "external-provider", + idempotence: "read-only", + maxAttempts: 2, + delayMs: 300_001, + sleep: async () => {}, + run: async () => "timeout", + classify: () => ({ outcome: "failed", failureClass: "transient-external" }), + }), + ).rejects.toThrow("between 0 and 300000"); + }); + + it("requires reconciliation before retrying an ambiguous mutation", async () => { + const run = vi.fn().mockResolvedValue("timeout"); + const result = await runBoundedRetry({ + operation: "sandbox.mutate", + owner: "nemoclaw", + idempotence: "reconciled-mutation", + maxAttempts: 3, + run, + classify: () => ({ outcome: "failed", failureClass: "transient-external" }), + reconcile: async () => false, + }); + + expect(result.evidence.outcome).toBe("failed-no-retry"); + expect(result.evidence.attempts[0]?.reconciled).toBe(false); + expect(run).toHaveBeenCalledOnce(); + }); + + it("retries a mutation only after reconciliation authorizes it", async () => { + const run = vi.fn().mockResolvedValueOnce("timeout").mockResolvedValueOnce("ok"); + const reconcile = vi.fn().mockResolvedValue(true); + const result = await runBoundedRetry({ + operation: "sandbox.mutate", + owner: "nemoclaw", + idempotence: "reconciled-mutation", + maxAttempts: 2, + run, + classify: (value) => + value === "ok" + ? { outcome: "passed" } + : { outcome: "failed", failureClass: "transient-external" }, + reconcile, + }); + + expect(result.evidence.outcome).toBe("passed-after-retry"); + expect(result.evidence.attempts[0]?.reconciled).toBe(true); + expect(reconcile).toHaveBeenCalledOnce(); + }); + + it("keeps cleanup failures visible and excludes exception text from evidence", async () => { + const secret = "nvapi-secret-value"; + let caught: RetryPolicyError | undefined; + try { + await runBoundedRetry({ + operation: "sandbox.cleanup", + owner: "nemoclaw", + idempotence: "idempotent", + maxAttempts: 3, + run: async () => { + throw new Error(`cleanup failed with ${secret}`); + }, + classify: () => ({ outcome: "failed", failureClass: "cleanup" }), + }); + } catch (error) { + caught = error as RetryPolicyError; + } + + expect(caught).toBeInstanceOf(RetryPolicyError); + expect(caught?.evidence.outcome).toBe("failed-no-retry"); + expect(String(caught)).not.toContain(secret); + expect(JSON.stringify(caught?.evidence)).not.toContain(secret); + }); + + it("rejects malformed policy bounds before running", async () => { + const run = vi.fn(); + await expect( + runBoundedRetry({ + operation: "provider.probe", + owner: "nemoclaw", + idempotence: "read-only", + maxAttempts: 0, + run, + classify: () => ({ outcome: "passed" }), + }), + ).rejects.toThrow("between 1 and 10"); + expect(run).not.toHaveBeenCalled(); + }); +}); diff --git a/tools/e2e/main-run-retry.mts b/tools/e2e/main-run-retry.mts index f252f3f3efd..bf5c336ef6a 100755 --- a/tools/e2e/main-run-retry.mts +++ b/tools/e2e/main-run-retry.mts @@ -13,17 +13,19 @@ const WORKFLOW_PATH = ".github/workflows/e2e.yaml"; const DISPLAY_TITLE = "E2E main"; const SHA_PATTERN = /^[a-f0-9]{40}$/u; const TIMESTAMP_PATTERN = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/u; -export const E2E_MAX_RETRIES = 2; -export const E2E_MAX_ATTEMPTS = E2E_MAX_RETRIES + 1; +// Broad failed-job reruns are not retry evidence: they can replay deterministic +// product, auth, policy, malformed-input, and cleanup failures. Keep observing +// manual attempts for history, but authorize no automatic workflow reruns. +export const E2E_MAX_RETRIES = 0; +export const E2E_MAX_ATTEMPTS = 3; export type MainRunRetryAction = - | "failed-after-retries" + | "failed-no-retry" | "ignored" | "passed-after-retry" - | "passed-first-attempt" - | "retry-requested"; + | "passed-first-attempt"; -type ApiRequest = (path: string, options?: { method?: "GET" | "POST" }) => Promise; +type ApiRequest = (path: string, options?: { method?: "GET" }) => Promise; type SourceRun = { id: number; @@ -120,7 +122,8 @@ function validateSourceRun(value: unknown): SourceRun { function validateLatestRun(value: unknown, source: SourceRun): boolean { const response = record(value); - if (!Array.isArray(response.workflow_runs)) throw new Error("GitHub returned no workflow run list"); + if (!Array.isArray(response.workflow_runs)) + throw new Error("GitHub returned no workflow run list"); const eligible = response.workflow_runs .map((item) => record(item)) .find( @@ -200,7 +203,8 @@ function validateAttemptEvidence(value: unknown, attempt: number): AttemptEviden const active = jobs.filter((job) => job.conclusion !== "skipped"); const runnerMilliseconds = active.reduce((total, job) => { const duration = Date.parse(job.completedAt!) - Date.parse(job.startedAt!); - if (!Number.isFinite(duration) || duration < 0) throw new Error("GitHub returned invalid job timing"); + if (!Number.isFinite(duration) || duration < 0) + throw new Error("GitHub returned invalid job timing"); return total + duration; }, 0); return { @@ -214,7 +218,10 @@ function validateAttemptEvidence(value: unknown, attempt: number): AttemptEviden }; } -export function decideMainRunRetry(source: SourceRun): { action: MainRunRetryAction; reason: string } { +export function decideMainRunRetry(source: SourceRun): { + action: MainRunRetryAction; + reason: string; +} { if (source.conclusion === "success") { return source.attempt === 1 ? { action: "passed-first-attempt", reason: "E2E passed on its first attempt" } @@ -223,9 +230,10 @@ export function decideMainRunRetry(source: SourceRun): { action: MainRunRetryAct if (source.conclusion !== "failure") { return { action: "ignored", reason: `E2E concluded with ${source.conclusion}` }; } - return source.attempt < E2E_MAX_ATTEMPTS - ? { action: "retry-requested", reason: `E2E failed on attempt ${source.attempt}` } - : { action: "failed-after-retries", reason: "E2E failed on its third attempt" }; + return { + action: "failed-no-retry", + reason: "E2E failed; retry requires operation-level transient evidence", + }; } export async function evaluateMainRunRetry(options: { @@ -268,16 +276,6 @@ export async function evaluateMainRunRetry(options: { const decision = isLatest ? decideMainRunRetry(source) : { action: "ignored" as const, reason: "a newer E2E main push exists" }; - if (decision.action === "retry-requested") { - const confirmed = validateSourceRun(await request(runPath)); - if (JSON.stringify(confirmed) !== JSON.stringify(source)) { - throw new Error("source run changed before retry request"); - } - if (!validateLatestRun(await request(latestPath), source)) { - throw new Error("a newer E2E main push appeared before retry request"); - } - await request(`${runPath}/rerun-failed-jobs`, { method: "POST" }); - } return { schemaVersion: 1, sourceRunId: source.id, @@ -331,7 +329,6 @@ function writeRetryEvidence(file: string, evidence: MainRunRetryEvidence): void } } - async function main(): Promise { const evidence = await evaluateMainRunRetry({ repository: requiredEnvironment("GITHUB_REPOSITORY"), From 35c06130f88d83c70992b7b64f2fce73902bf3b9 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 14:33:10 -0700 Subject: [PATCH 02/23] test(e2e): close retry evidence gaps Signed-off-by: Carlos Villela --- test/e2e/live/agent-turn-latency-helpers.ts | 20 +++++++++++++++---- .../e2e/live/cloud-inference-provider-skip.ts | 2 ++ test/e2e/live/cloud-inference.test.ts | 13 ++++++++++-- .../agent-turn-latency-progress.test.ts | 12 +++++++++++ .../cloud-inference-provider-skip.test.ts | 1 + .../support/inference-switch-retry.test.ts | 3 +-- 6 files changed, 43 insertions(+), 8 deletions(-) diff --git a/test/e2e/live/agent-turn-latency-helpers.ts b/test/e2e/live/agent-turn-latency-helpers.ts index a69c7a140b7..52195b84880 100644 --- a/test/e2e/live/agent-turn-latency-helpers.ts +++ b/test/e2e/live/agent-turn-latency-helpers.ts @@ -36,7 +36,9 @@ export const HERMES_SANDBOX = validateSandboxName(OPENCLAW_SANDBOX); validateSandboxName(HERMES_SANDBOX); export const MAX_TURN_SECONDS = positiveInt(process.env.NEMOCLAW_TURN_LATENCY_MAX_SECONDS, 300); -const INSTALL_ATTEMPTS = boundedAttempts(process.env.NEMOCLAW_TURN_LATENCY_INSTALL_ATTEMPTS, 2); +const INSTALL_ATTEMPTS = turnLatencyInstallAttemptCount( + process.env.NEMOCLAW_TURN_LATENCY_INSTALL_ATTEMPTS, +); const INSTALL_TIMEOUT_MS = 30 * 60_000; type AgentTurnProgress = Pick; @@ -45,9 +47,19 @@ function positiveInt(value: string | undefined, fallback: number): number { return value && /^[1-9][0-9]*$/u.test(value) ? Number.parseInt(value, 10) : fallback; } -function boundedAttempts(value: string | undefined, fallback: number): number { - const attempts = positiveInt(value, fallback); - if (attempts > 10) throw new Error("install attempts must be between 1 and 10"); +export function turnLatencyInstallAttemptCount(value: string | undefined): number { + if (value === undefined) return 2; + if (!/^[1-9][0-9]?$/u.test(value)) { + throw new Error( + `NEMOCLAW_TURN_LATENCY_INSTALL_ATTEMPTS must be an integer between 1 and 10; got ${value}`, + ); + } + const attempts = Number.parseInt(value, 10); + if (attempts > 10) { + throw new Error( + `NEMOCLAW_TURN_LATENCY_INSTALL_ATTEMPTS must be an integer between 1 and 10; got ${value}`, + ); + } return attempts; } diff --git a/test/e2e/live/cloud-inference-provider-skip.ts b/test/e2e/live/cloud-inference-provider-skip.ts index c7017d94286..3a265c57b64 100644 --- a/test/e2e/live/cloud-inference-provider-skip.ts +++ b/test/e2e/live/cloud-inference-provider-skip.ts @@ -49,7 +49,9 @@ export function classifyCloudChatFailure( transportOutput: string, failure: string, error: unknown, + timedOut = false, ): RetryFailureClass { + if (timedOut) return "transient-external"; const thrownDetail = error instanceof Error ? error.message : ""; if (httpStatus === "429" || /^5\d{2}$/u.test(httpStatus)) return "transient-external"; if (TRANSIENT_CHAT_FAILURE.test(`${transportOutput}\n${thrownDetail}`)) { diff --git a/test/e2e/live/cloud-inference.test.ts b/test/e2e/live/cloud-inference.test.ts index 64a5ccee850..5c593294baa 100644 --- a/test/e2e/live/cloud-inference.test.ts +++ b/test/e2e/live/cloud-inference.test.ts @@ -255,7 +255,13 @@ async function expectLiveChatPong( } return { outcome: "failed", - failureClass: classifyCloudChatFailure(httpStatus, response.stderr, failure, error), + failureClass: classifyCloudChatFailure( + httpStatus, + response.stderr, + failure, + error, + response.timedOut, + ), }; }, onEvidence: async (evidence) => { @@ -263,7 +269,10 @@ async function expectLiveChatPong( }, }); const value = execution.value; - if (value && /pong/iu.test(value.content)) { + const passed = + execution.evidence.outcome === "passed-first-attempt" || + execution.evidence.outcome === "passed-after-retry"; + if (passed && value) { return { attempt: execution.evidence.attempts.length, content: value.content }; } throw new Error( diff --git a/test/e2e/support/agent-turn-latency-progress.test.ts b/test/e2e/support/agent-turn-latency-progress.test.ts index d8647723ca5..c0fa950f89d 100644 --- a/test/e2e/support/agent-turn-latency-progress.test.ts +++ b/test/e2e/support/agent-turn-latency-progress.test.ts @@ -16,6 +16,7 @@ import { bestEffortPreclean, cleanupTurnSandboxes, installSandbox, + turnLatencyInstallAttemptCount, } from "../live/agent-turn-latency-helpers.ts"; function fakeInference(apiKey = "secret-api-key"): AgentTurnInference { @@ -88,6 +89,17 @@ describe("live test progress", () => { vi.useRealTimers(); }); + it("rejects invalid configured install attempt counts", () => { + expect(turnLatencyInstallAttemptCount(undefined)).toBe(2); + expect(turnLatencyInstallAttemptCount("1")).toBe(1); + expect(turnLatencyInstallAttemptCount("10")).toBe(10); + for (const value of ["0", "-1", "abc", "01", "11"]) { + expect(() => turnLatencyInstallAttemptCount(value)).toThrow( + /NEMOCLAW_TURN_LATENCY_INSTALL_ATTEMPTS must be an integer between 1 and 10/u, + ); + } + }); + it("reports semantic transitions and adds command-safe evidence only after a stall", () => { const { options, state } = progressHarness(); const progress = startTestProgress( diff --git a/test/e2e/support/cloud-inference-provider-skip.test.ts b/test/e2e/support/cloud-inference-provider-skip.test.ts index 221b394ce61..ef5a1f9ed67 100644 --- a/test/e2e/support/cloud-inference-provider-skip.test.ts +++ b/test/e2e/support/cloud-inference-provider-skip.test.ts @@ -28,6 +28,7 @@ describe("cloud inference pre-contract provider skip classifier", () => { expect(classifyCloudChatFailure("", "", "", new Error("request timeout"))).toBe( "transient-external", ); + expect(classifyCloudChatFailure("", "", "", undefined, true)).toBe("transient-external"); expect(classifyCloudChatFailure("200", "", "response was not parseable JSON", undefined)).toBe( "malformed-input", ); diff --git a/test/e2e/support/inference-switch-retry.test.ts b/test/e2e/support/inference-switch-retry.test.ts index 24ade16a8ed..213ec213f76 100644 --- a/test/e2e/support/inference-switch-retry.test.ts +++ b/test/e2e/support/inference-switch-retry.test.ts @@ -112,9 +112,8 @@ printf 'terminal_rc=%s\n' "$rc" env: { ...process.env, INVOCATION_LOG: invocationLog }, input: harness, }); - const invocations = fs.readFileSync(invocationLog, "utf8").trim().split("\n"); - expect(result.status, result.stderr).toBe(0); + const invocations = fs.readFileSync(invocationLog, "utf8").trim().split("\n"); expect(result.stdout).toContain("terminal_rc=17"); expect(invocations).toEqual(["provider set --model target", "provider set --model target"]); expect(invocations.join(" ")).not.toContain("--no-verify"); From 5e403038bb55ade81cc0c6ce61c520bf539d86e2 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 14:44:25 -0700 Subject: [PATCH 03/23] test(e2e): require retry result handling Signed-off-by: Carlos Villela --- test/e2e/fixtures/inference-switch-retry.ts | 4 +++- test/e2e/fixtures/retry-policy.ts | 16 ++++++++++------ test/e2e/live/cloud-inference.test.ts | 9 +++------ .../support/agent-turn-latency-progress.test.ts | 5 +++-- test/e2e/support/retry-policy.test.ts | 3 +++ 5 files changed, 22 insertions(+), 15 deletions(-) diff --git a/test/e2e/fixtures/inference-switch-retry.ts b/test/e2e/fixtures/inference-switch-retry.ts index c5d2772c9af..98070633201 100644 --- a/test/e2e/fixtures/inference-switch-retry.ts +++ b/test/e2e/fixtures/inference-switch-retry.ts @@ -49,5 +49,7 @@ export async function runInferenceSetWithRetry(options: { sleep: options.delay, onEvidence: options.onEvidence, }); - return execution.value!; + if (execution.outcome === "passed") return execution.value; + if (execution.value !== undefined) return execution.value; + throw new Error("inference switch retry failed without a command result"); } diff --git a/test/e2e/fixtures/retry-policy.ts b/test/e2e/fixtures/retry-policy.ts index 56c099cc413..ffdd51e1727 100644 --- a/test/e2e/fixtures/retry-policy.ts +++ b/test/e2e/fixtures/retry-policy.ts @@ -57,6 +57,10 @@ export interface BoundedRetryOptions { onEvidence?: (evidence: RetryEvidence) => Promise | void; } +export type BoundedRetryResult = + | { outcome: "passed"; value: T; evidence: RetryEvidence } + | { outcome: "failed"; value: T | undefined; evidence: RetryEvidence }; + /** Reject unbounded or artifact-unsafe retry metadata before an operation runs. */ function validateOptions(options: BoundedRetryOptions): void { if (!/^[a-z0-9][a-z0-9._-]{0,127}$/u.test(options.operation)) { @@ -110,13 +114,12 @@ async function emit( * Evidence deliberately contains no command output, exception text, or request * data, so credential-bearing values cannot enter retained retry artifacts. * - * A resolved operation returns its value with a terminal `evidence.outcome`, - * including failed outcomes. A thrown operation raises `RetryPolicyError` with - * the same evidence. Callers must inspect the outcome after a resolved promise. + * A resolved operation returns a discriminated pass or failure result. A + * thrown operation raises `RetryPolicyError` with the same evidence. */ export async function runBoundedRetry( options: BoundedRetryOptions, -): Promise<{ value: T | undefined; evidence: RetryEvidence }> { +): Promise> { validateOptions(options); const sleep = options.sleep ?? @@ -135,6 +138,7 @@ export async function runBoundedRetry( const classification = options.classify(value, error); if (classification.outcome === "passed") { if (error !== undefined) throw new Error("retry classifier reported success after an error"); + if (value === undefined) throw new Error("retry classifier reported success without a value"); attempts.push({ attempt, outcome: "passed", retryScheduled: false }); const evidence = finalEvidence( options, @@ -142,7 +146,7 @@ export async function runBoundedRetry( attempt === 1 ? "passed-first-attempt" : "passed-after-retry", ); await emit(options, evidence); - return { value, evidence }; + return { outcome: "passed", value, evidence }; } const isTransient = classification.failureClass === "transient-external"; @@ -178,7 +182,7 @@ export async function runBoundedRetry( if (error !== undefined) { throw new RetryPolicyError(`${options.operation} ${outcome}`, evidence); } - return { value, evidence }; + return { outcome: "failed", value, evidence }; } throw new Error("bounded retry loop completed without an attempt"); diff --git a/test/e2e/live/cloud-inference.test.ts b/test/e2e/live/cloud-inference.test.ts index 5c593294baa..fca3f231b0b 100644 --- a/test/e2e/live/cloud-inference.test.ts +++ b/test/e2e/live/cloud-inference.test.ts @@ -268,13 +268,10 @@ async function expectLiveChatPong( await artifacts.writeJson("phase-2-inference-local-chat-retry.json", evidence); }, }); - const value = execution.value; - const passed = - execution.evidence.outcome === "passed-first-attempt" || - execution.evidence.outcome === "passed-after-retry"; - if (passed && value) { - return { attempt: execution.evidence.attempts.length, content: value.content }; + if (execution.outcome === "passed") { + return { attempt: execution.evidence.attempts.length, content: execution.value.content }; } + const value = execution.value; throw new Error( `Live chat failed after ${execution.evidence.attempts.length} attempt(s): ${ value?.failure || `exit ${value?.response.exitCode ?? "unknown"}` diff --git a/test/e2e/support/agent-turn-latency-progress.test.ts b/test/e2e/support/agent-turn-latency-progress.test.ts index c0fa950f89d..0eb187d62c8 100644 --- a/test/e2e/support/agent-turn-latency-progress.test.ts +++ b/test/e2e/support/agent-turn-latency-progress.test.ts @@ -91,8 +91,9 @@ describe("live test progress", () => { it("rejects invalid configured install attempt counts", () => { expect(turnLatencyInstallAttemptCount(undefined)).toBe(2); - expect(turnLatencyInstallAttemptCount("1")).toBe(1); - expect(turnLatencyInstallAttemptCount("10")).toBe(10); + for (let expected = 1; expected <= 10; expected += 1) { + expect(turnLatencyInstallAttemptCount(String(expected))).toBe(expected); + } for (const value of ["0", "-1", "abc", "01", "11"]) { expect(() => turnLatencyInstallAttemptCount(value)).toThrow( /NEMOCLAW_TURN_LATENCY_INSTALL_ATTEMPTS must be an integer between 1 and 10/u, diff --git a/test/e2e/support/retry-policy.test.ts b/test/e2e/support/retry-policy.test.ts index a9122b510f5..a9a65e99f75 100644 --- a/test/e2e/support/retry-policy.test.ts +++ b/test/e2e/support/retry-policy.test.ts @@ -19,6 +19,7 @@ describe("bounded E2E operation retry policy", () => { : { outcome: "failed", failureClass: "deterministic" }, }); + expect(result.outcome).toBe("passed"); expect(result.evidence).toMatchObject({ outcome: "passed-first-attempt", attempts: [{ attempt: 1, outcome: "passed" }], @@ -68,6 +69,7 @@ describe("bounded E2E operation retry policy", () => { classify: () => ({ outcome: "failed", failureClass }), }); + expect(result.outcome).toBe("failed"); expect(result.evidence.outcome).toBe("failed-no-retry"); expect(run).toHaveBeenCalledOnce(); }); @@ -82,6 +84,7 @@ describe("bounded E2E operation retry policy", () => { classify: () => ({ outcome: "failed", failureClass: "transient-external" }), }); + expect(result.outcome).toBe("failed"); expect(result.evidence.outcome).toBe("exhausted"); expect(result.evidence.attempts).toHaveLength(2); expect(result.evidence.attempts[1]?.retryScheduled).toBe(false); From c1444e92ceed6e0edf8d4b1f0acd9e1fa7635e6f Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 15:29:18 -0700 Subject: [PATCH 04/23] test(e2e): verify mock provider switches --- .../fixtures/compatible-anthropic-switch.ts | 56 +++++++++++++++++ test/e2e/live/hermes-inference-switch.test.ts | 15 +++-- .../live/openclaw-inference-switch.test.ts | 12 ++-- .../compatible-anthropic-switch.test.ts | 62 ++++++++++++++++++- 4 files changed, 132 insertions(+), 13 deletions(-) diff --git a/test/e2e/fixtures/compatible-anthropic-switch.ts b/test/e2e/fixtures/compatible-anthropic-switch.ts index 6b51307486b..752a37be9e4 100644 --- a/test/e2e/fixtures/compatible-anthropic-switch.ts +++ b/test/e2e/fixtures/compatible-anthropic-switch.ts @@ -1,12 +1,17 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + import type { HostCliClient } from "./clients/host.ts"; import { resultText } from "./clients/index.ts"; export const COMPATIBLE_ANTHROPIC_PROVIDER = "compatible-anthropic-endpoint"; export const COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV = "COMPATIBLE_ANTHROPIC_API_KEY"; const DEFAULT_COMPATIBLE_ANTHROPIC_CREDENTIAL = "test-compatible-anthropic-key"; +const OPENSHELL_HOST_ALIAS = "host.openshell.internal"; export interface CompatibleAnthropicSwitchBinding { endpointUrl: string; @@ -39,6 +44,57 @@ export function compatibleAnthropicSwitchEnv( return binding ? { [COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV]: binding.credentialValue } : {}; } +export function hostVerificationHostsFile(source: string): string { + const existing = source.endsWith("\n") ? source : `${source}\n`; + return `127.0.0.1 ${OPENSHELL_HOST_ALIAS}\n${existing}`; +} + +export async function withHostVerificationLoopbackAlias( + host: HostCliClient, + cleanup: { trackDisposable(name: string, run: () => Promise | void): void }, + run: () => Promise, +): Promise { + const fixtureDirectory = fs.mkdtempSync( + path.join(os.tmpdir(), "nemoclaw-compatible-endpoint-hosts-"), + ); + const originalHosts = fs.readFileSync("/etc/hosts", "utf8"); + const originalPath = path.join(fixtureDirectory, "hosts.original"); + const mappedPath = path.join(fixtureDirectory, "hosts.mapped"); + fs.writeFileSync(originalPath, originalHosts, { mode: 0o600 }); + fs.writeFileSync(mappedPath, hostVerificationHostsFile(originalHosts), { mode: 0o600 }); + + let restored = false; + const restore = async (): Promise => { + if (restored) return; + const result = await host.command("sudo", ["cp", "--", originalPath, "/etc/hosts"], { + artifactName: "restore-host-verifier-alias", + timeoutMs: 30_000, + }); + if (result.exitCode !== 0) { + throw new Error(`could not restore /etc/hosts: ${resultText(result)}`); + } + if (fs.readFileSync("/etc/hosts", "utf8") !== originalHosts) { + throw new Error("/etc/hosts differs after host verifier alias restoration"); + } + restored = true; + fs.rmSync(fixtureDirectory, { force: true, recursive: true }); + }; + cleanup.trackDisposable("restore the host verifier alias mapping", restore); + + try { + const mapped = await host.command("sudo", ["cp", "--", mappedPath, "/etc/hosts"], { + artifactName: "map-host-verifier-alias", + timeoutMs: 30_000, + }); + if (mapped.exitCode !== 0) { + throw new Error(`could not map the host verifier alias: ${resultText(mapped)}`); + } + return await run(); + } finally { + await restore(); + } +} + export async function requireCompatibleAnthropicProviderAbsent( host: HostCliClient, options: { diff --git a/test/e2e/live/hermes-inference-switch.test.ts b/test/e2e/live/hermes-inference-switch.test.ts index 461000706d9..7c22fe4f4e1 100644 --- a/test/e2e/live/hermes-inference-switch.test.ts +++ b/test/e2e/live/hermes-inference-switch.test.ts @@ -4,6 +4,7 @@ import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import { resultText } from "../fixtures/clients/index.ts"; import { trustedSandboxShellScript } from "../fixtures/clients/sandbox.ts"; +import { withHostVerificationLoopbackAlias } from "../fixtures/compatible-anthropic-switch.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { startFakeOpenAiCompatibleServer } from "../fixtures/fake-openai-compatible.ts"; import { DEFAULT_HOSTED_INFERENCE_BASE_URL } from "../fixtures/hosted-inference.ts"; @@ -190,12 +191,14 @@ test("Hermes inference set updates route/config and preserves live runtime", { progress.phase("switch Hermes inference provider"); const compatibleMetadataArgs = compatibleAnthropicMetadataArgs(switchEndpointUrl); - const switched = await runHermesInferenceSetWithRetry( - host, - redactionValues, - compatibleMetadataArgs, - { compatibleBinding: switchBinding }, - ); + const switchInference = () => + runHermesInferenceSetWithRetry(host, redactionValues, compatibleMetadataArgs, { + compatibleBinding: switchBinding, + }); + const switched = + switchBinding && mockAnthropicSwitchEnabled() + ? await withHostVerificationLoopbackAlias(host, cleanup, switchInference) + : await switchInference(); expect(switched.exitCode, resultText(switched)).toBe(0); expect(resultText(switched)).not.toContain("writing the in-sandbox config failed"); expect(resultText(switched)).toContain(`Inference route synced for '${SANDBOX_NAME}'`); diff --git a/test/e2e/live/openclaw-inference-switch.test.ts b/test/e2e/live/openclaw-inference-switch.test.ts index 079f0b1e9f4..191a7a219d9 100644 --- a/test/e2e/live/openclaw-inference-switch.test.ts +++ b/test/e2e/live/openclaw-inference-switch.test.ts @@ -29,6 +29,7 @@ import { compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, requireCompatibleAnthropicProviderAbsent, + withHostVerificationLoopbackAlias, } from "../fixtures/compatible-anthropic-switch.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { @@ -1083,12 +1084,11 @@ test("openclaw-inference-switch: switches route and preserves live OpenClaw beha gatewayRestartExpected ? "anthropic-messages" : "openai-completions", ); const pidBefore = await openclawGatewayPid(sandbox, home); - const switchResult = await runOpenClawInferenceSetWithRetry( - host, - home, - redactionValues, - switchBinding, - ); + const switchInference = () => + runOpenClawInferenceSetWithRetry(host, home, redactionValues, switchBinding); + const switchResult = mockProvider + ? await withHostVerificationLoopbackAlias(host, cleanup, switchInference) + : await switchInference(); expect(switchResult.exitCode, resultText(switchResult)).toBe(0); expect( resultText(switchResult).includes( diff --git a/test/e2e/support/compatible-anthropic-switch.test.ts b/test/e2e/support/compatible-anthropic-switch.test.ts index c1378b7d41a..674c5417bc5 100644 --- a/test/e2e/support/compatible-anthropic-switch.test.ts +++ b/test/e2e/support/compatible-anthropic-switch.test.ts @@ -1,7 +1,9 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it, vi } from "vitest"; +import fs from "node:fs"; + +import { afterEach, describe, expect, it, vi } from "vitest"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import { @@ -9,10 +11,14 @@ import { COMPATIBLE_ANTHROPIC_PROVIDER, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, + hostVerificationHostsFile, requireCompatibleAnthropicProviderAbsent, + withHostVerificationLoopbackAlias, } from "../fixtures/compatible-anthropic-switch.ts"; describe("compatible Anthropic inference switch setup", () => { + afterEach(() => vi.restoreAllMocks()); + it("passes the direct binding credential only to the inference-set command", () => { const binding = compatibleAnthropicSwitchBinding("http://host.openshell.internal:18766", { COMPATIBLE_ANTHROPIC_API_KEY: "fixture-key", @@ -44,6 +50,60 @@ describe("compatible Anthropic inference switch setup", () => { ).toThrow("COMPATIBLE_ANTHROPIC_API_KEY is required"); }); + it("maps the sandbox host alias to loopback for host-side verification", () => { + expect(hostVerificationHostsFile("127.0.0.1 localhost\n")).toBe( + "127.0.0.1 host.openshell.internal\n127.0.0.1 localhost\n", + ); + }); + + it("restores the host resolver file when verification fails", async () => { + const originalHosts = "127.0.0.1 localhost\n"; + vi.spyOn(fs, "mkdtempSync").mockReturnValue("/tmp/nemoclaw-compatible-endpoint-hosts-test"); + vi.spyOn(fs, "readFileSync").mockReturnValue(originalHosts); + vi.spyOn(fs, "writeFileSync").mockImplementation(() => {}); + const remove = vi.spyOn(fs, "rmSync").mockImplementation(() => {}); + const command = vi.fn().mockResolvedValue({ exitCode: 0, stderr: "", stdout: "" }); + const trackDisposable = vi.fn(); + + await expect( + withHostVerificationLoopbackAlias( + { command } as unknown as HostCliClient, + { trackDisposable }, + async () => { + throw new Error("verification failed"); + }, + ), + ).rejects.toThrow("verification failed"); + + expect(command.mock.calls.map(([program, args]) => [program, args])).toEqual([ + [ + "sudo", + [ + "cp", + "--", + "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.mapped", + "/etc/hosts", + ], + ], + [ + "sudo", + [ + "cp", + "--", + "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.original", + "/etc/hosts", + ], + ], + ]); + expect(trackDisposable.mock.invocationCallOrder[0]).toBeLessThan( + command.mock.invocationCallOrder[0] ?? Number.POSITIVE_INFINITY, + ); + expect(remove).toHaveBeenCalledWith("/tmp/nemoclaw-compatible-endpoint-hosts-test", { + force: true, + recursive: true, + }); + }); + it("requires the direct provider to be absent before inference set owns its creation", async () => { const command = vi.fn().mockResolvedValue({ exitCode: 1, From abbe5604a926ec40fae9c293087cb1960f5664cf Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 16:05:28 -0700 Subject: [PATCH 05/23] test(e2e): retain inference retry evidence --- test/e2e/README.md | 19 +++----- test/e2e/RETRY_INVENTORY.md | 45 +++++++++---------- test/e2e/fixtures/inference-switch-retry.ts | 11 +++++ .../live/hermes-inference-switch-helpers.ts | 7 +++ test/e2e/live/hermes-inference-switch.test.ts | 1 + .../live/openclaw-inference-switch.test.ts | 5 ++- ...mes-inference-switch-command-shape.test.ts | 39 +++++++++++++++- .../support/inference-switch-retry.test.ts | 26 +++++++++-- 8 files changed, 111 insertions(+), 42 deletions(-) diff --git a/test/e2e/README.md b/test/e2e/README.md index 5b2fe60fe94..33780383ef0 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -669,18 +669,13 @@ It does not handle `E2E main`. The complete non-passing job listing must contain only authenticated hosted-runner-loss evidence for the workflow's approved runner labels. An ordinary assertion failure, mixed failure set, incomplete listing, custom or self-hosted label, changed evidence, or ambiguous pagination prevents recovery. -For eligible `E2E main` push runs, `E2E / Main Retry` records first-attempt, -manual-retry, and exhausted-attempt outcomes without requesting a workflow -rerun. A failed job can represent a deterministic product assertion, -authentication or authorization failure, policy denial, malformed input, -ambiguous mutation, cleanup failure, or an external transient. GitHub job -conclusions do not distinguish those classes, so a broad failed-job rerun is -not authorized evidence. External operations use the checked-in retry inventory -and an explicit bounded policy; new shared paths use the bounded operation -helper. Their artifacts retain each attempt. -Hosted runner loss remains owned by Hosted Runner Recovery. The observer ignores -manual source runs and source runs superseded by a newer `main` push, checks out -only trusted default-branch code, and receives no repository secrets. +For eligible `E2E main` push runs, `E2E / Main Retry` records `passed-first-attempt`, `passed-after-retry`, `failed-no-retry`, or `ignored` without requesting a workflow rerun. +A failed job can represent a deterministic product assertion, authentication or authorization failure, policy denial, malformed input, ambiguous mutation, cleanup failure, or an external transient. +GitHub job conclusions do not distinguish those classes, so a broad failed-job rerun is not authorized evidence. +External operations use the checked-in retry inventory and an explicit bounded policy; new shared paths use the bounded operation helper. +Their artifacts retain each attempt. +Hosted runner loss remains owned by Hosted Runner Recovery. +The observer ignores manual source runs and source runs superseded by a newer `main` push, checks out only trusted default-branch code, and receives no repository secrets. The runner-allocation and internal-error failures handled by Hosted Runner Recovery originate in GitHub Actions, outside repository-controlled workflow diff --git a/test/e2e/RETRY_INVENTORY.md b/test/e2e/RETRY_INVENTORY.md index 01cb4f5e704..26c3805b0dd 100644 --- a/test/e2e/RETRY_INVENTORY.md +++ b/test/e2e/RETRY_INVENTORY.md @@ -1,18 +1,19 @@ -# E2E retry inventory + + -This inventory governs operation retries, workflow reruns, and transient skips in -the E2E suite. Readiness and eventual-consistency polling are observations, not -operation retries: they do not repeat a mutation and retain their final probe or -per-attempt artifacts through the existing E2E fixture APIs. +# E2E Retry Inventory -Only a `transient-external` classification may retry. Authentication, -authorization, policy denial, malformed input, deterministic product failure, -and cleanup failure are terminal. An ambiguous mutation must reconcile external -state before another attempt. Exhaustion remains failed. +This inventory governs operation retries, workflow reruns, and transient skips in the E2E suite. +Readiness and eventual-consistency polling are observations, not operation retries: they do not repeat a mutation and retain their final probe or per-attempt artifacts through the existing E2E fixture APIs. + +Only a `transient-external` classification may retry. +Authentication, authorization, policy denial, malformed input, deterministic product failure, and cleanup failure are terminal. +An ambiguous mutation must reconcile external state before another attempt. +Exhaustion remains failed. | ID | Operation and source | Failure signature | Bound and backoff | Idempotence basis | Owner | Result classification | Retained evidence | Disposition | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `workflow-main-observer` | Completed `E2E main` run; `tools/e2e/main-run-retry.mts`, `.github/workflows/e2e-main-retry.yaml` | Any failed job | 0 automatic reruns | Not established at workflow scope | Test owner | `failed-no-retry`, `passed-first-attempt`, or `passed-after-retry` for a manual rerun | Per-attempt jobs, conclusions, duration, source SHA, and owner | Broad rerun removed; operation-level evidence is required | +| `workflow-main-observer` | Completed `E2E main` run; `tools/e2e/main-run-retry.mts`, `.github/workflows/e2e-main-retry.yaml` | Any failed job | 0 automatic reruns | Not established at workflow scope | Test owner | `failed-no-retry`, `ignored`, `passed-first-attempt`, or `passed-after-retry` for a manual rerun | Failed job names, non-skipped job count, runner minutes, and source SHA for each attempt | Broad rerun removed; operation-level evidence is required | | `hosted-runner-recovery` | Confirmed GitHub-hosted runner loss; `tools/e2e/hosted-runner-recovery.mts`, `tools/e2e/hosted-runner-loss*.mts` | Authenticated runner-allocation or internal-runner evidence only | 1 recovery request; controller-defined delay | GitHub reruns a workflow attempt | GitHub Actions | Dedicated runner-loss classifications | Source and recovery run links plus authenticated job evidence | External owner; governed by #7146, not this policy | | `pr-rerun-reconciliation` | PR E2E dispatch reconciliation; `tools/e2e/pr-e2e-dispatch-reconciliation.mts`, `tools/e2e/pr-e2e-retry-receipt.mts` | Trusted dispatch receipt state | Contract-defined single reconciliation | Reconciles workflow and commit identity before action | GitHub Actions | Receipt-specific terminal states | Signed workflow identity and receipt | External scope; governed by #7206 | | `github-publication-read` | GitHub API reads; `tools/e2e/base-image-publication.mts` | Fetch error, 408, rate limit, or 5xx | 3 attempts; Retry-After/rate-limit reset or linear delay capped at 10s | Read-only | GitHub API | Returned parsed selection on success; thrown terminal HTTP/fetch error on failure or exhaustion | Caller artifact records the returned publication selection; terminal errors identify exhausted fetch or HTTP status without response content | Eligible bounded read; existing implementation retained | @@ -35,19 +36,15 @@ state before another attempt. Exhaustion remains failed. | `eventual-consistency-polling` | Generic readiness observation; `test/e2e/fixtures/polling.ts` and callers | Caller-defined expected not-ready state | Helper requires a finite `attempts` or `deadlineMs`; caller supplies explicit `delayMs` | Read-only observation | Component named by caller | Passed before bound, terminal observation, aborted, or exhausted | Last attempt and deterministic per-attempt artifact name | Not an operation retry; each caller owns and inventories its concrete bound | | `filesystem-cleanup` | Node `fs.rmSync` retry options; `tools/e2e/openshell-gateway-auth-artifact-safety.mts` | OS-level transient remove failure | 4 total filesystem attempts; 50ms | Idempotent removal of quarantined local artifacts | Host filesystem | Cleanup failure remains terminal | Quarantine path state without artifact content | Bounded cleanup; credentials are never serialized | -## Evidence contract +## Evidence Contract -`test/e2e/fixtures/retry-policy.ts` emits schema version 1. The aggregate -`outcome` is exactly one of `passed-first-attempt`, `passed-after-retry`, -`failed-no-retry`, or `exhausted`. Cleanup failures use `failed-no-retry` with a -final attempt whose `failureClass` is `cleanup`. Each attempt also records its -number, failure class, reconciliation result when applicable, and whether -another attempt was scheduled. The record deliberately excludes command output, -errors, request bodies, headers, and environment values. Callers retain their -normal redacted artifacts separately and may write the aggregate record through -`onEvidence`. +`test/e2e/fixtures/retry-policy.ts` emits schema version 1. +The aggregate `outcome` is exactly one of `passed-first-attempt`, `passed-after-retry`, `failed-no-retry`, or `exhausted`. +Cleanup failures use `failed-no-retry` with a final attempt whose `failureClass` is `cleanup`. +Each attempt also records its number, failure class, reconciliation result when applicable, and whether another attempt was scheduled. +The record deliberately excludes command output, errors, request bodies, headers, and environment values. +Callers retain their normal redacted artifacts separately and may write the aggregate record through `onEvidence`. +The TypeScript inference-switch targets write the aggregate record to `inference-switch-retry-evidence.json` through each target's redacting artifact sink. -Adding or changing a retry, rerun, transient skip, or polling family requires an -inventory update in the same pull request. A path is not eligible merely because -it sometimes passes later; it needs a narrow transient signature and a stated -idempotence or reconciliation basis. +Adding or changing a retry, rerun, transient skip, or polling family requires an inventory update in the same pull request. +A path is not eligible merely because it sometimes passes later; it needs a narrow transient signature and a stated idempotence or reconciliation basis. diff --git a/test/e2e/fixtures/inference-switch-retry.ts b/test/e2e/fixtures/inference-switch-retry.ts index 98070633201..bdcc1061dc8 100644 --- a/test/e2e/fixtures/inference-switch-retry.ts +++ b/test/e2e/fixtures/inference-switch-retry.ts @@ -7,6 +7,17 @@ import { runBoundedRetry, type RetryEvidence } from "./retry-policy.ts"; const TRANSIENT_INFERENCE_SET_FAILURE = /timed? out|timeout|ETIMEDOUT|ECONNRESET|EAI_AGAIN|ENOTFOUND|failed to connect|error sending request|\b50[234]\b/iu; +export interface InferenceSwitchRetryArtifactSink { + writeJson(path: string, value: unknown): Promise; +} + +export async function writeInferenceSwitchRetryEvidence( + artifacts: InferenceSwitchRetryArtifactSink, + evidence: RetryEvidence, +): Promise { + await artifacts.writeJson("inference-switch-retry-evidence.json", evidence); +} + export function inferenceSetAttemptCount(raw: string | undefined, fallback = 3): number { if (raw === undefined) return fallback; const parsed = Number(raw); diff --git a/test/e2e/live/hermes-inference-switch-helpers.ts b/test/e2e/live/hermes-inference-switch-helpers.ts index 3d65f15be19..5ed48777aee 100644 --- a/test/e2e/live/hermes-inference-switch-helpers.ts +++ b/test/e2e/live/hermes-inference-switch-helpers.ts @@ -36,7 +36,9 @@ import { import { inferenceResponseModel, inferenceSetAttemptCount, + type InferenceSwitchRetryArtifactSink, runInferenceSetWithRetry, + writeInferenceSwitchRetryEvidence, } from "../fixtures/inference-switch-retry.ts"; import { CLI_ENTRYPOINT, REPO_ROOT } from "../fixtures/paths.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; @@ -593,6 +595,7 @@ export async function runHermesInferenceSetWithRetry( compatibleMetadataArgs: string[], options: { attempts?: number; + artifacts?: InferenceSwitchRetryArtifactSink; compatibleBinding?: CompatibleAnthropicSwitchBinding | null; delay?: (milliseconds: number) => Promise; } = {}, @@ -607,10 +610,14 @@ export async function runHermesInferenceSetWithRetry( SWITCH_MODEL, ...compatibleMetadataArgs, ]; + const evidenceArtifacts = options.artifacts; return runInferenceSetWithRetry({ attempts: options.attempts ?? inferenceSetAttemptCount(process.env.NEMOCLAW_SWITCH_SET_ATTEMPTS), delay: options.delay, + onEvidence: evidenceArtifacts + ? (evidence) => writeInferenceSwitchRetryEvidence(evidenceArtifacts, evidence) + : undefined, run: (attempt, verify) => host.command("node", verify ? args : [...args, "--no-verify"], { artifactName: verify diff --git a/test/e2e/live/hermes-inference-switch.test.ts b/test/e2e/live/hermes-inference-switch.test.ts index 7c22fe4f4e1..ba839f378c5 100644 --- a/test/e2e/live/hermes-inference-switch.test.ts +++ b/test/e2e/live/hermes-inference-switch.test.ts @@ -193,6 +193,7 @@ test("Hermes inference set updates route/config and preserves live runtime", { const compatibleMetadataArgs = compatibleAnthropicMetadataArgs(switchEndpointUrl); const switchInference = () => runHermesInferenceSetWithRetry(host, redactionValues, compatibleMetadataArgs, { + artifacts, compatibleBinding: switchBinding, }); const switched = diff --git a/test/e2e/live/openclaw-inference-switch.test.ts b/test/e2e/live/openclaw-inference-switch.test.ts index 191a7a219d9..06a37154ed1 100644 --- a/test/e2e/live/openclaw-inference-switch.test.ts +++ b/test/e2e/live/openclaw-inference-switch.test.ts @@ -41,6 +41,7 @@ import { inferenceResponseModel, inferenceSetAttemptCount, runInferenceSetWithRetry, + writeInferenceSwitchRetryEvidence, } from "../fixtures/inference-switch-retry.ts"; import { CLI_ENTRYPOINT, REPO_ROOT } from "../fixtures/paths.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; @@ -867,6 +868,7 @@ async function runOpenClawInferenceSetWithRetry( home: string, redactionValues: string[], switchBinding: CompatibleAnthropicSwitchBinding | null, + artifacts: { writeJson(path: string, value: unknown): Promise }, ): Promise { const attempts = inferenceSetAttemptCount(process.env.NEMOCLAW_SWITCH_SET_ATTEMPTS); const compatibleCredentialEnv = (() => { @@ -903,6 +905,7 @@ async function runOpenClawInferenceSetWithRetry( return runInferenceSetWithRetry({ attempts, + onEvidence: (evidence) => writeInferenceSwitchRetryEvidence(artifacts, evidence), run: (attempt, verify) => runNemoclaw(host, home, verify ? args : [...args, "--no-verify"], { artifactName: verify @@ -1085,7 +1088,7 @@ test("openclaw-inference-switch: switches route and preserves live OpenClaw beha ); const pidBefore = await openclawGatewayPid(sandbox, home); const switchInference = () => - runOpenClawInferenceSetWithRetry(host, home, redactionValues, switchBinding); + runOpenClawInferenceSetWithRetry(host, home, redactionValues, switchBinding, artifacts); const switchResult = mockProvider ? await withHostVerificationLoopbackAlias(host, cleanup, switchInference) : await switchInference(); diff --git a/test/e2e/support/hermes-inference-switch-command-shape.test.ts b/test/e2e/support/hermes-inference-switch-command-shape.test.ts index af2d2011d21..fdced5ad593 100644 --- a/test/e2e/support/hermes-inference-switch-command-shape.test.ts +++ b/test/e2e/support/hermes-inference-switch-command-shape.test.ts @@ -376,6 +376,7 @@ describe("Hermes inference switch command shape", () => { stderr: "failed to verify inference endpoint: failed to connect", stdout: "", }); + const writeJson = vi.fn().mockResolvedValue("inference-switch-retry-evidence.json"); const compatibleBinding = compatibleAnthropicSwitchBinding( "http://host.openshell.internal:18766/v1", { COMPATIBLE_ANTHROPIC_API_KEY: "switch-key" }, @@ -386,7 +387,12 @@ describe("Hermes inference switch command shape", () => { { command } as unknown as HostCliClient, ["hosted-key", compatibleBinding.credentialValue], compatibleAnthropicMetadataArgs(compatibleBinding.endpointUrl), - { attempts: 1, compatibleBinding, delay: async () => {} }, + { + artifacts: { writeJson }, + attempts: 1, + compatibleBinding, + delay: async () => {}, + }, ), ).resolves.toMatchObject({ exitCode: 1 }); @@ -397,5 +403,36 @@ describe("Hermes inference switch command shape", () => { redactionValues: ["hosted-key", "switch-key"], }); expect(command.mock.calls[0]?.[2]?.env).not.toHaveProperty("NVIDIA_INFERENCE_API_KEY"); + expect(writeJson).toHaveBeenCalledWith( + "inference-switch-retry-evidence.json", + expect.objectContaining({ outcome: "exhausted" }), + ); + }); + + it("retains recovered route verification evidence", async () => { + const command = vi + .fn() + .mockResolvedValueOnce({ + exitCode: 1, + stderr: "failed to verify inference endpoint: timeout", + stdout: "", + }) + .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "route synced" }); + const writeJson = vi.fn().mockResolvedValue("inference-switch-retry-evidence.json"); + + await expect( + runHermesInferenceSetWithRetry( + { command } as unknown as HostCliClient, + ["hosted-key"], + [], + { artifacts: { writeJson }, attempts: 2, delay: async () => {} }, + ), + ).resolves.toMatchObject({ exitCode: 0 }); + + expect(command).toHaveBeenCalledTimes(2); + expect(writeJson).toHaveBeenCalledWith( + "inference-switch-retry-evidence.json", + expect.objectContaining({ outcome: "passed-after-retry" }), + ); }); }); diff --git a/test/e2e/support/inference-switch-retry.test.ts b/test/e2e/support/inference-switch-retry.test.ts index 213ec213f76..eba942830ae 100644 --- a/test/e2e/support/inference-switch-retry.test.ts +++ b/test/e2e/support/inference-switch-retry.test.ts @@ -12,6 +12,7 @@ import { inferenceResponseModel, inferenceSetAttemptCount, runInferenceSetWithRetry, + writeInferenceSwitchRetryEvidence, } from "../fixtures/inference-switch-retry.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; @@ -56,16 +57,17 @@ describe("inference switch retry", () => { .fn() .mockResolvedValueOnce(result(1, "failed to verify inference endpoint: timeout")) .mockResolvedValueOnce(result(0)); - const evidence = vi.fn(); + const writeJson = vi.fn().mockResolvedValue("inference-switch-retry-evidence.json"); await runInferenceSetWithRetry({ attempts: 2, delay: async () => {}, run, - onEvidence: evidence, + onEvidence: (evidence) => writeInferenceSwitchRetryEvidence({ writeJson }, evidence), }); - expect(evidence).toHaveBeenCalledWith( + expect(writeJson).toHaveBeenCalledWith( + "inference-switch-retry-evidence.json", expect.objectContaining({ outcome: "passed-after-retry", attempts: [ @@ -79,14 +81,30 @@ describe("inference switch retry", () => { it("keeps exhausted verified attempts failed without bypassing verification", async () => { const transient = result(1, "failed to connect to endpoint"); const run = vi.fn().mockResolvedValueOnce(transient).mockResolvedValueOnce(transient); + const writeJson = vi.fn().mockResolvedValue("inference-switch-retry-evidence.json"); await expect( - runInferenceSetWithRetry({ attempts: 2, delay: async () => {}, run }), + runInferenceSetWithRetry({ + attempts: 2, + delay: async () => {}, + onEvidence: (evidence) => writeInferenceSwitchRetryEvidence({ writeJson }, evidence), + run, + }), ).resolves.toMatchObject({ exitCode: 1 }); expect(run.mock.calls).toEqual([ [1, true], [2, true], ]); + expect(writeJson).toHaveBeenCalledWith( + "inference-switch-retry-evidence.json", + expect.objectContaining({ + outcome: "exhausted", + attempts: [ + expect.objectContaining({ failureClass: "transient-external", retryScheduled: true }), + expect.objectContaining({ failureClass: "transient-external", retryScheduled: false }), + ], + }), + ); }); it("keeps the shell helper failed on exhaustion without adding a verification bypass", () => { From d91373bceb5f590453c63c61ed79ae7b53618fca Mon Sep 17 00:00:00 2001 From: Rebecca Sliter <571084+rsliter@users.noreply.github.com> Date: Fri, 14 Aug 2026 18:03:06 -0700 Subject: [PATCH 06/23] fix(e2e): preserve concurrent host resolver state Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> --- test/e2e/README.md | 12 +- .../fixtures/compatible-anthropic-switch.ts | 144 +++++++++++++++--- .../compatible-anthropic-switch.test.ts | 133 +++++++++++++--- 3 files changed, 248 insertions(+), 41 deletions(-) diff --git a/test/e2e/README.md b/test/e2e/README.md index 33780383ef0..0d652ce637f 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -969,7 +969,8 @@ flowchart LR Selected jobs retain their runner, credential, evidence, and cleanup boundaries. A main push can queue repository-owned GPU runners or create external resources when a selected target requires them. -The retry workflow reruns failed jobs at most twice. +The main-run observer records attempt evidence but does not request broad failed-job reruns. +Each E2E test owns any bounded operation-level retry policy. `Exact staging Brev Launchable` runs only for a trusted manual dispatch against `main`. The job reads these credentials from repository Actions secrets: @@ -987,13 +988,14 @@ These credentials remain valid until they expire or an administrator revokes them in their issuing services. If cleanup fails, remove the recorded Brev workspace. Rotate or revoke each credential to remove later access. -When an eligible `E2E main` push workflow concludes with `failure`, `E2E / Main Retry` asks GitHub Actions to rerun failed jobs and their dependent jobs. -The controller permits two reruns but does not verify that GitHub schedules a different runner. -After evaluation succeeds, it uploads an artifact named for the current attempt. +When an eligible `E2E main` push workflow completes, `E2E / Main Retry` records its conclusion and the available source-attempt evidence. +It does not request a broad failed-job or workflow rerun. +An owning E2E test can retry an external operation only through its checked-in bounded policy. +After evaluation succeeds, the observer uploads an artifact named for the current attempt. The artifact contains one `attempts` summary for each source attempt through the current attempt. The `totalRunnerMinutes` field contains the cumulative runner time for those summaries. A later successful attempt sets `action` to `passed-after-retry` and `flaky` to `true`. -The controller does not retry manual PR runs or a run superseded by a newer `main` push. +The observer ignores manual PR runs and a run superseded by a newer `main` push. For a PR revision run, a repository maintainer or administrator leaves `jobs` and `targets` empty. The run selects: diff --git a/test/e2e/fixtures/compatible-anthropic-switch.ts b/test/e2e/fixtures/compatible-anthropic-switch.ts index 752a37be9e4..e3d2d1f46a5 100644 --- a/test/e2e/fixtures/compatible-anthropic-switch.ts +++ b/test/e2e/fixtures/compatible-anthropic-switch.ts @@ -12,6 +12,12 @@ export const COMPATIBLE_ANTHROPIC_PROVIDER = "compatible-anthropic-endpoint"; export const COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV = "COMPATIBLE_ANTHROPIC_API_KEY"; const DEFAULT_COMPATIBLE_ANTHROPIC_CREDENTIAL = "test-compatible-anthropic-key"; const OPENSHELL_HOST_ALIAS = "host.openshell.internal"; +const HOST_VERIFICATION_LOCK_PATH = path.join( + os.tmpdir(), + "nemoclaw-compatible-endpoint-hosts.lock", +); +const HOST_VERIFICATION_LOCK_TIMEOUT_MS = 5 * 60_000; +const HOST_VERIFICATION_LOCK_POLL_MS = 100; export interface CompatibleAnthropicSwitchBinding { endpointUrl: string; @@ -49,6 +55,49 @@ export function hostVerificationHostsFile(source: string): string { return `127.0.0.1 ${OPENSHELL_HOST_ALIAS}\n${existing}`; } +function errorCode(error: unknown): string | undefined { + return typeof error === "object" && error !== null && "code" in error + ? String(error.code) + : undefined; +} + +async function withHostVerificationHostsLock(run: () => Promise): Promise { + const deadline = Date.now() + HOST_VERIFICATION_LOCK_TIMEOUT_MS; + let descriptor: number; + while (true) { + try { + descriptor = fs.openSync(HOST_VERIFICATION_LOCK_PATH, "wx", 0o600); + break; + } catch (error) { + if (errorCode(error) !== "EEXIST") throw error; + if (Date.now() >= deadline) { + throw new Error( + `timed out waiting for exclusive host resolver fixture ownership: ${HOST_VERIFICATION_LOCK_PATH}`, + ); + } + await new Promise((resolve) => setTimeout(resolve, HOST_VERIFICATION_LOCK_POLL_MS)); + } + } + + try { + return await run(); + } finally { + fs.closeSync(descriptor); + fs.unlinkSync(HOST_VERIFICATION_LOCK_PATH); + } +} + +const REPLACE_HOSTS_IF_UNCHANGED = [ + "set -euo pipefail", + 'expected="$1"', + 'replacement="$2"', + 'if ! cmp -s -- "$expected" /etc/hosts; then', + ' echo "/etc/hosts changed while the NemoClaw host verifier alias was active" >&2', + " exit 3", + "fi", + 'cp -- "$replacement" /etc/hosts', +].join("\n"); + export async function withHostVerificationLoopbackAlias( host: HostCliClient, cleanup: { trackDisposable(name: string, run: () => Promise | void): void }, @@ -57,19 +106,46 @@ export async function withHostVerificationLoopbackAlias( const fixtureDirectory = fs.mkdtempSync( path.join(os.tmpdir(), "nemoclaw-compatible-endpoint-hosts-"), ); - const originalHosts = fs.readFileSync("/etc/hosts", "utf8"); const originalPath = path.join(fixtureDirectory, "hosts.original"); const mappedPath = path.join(fixtureDirectory, "hosts.mapped"); - fs.writeFileSync(originalPath, originalHosts, { mode: 0o600 }); - fs.writeFileSync(mappedPath, hostVerificationHostsFile(originalHosts), { mode: 0o600 }); let restored = false; - const restore = async (): Promise => { + let originalHosts: string | null = null; + let mappedHosts: string | null = null; + let mappingMayNeedRecovery = false; + const restoreWhileLocked = async (): Promise => { if (restored) return; - const result = await host.command("sudo", ["cp", "--", originalPath, "/etc/hosts"], { - artifactName: "restore-host-verifier-alias", - timeoutMs: 30_000, - }); + if (!mappingMayNeedRecovery || originalHosts === null || mappedHosts === null) { + restored = true; + fs.rmSync(fixtureDirectory, { force: true, recursive: true }); + return; + } + const currentHosts = fs.readFileSync("/etc/hosts", "utf8"); + if (currentHosts === originalHosts) { + restored = true; + fs.rmSync(fixtureDirectory, { force: true, recursive: true }); + return; + } + if (currentHosts !== mappedHosts) { + throw new Error( + "/etc/hosts changed while the host verifier alias was active; refusing to overwrite concurrent resolver state", + ); + } + const result = await host.command( + "sudo", + [ + "bash", + "-ceu", + REPLACE_HOSTS_IF_UNCHANGED, + "restore-host-verifier-alias", + mappedPath, + originalPath, + ], + { + artifactName: "restore-host-verifier-alias", + timeoutMs: 30_000, + }, + ); if (result.exitCode !== 0) { throw new Error(`could not restore /etc/hosts: ${resultText(result)}`); } @@ -79,20 +155,50 @@ export async function withHostVerificationLoopbackAlias( restored = true; fs.rmSync(fixtureDirectory, { force: true, recursive: true }); }; + const restore = async (): Promise => { + if (restored) return; + await withHostVerificationHostsLock(restoreWhileLocked); + }; cleanup.trackDisposable("restore the host verifier alias mapping", restore); - try { - const mapped = await host.command("sudo", ["cp", "--", mappedPath, "/etc/hosts"], { - artifactName: "map-host-verifier-alias", - timeoutMs: 30_000, - }); - if (mapped.exitCode !== 0) { - throw new Error(`could not map the host verifier alias: ${resultText(mapped)}`); + return await withHostVerificationHostsLock(async () => { + originalHosts = fs.readFileSync("/etc/hosts", "utf8"); + mappedHosts = hostVerificationHostsFile(originalHosts); + fs.writeFileSync(originalPath, originalHosts, { mode: 0o600 }); + fs.writeFileSync(mappedPath, mappedHosts, { mode: 0o600 }); + + try { + let mapped: Awaited>; + try { + mapped = await host.command( + "sudo", + [ + "bash", + "-ceu", + REPLACE_HOSTS_IF_UNCHANGED, + "map-host-verifier-alias", + originalPath, + mappedPath, + ], + { + artifactName: "map-host-verifier-alias", + timeoutMs: 30_000, + }, + ); + } finally { + mappingMayNeedRecovery = fs.readFileSync("/etc/hosts", "utf8") !== originalHosts; + } + if (mapped.exitCode !== 0) { + throw new Error(`could not map the host verifier alias: ${resultText(mapped)}`); + } + if (fs.readFileSync("/etc/hosts", "utf8") !== mappedHosts) { + throw new Error("/etc/hosts differs after host verifier alias installation"); + } + return await run(); + } finally { + await restoreWhileLocked(); } - return await run(); - } finally { - await restore(); - } + }); } export async function requireCompatibleAnthropicProviderAbsent( diff --git a/test/e2e/support/compatible-anthropic-switch.test.ts b/test/e2e/support/compatible-anthropic-switch.test.ts index 674c5417bc5..29e649e7819 100644 --- a/test/e2e/support/compatible-anthropic-switch.test.ts +++ b/test/e2e/support/compatible-anthropic-switch.test.ts @@ -19,6 +19,17 @@ import { describe("compatible Anthropic inference switch setup", () => { afterEach(() => vi.restoreAllMocks()); + function mockHostsFixture(reads: string[]) { + vi.spyOn(fs, "mkdtempSync").mockReturnValue("/tmp/nemoclaw-compatible-endpoint-hosts-test"); + vi.spyOn(fs, "openSync").mockReturnValue(42); + const close = vi.spyOn(fs, "closeSync").mockImplementation(() => {}); + const unlink = vi.spyOn(fs, "unlinkSync").mockImplementation(() => {}); + vi.spyOn(fs, "readFileSync").mockImplementation(() => reads.shift() ?? ""); + vi.spyOn(fs, "writeFileSync").mockImplementation(() => {}); + const remove = vi.spyOn(fs, "rmSync").mockImplementation(() => {}); + return { close, remove, unlink }; + } + it("passes the direct binding credential only to the inference-set command", () => { const binding = compatibleAnthropicSwitchBinding("http://host.openshell.internal:18766", { COMPATIBLE_ANTHROPIC_API_KEY: "fixture-key", @@ -58,10 +69,14 @@ describe("compatible Anthropic inference switch setup", () => { it("restores the host resolver file when verification fails", async () => { const originalHosts = "127.0.0.1 localhost\n"; - vi.spyOn(fs, "mkdtempSync").mockReturnValue("/tmp/nemoclaw-compatible-endpoint-hosts-test"); - vi.spyOn(fs, "readFileSync").mockReturnValue(originalHosts); - vi.spyOn(fs, "writeFileSync").mockImplementation(() => {}); - const remove = vi.spyOn(fs, "rmSync").mockImplementation(() => {}); + const mappedHosts = hostVerificationHostsFile(originalHosts); + const { close, remove, unlink } = mockHostsFixture([ + originalHosts, + mappedHosts, + mappedHosts, + mappedHosts, + originalHosts, + ]); const command = vi.fn().mockResolvedValue({ exitCode: 0, stderr: "", stdout: "" }); const trackDisposable = vi.fn(); @@ -75,24 +90,21 @@ describe("compatible Anthropic inference switch setup", () => { ), ).rejects.toThrow("verification failed"); - expect(command.mock.calls.map(([program, args]) => [program, args])).toEqual([ + expect(command).toHaveBeenCalledTimes(2); + expect( + command.mock.calls.map(([program, args]) => [program, args?.[0], args?.at(-2), args?.at(-1)]), + ).toEqual([ [ "sudo", - [ - "cp", - "--", - "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.mapped", - "/etc/hosts", - ], + "bash", + "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.original", + "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.mapped", ], [ "sudo", - [ - "cp", - "--", - "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.original", - "/etc/hosts", - ], + "bash", + "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.mapped", + "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.original", ], ]); expect(trackDisposable.mock.invocationCallOrder[0]).toBeLessThan( @@ -102,6 +114,93 @@ describe("compatible Anthropic inference switch setup", () => { force: true, recursive: true, }); + expect(close).toHaveBeenCalledWith(42); + expect(unlink).toHaveBeenCalledTimes(1); + }); + + it("does not discard a concurrent host resolver update", async () => { + const originalHosts = "127.0.0.1 localhost\n"; + const mappedHosts = hostVerificationHostsFile(originalHosts); + const concurrentHosts = `${mappedHosts}192.0.2.10 concurrent.example.test\n`; + const { close, remove, unlink } = mockHostsFixture([ + originalHosts, + mappedHosts, + mappedHosts, + concurrentHosts, + ]); + const command = vi.fn().mockResolvedValue({ exitCode: 0, stderr: "", stdout: "" }); + + await expect( + withHostVerificationLoopbackAlias( + { command } as unknown as HostCliClient, + { trackDisposable: vi.fn() }, + async () => undefined, + ), + ).rejects.toThrow("refusing to overwrite concurrent resolver state"); + + expect(command).toHaveBeenCalledTimes(1); + expect(remove).not.toHaveBeenCalled(); + expect(close).toHaveBeenCalledWith(42); + expect(unlink).toHaveBeenCalledTimes(1); + }); + + it("reports a failed owned resolver restoration", async () => { + const originalHosts = "127.0.0.1 localhost\n"; + const mappedHosts = hostVerificationHostsFile(originalHosts); + const { close, remove, unlink } = mockHostsFixture([ + originalHosts, + mappedHosts, + mappedHosts, + mappedHosts, + ]); + const command = vi + .fn() + .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "" }) + .mockResolvedValueOnce({ exitCode: 1, stderr: "permission denied", stdout: "" }); + + await expect( + withHostVerificationLoopbackAlias( + { command } as unknown as HostCliClient, + { trackDisposable: vi.fn() }, + async () => undefined, + ), + ).rejects.toThrow("could not restore /etc/hosts: permission denied"); + + expect(command).toHaveBeenCalledTimes(2); + expect(remove).not.toHaveBeenCalled(); + expect(close).toHaveBeenCalledWith(42); + expect(unlink).toHaveBeenCalledTimes(1); + }); + + it("restores an observed mapping after the mapping runner fails", async () => { + const originalHosts = "127.0.0.1 localhost\n"; + const mappedHosts = hostVerificationHostsFile(originalHosts); + const { close, remove, unlink } = mockHostsFixture([ + originalHosts, + mappedHosts, + mappedHosts, + originalHosts, + ]); + const command = vi + .fn() + .mockRejectedValueOnce(new Error("mapping runner disconnected")) + .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "" }); + + await expect( + withHostVerificationLoopbackAlias( + { command } as unknown as HostCliClient, + { trackDisposable: vi.fn() }, + async () => undefined, + ), + ).rejects.toThrow("mapping runner disconnected"); + + expect(command).toHaveBeenCalledTimes(2); + expect(remove).toHaveBeenCalledWith("/tmp/nemoclaw-compatible-endpoint-hosts-test", { + force: true, + recursive: true, + }); + expect(close).toHaveBeenCalledWith(42); + expect(unlink).toHaveBeenCalledTimes(1); }); it("requires the direct provider to be absent before inference set owns its creation", async () => { From e1fd5fe98ef8675904bc90bce0dceb22951f4caf Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 18:47:42 -0700 Subject: [PATCH 07/23] fix(e2e): recover host resolver fixture ownership Signed-off-by: Carlos Villela --- test/e2e/docs/README.md | 24 +- .../fixtures/compatible-anthropic-switch.ts | 277 +++++++++--------- .../compatible-anthropic-switch.test.ts | 249 +++++++++------- 3 files changed, 292 insertions(+), 258 deletions(-) diff --git a/test/e2e/docs/README.md b/test/e2e/docs/README.md index ccfd7c073c7..3d29391fd80 100644 --- a/test/e2e/docs/README.md +++ b/test/e2e/docs/README.md @@ -97,24 +97,18 @@ npm run test:runtime-audit -- e2e-artifacts/run-1 e2e-artifacts/run-2 The aggregate local command rebuilds the CLI before Vitest starts and runs E2E test files serially. It does not retry a failed test. -After an eligible `E2E main` push workflow fails, `E2E / Main Retry` asks GitHub Actions to rerun failed jobs and their dependent jobs. -A successful CLI artifact producer is not rerun. -The workflow retains its CLI artifact for 3 days. -During that period, consumers reuse the immutable, content-addressed artifact from the earlier producer attempt in the same workflow run. -If the artifact is unavailable when a consumer downloads it, restoration fails because the failed-job rerun does not rerun the successful producer. -Restore validation binds the producer provenance to the workflow run, workflow SHA, and candidate checkout. -It downloads by immutable artifact ID and verifies the manifest and the payload digest. -It rejects a producer attempt that is newer than the consumer attempt. -The controller can request two reruns, for three total attempts. -It does not verify that GitHub schedules a different runner, so do not treat a rerun as evidence of a fresh host. -If a later attempt succeeds, the source workflow concludes with `success`. -The evidence sets `action` to `passed-after-retry` and `flaky` to `true`. - -After the controller evaluates attempt N, it uploads an artifact named for that +After an eligible `E2E main` push workflow completes, `E2E / Main Retry` records its conclusion and source-attempt evidence. +It does not request a broad failed-job or workflow rerun. +An E2E test can retry an external operation only through its checked-in bounded policy. +The observer records `passed-first-attempt`, `passed-after-retry`, `failed-no-retry`, or `ignored`. +The `flaky` field is `true` only for `passed-after-retry`. +Hosted Runner Recovery separately owns a rerun with authenticated GitHub-hosted runner-loss evidence. + +After the observer evaluates attempt N, it uploads an artifact named for that attempt. The artifact contains one `attempts` entry for each source attempt through N. `totalRunnerMinutes` is the sum across those entries. If evaluation or file creation fails, the upload step warns that the file is missing and publishes no -evidence artifact. The controller does not retry manual PR runs or a run +evidence artifact. The observer ignores manual PR runs and a run superseded by a newer `main` push. During fixture teardown, every passing or failing live test writes diff --git a/test/e2e/fixtures/compatible-anthropic-switch.ts b/test/e2e/fixtures/compatible-anthropic-switch.ts index e3d2d1f46a5..53cd84b887f 100644 --- a/test/e2e/fixtures/compatible-anthropic-switch.ts +++ b/test/e2e/fixtures/compatible-anthropic-switch.ts @@ -1,9 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { randomBytes } from "node:crypto"; import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; import type { HostCliClient } from "./clients/host.ts"; import { resultText } from "./clients/index.ts"; @@ -11,13 +10,88 @@ import { resultText } from "./clients/index.ts"; export const COMPATIBLE_ANTHROPIC_PROVIDER = "compatible-anthropic-endpoint"; export const COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV = "COMPATIBLE_ANTHROPIC_API_KEY"; const DEFAULT_COMPATIBLE_ANTHROPIC_CREDENTIAL = "test-compatible-anthropic-key"; -const OPENSHELL_HOST_ALIAS = "host.openshell.internal"; -const HOST_VERIFICATION_LOCK_PATH = path.join( - os.tmpdir(), - "nemoclaw-compatible-endpoint-hosts.lock", -); -const HOST_VERIFICATION_LOCK_TIMEOUT_MS = 5 * 60_000; -const HOST_VERIFICATION_LOCK_POLL_MS = 100; +const HOST_VERIFICATION_HOSTS_PATH = "/etc/hosts"; +const HOST_VERIFICATION_LOCK_PATH = "/run/lock/nemoclaw-compatible-endpoint-hosts.lock"; +const HOST_VERIFICATION_COMMAND_TIMEOUT_MS = 60_000; + +export const HOST_VERIFICATION_ALIAS_SCRIPT = [ + "set -euo pipefail", + "", + 'operation="$1"', + 'hosts_path="$2"', + 'lock_path="$3"', + 'owner_pid="$4"', + 'owner_start="$5"', + 'owner_token="$6"', + 'alias_name="host.openshell.internal"', + "", + 'case "$operation" in', + " add | remove) ;;", + ' *) echo "unsupported host verifier alias operation: $operation" >&2; exit 2 ;;', + "esac", + '[[ "$owner_pid" =~ ^[1-9][0-9]*$ ]] || { echo "invalid host verifier owner PID" >&2; exit 2; }', + '[[ "$owner_start" =~ ^[1-9][0-9]*$ ]] || { echo "invalid host verifier owner start time" >&2; exit 2; }', + '[[ "$owner_token" =~ ^[a-f0-9]{32}$ ]] || { echo "invalid host verifier owner token" >&2; exit 2; }', + '[[ -f "$hosts_path" && ! -L "$hosts_path" ]] || { echo "host resolver file is not a regular file" >&2; exit 2; }', + '[[ ! -L "$lock_path" ]] || { echo "host resolver lock path must not be a symbolic link" >&2; exit 2; }', + 'command -v flock >/dev/null 2>&1 || { echo "flock is required for host resolver fixture ownership" >&2; exit 2; }', + "", + "owner_is_alive() {", + ' local pid="$1" expected_start="$2" stat remainder', + ' [[ -r "/proc/${pid}/stat" ]] || return 1', + ' IFS= read -r stat < "/proc/${pid}/stat"', + ' remainder="${stat##*) }"', + " set -- $remainder", + ' [[ "${20:-}" == "$expected_start" ]]', + "}", + "", + 'if [[ "$operation" == "add" ]] && ! owner_is_alive "$owner_pid" "$owner_start"; then', + ' echo "host verifier owner process is not alive" >&2', + " exit 2", + "fi", + "", + "umask 077", + 'exec 9>>"$lock_path"', + 'chmod 0600 "$lock_path"', + 'flock -x -w 30 9 || { echo "timed out waiting for host resolver fixture ownership" >&2; exit 3; }', + "", + 'marker="# nemoclaw-host-verifier:${owner_pid}:${owner_start}:${owner_token}"', + 'owned_line="127.0.0.1 ${alias_name} ${marker}"', + 'snapshot="$(mktemp "${hosts_path}.nemoclaw-snapshot.XXXXXX")"', + 'replacement="$(mktemp "${hosts_path}.nemoclaw-replacement.XXXXXX")"', + "trap 'rm -f \"$snapshot\" \"$replacement\"' EXIT", + 'cp --preserve=all -- "$hosts_path" "$snapshot"', + 'cp --preserve=all -- "$hosts_path" "$replacement"', + ': > "$replacement"', + "", + 'while IFS= read -r line || [[ -n "$line" ]]; do', + ' if [[ "$line" == "$owned_line" ]]; then', + " continue", + " fi", + ' if [[ "$line" =~ ^127\\.0\\.0\\.1[[:space:]]+host\\.openshell\\.internal[[:space:]]+#[[:space:]]nemoclaw-host-verifier:([1-9][0-9]*):([1-9][0-9]*):([a-f0-9]{32})$ ]]; then', + ' if owner_is_alive "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}"; then', + " printf '%s\\n' \"$line\" >> \"$replacement\"", + " fi", + " continue", + " fi", + " printf '%s\\n' \"$line\" >> \"$replacement\"", + 'done < "$snapshot"', + "", + 'if [[ "$operation" == "add" ]]; then', + " printf '%s\\n' \"$owned_line\" >> \"$replacement\"", + "fi", + "", + "# The kernel lock serializes every cooperative fixture writer. Replacing the", + "# file in one rename prevents readers from observing a partial resolver file.", + 'mv -f -- "$replacement" "$hosts_path"', + "trap 'rm -f \"$snapshot\"' EXIT", + "", + 'if [[ "$operation" == "add" ]]; then', + ' grep -Fqx -- "$owned_line" "$hosts_path" || { echo "host verifier alias was not installed" >&2; exit 4; }', + "else", + ' ! grep -Fqx -- "$owned_line" "$hosts_path" || { echo "host verifier alias was not removed" >&2; exit 4; }', + "fi", +].join("\n"); export interface CompatibleAnthropicSwitchBinding { endpointUrl: string; @@ -50,155 +124,80 @@ export function compatibleAnthropicSwitchEnv( return binding ? { [COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV]: binding.credentialValue } : {}; } -export function hostVerificationHostsFile(source: string): string { - const existing = source.endsWith("\n") ? source : `${source}\n`; - return `127.0.0.1 ${OPENSHELL_HOST_ALIAS}\n${existing}`; +function hostVerificationOwnerStartTime(): string { + const stat = fs.readFileSync(`/proc/${process.pid}/stat`, "utf8"); + const close = stat.lastIndexOf(") "); + const fields = close >= 0 ? stat.slice(close + 2).trim().split(/\s+/u) : []; + const startTime = fields[19]; + if (!startTime || !/^[1-9][0-9]*$/u.test(startTime)) { + throw new Error("could not read the host verifier owner process start time"); + } + return startTime; } -function errorCode(error: unknown): string | undefined { - return typeof error === "object" && error !== null && "code" in error - ? String(error.code) - : undefined; +interface HostVerificationOwner { + pid: number; + startTime: string; + token: string; } -async function withHostVerificationHostsLock(run: () => Promise): Promise { - const deadline = Date.now() + HOST_VERIFICATION_LOCK_TIMEOUT_MS; - let descriptor: number; - while (true) { - try { - descriptor = fs.openSync(HOST_VERIFICATION_LOCK_PATH, "wx", 0o600); - break; - } catch (error) { - if (errorCode(error) !== "EEXIST") throw error; - if (Date.now() >= deadline) { - throw new Error( - `timed out waiting for exclusive host resolver fixture ownership: ${HOST_VERIFICATION_LOCK_PATH}`, - ); - } - await new Promise((resolve) => setTimeout(resolve, HOST_VERIFICATION_LOCK_POLL_MS)); - } - } +function createHostVerificationOwner(): HostVerificationOwner { + return { + pid: process.pid, + startTime: hostVerificationOwnerStartTime(), + token: randomBytes(16).toString("hex"), + }; +} - try { - return await run(); - } finally { - fs.closeSync(descriptor); - fs.unlinkSync(HOST_VERIFICATION_LOCK_PATH); +async function updateHostVerificationAlias( + host: HostCliClient, + operation: "add" | "remove", + owner: HostVerificationOwner, +): Promise { + const artifactName = `${operation === "add" ? "map" : "restore"}-host-verifier-alias`; + const result = await host.command( + "sudo", + [ + "bash", + "-ceu", + HOST_VERIFICATION_ALIAS_SCRIPT, + artifactName, + operation, + HOST_VERIFICATION_HOSTS_PATH, + HOST_VERIFICATION_LOCK_PATH, + String(owner.pid), + owner.startTime, + owner.token, + ], + { artifactName, timeoutMs: HOST_VERIFICATION_COMMAND_TIMEOUT_MS }, + ); + if (result.exitCode !== 0) { + throw new Error( + `could not ${operation === "add" ? "install" : "remove"} the host verifier alias: ${resultText(result)}`, + ); } } -const REPLACE_HOSTS_IF_UNCHANGED = [ - "set -euo pipefail", - 'expected="$1"', - 'replacement="$2"', - 'if ! cmp -s -- "$expected" /etc/hosts; then', - ' echo "/etc/hosts changed while the NemoClaw host verifier alias was active" >&2', - " exit 3", - "fi", - 'cp -- "$replacement" /etc/hosts', -].join("\n"); - export async function withHostVerificationLoopbackAlias( host: HostCliClient, cleanup: { trackDisposable(name: string, run: () => Promise | void): void }, run: () => Promise, ): Promise { - const fixtureDirectory = fs.mkdtempSync( - path.join(os.tmpdir(), "nemoclaw-compatible-endpoint-hosts-"), - ); - const originalPath = path.join(fixtureDirectory, "hosts.original"); - const mappedPath = path.join(fixtureDirectory, "hosts.mapped"); - + const owner = createHostVerificationOwner(); let restored = false; - let originalHosts: string | null = null; - let mappedHosts: string | null = null; - let mappingMayNeedRecovery = false; - const restoreWhileLocked = async (): Promise => { - if (restored) return; - if (!mappingMayNeedRecovery || originalHosts === null || mappedHosts === null) { - restored = true; - fs.rmSync(fixtureDirectory, { force: true, recursive: true }); - return; - } - const currentHosts = fs.readFileSync("/etc/hosts", "utf8"); - if (currentHosts === originalHosts) { - restored = true; - fs.rmSync(fixtureDirectory, { force: true, recursive: true }); - return; - } - if (currentHosts !== mappedHosts) { - throw new Error( - "/etc/hosts changed while the host verifier alias was active; refusing to overwrite concurrent resolver state", - ); - } - const result = await host.command( - "sudo", - [ - "bash", - "-ceu", - REPLACE_HOSTS_IF_UNCHANGED, - "restore-host-verifier-alias", - mappedPath, - originalPath, - ], - { - artifactName: "restore-host-verifier-alias", - timeoutMs: 30_000, - }, - ); - if (result.exitCode !== 0) { - throw new Error(`could not restore /etc/hosts: ${resultText(result)}`); - } - if (fs.readFileSync("/etc/hosts", "utf8") !== originalHosts) { - throw new Error("/etc/hosts differs after host verifier alias restoration"); - } - restored = true; - fs.rmSync(fixtureDirectory, { force: true, recursive: true }); - }; const restore = async (): Promise => { if (restored) return; - await withHostVerificationHostsLock(restoreWhileLocked); + await updateHostVerificationAlias(host, "remove", owner); + restored = true; }; cleanup.trackDisposable("restore the host verifier alias mapping", restore); - return await withHostVerificationHostsLock(async () => { - originalHosts = fs.readFileSync("/etc/hosts", "utf8"); - mappedHosts = hostVerificationHostsFile(originalHosts); - fs.writeFileSync(originalPath, originalHosts, { mode: 0o600 }); - fs.writeFileSync(mappedPath, mappedHosts, { mode: 0o600 }); - - try { - let mapped: Awaited>; - try { - mapped = await host.command( - "sudo", - [ - "bash", - "-ceu", - REPLACE_HOSTS_IF_UNCHANGED, - "map-host-verifier-alias", - originalPath, - mappedPath, - ], - { - artifactName: "map-host-verifier-alias", - timeoutMs: 30_000, - }, - ); - } finally { - mappingMayNeedRecovery = fs.readFileSync("/etc/hosts", "utf8") !== originalHosts; - } - if (mapped.exitCode !== 0) { - throw new Error(`could not map the host verifier alias: ${resultText(mapped)}`); - } - if (fs.readFileSync("/etc/hosts", "utf8") !== mappedHosts) { - throw new Error("/etc/hosts differs after host verifier alias installation"); - } - return await run(); - } finally { - await restoreWhileLocked(); - } - }); + try { + await updateHostVerificationAlias(host, "add", owner); + return await run(); + } finally { + await restore(); + } } export async function requireCompatibleAnthropicProviderAbsent( diff --git a/test/e2e/support/compatible-anthropic-switch.test.ts b/test/e2e/support/compatible-anthropic-switch.test.ts index 29e649e7819..e8584414198 100644 --- a/test/e2e/support/compatible-anthropic-switch.test.ts +++ b/test/e2e/support/compatible-anthropic-switch.test.ts @@ -1,7 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { spawn, spawnSync } from "node:child_process"; import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -11,7 +14,7 @@ import { COMPATIBLE_ANTHROPIC_PROVIDER, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, - hostVerificationHostsFile, + HOST_VERIFICATION_ALIAS_SCRIPT, requireCompatibleAnthropicProviderAbsent, withHostVerificationLoopbackAlias, } from "../fixtures/compatible-anthropic-switch.ts"; @@ -19,15 +22,11 @@ import { describe("compatible Anthropic inference switch setup", () => { afterEach(() => vi.restoreAllMocks()); - function mockHostsFixture(reads: string[]) { - vi.spyOn(fs, "mkdtempSync").mockReturnValue("/tmp/nemoclaw-compatible-endpoint-hosts-test"); - vi.spyOn(fs, "openSync").mockReturnValue(42); - const close = vi.spyOn(fs, "closeSync").mockImplementation(() => {}); - const unlink = vi.spyOn(fs, "unlinkSync").mockImplementation(() => {}); - vi.spyOn(fs, "readFileSync").mockImplementation(() => reads.shift() ?? ""); - vi.spyOn(fs, "writeFileSync").mockImplementation(() => {}); - const remove = vi.spyOn(fs, "rmSync").mockImplementation(() => {}); - return { close, remove, unlink }; + function mockOwnerStartTime(): void { + const fields = ["S", ...Array.from({ length: 18 }, () => "0"), "12345"]; + vi.spyOn(fs, "readFileSync").mockReturnValue( + `${process.pid} (node fixture) ${fields.join(" ")}`, + ); } it("passes the direct binding credential only to the inference-set command", () => { @@ -61,22 +60,8 @@ describe("compatible Anthropic inference switch setup", () => { ).toThrow("COMPATIBLE_ANTHROPIC_API_KEY is required"); }); - it("maps the sandbox host alias to loopback for host-side verification", () => { - expect(hostVerificationHostsFile("127.0.0.1 localhost\n")).toBe( - "127.0.0.1 host.openshell.internal\n127.0.0.1 localhost\n", - ); - }); - - it("restores the host resolver file when verification fails", async () => { - const originalHosts = "127.0.0.1 localhost\n"; - const mappedHosts = hostVerificationHostsFile(originalHosts); - const { close, remove, unlink } = mockHostsFixture([ - originalHosts, - mappedHosts, - mappedHosts, - mappedHosts, - originalHosts, - ]); + it("removes its host alias when verification fails", async () => { + mockOwnerStartTime(); const command = vi.fn().mockResolvedValue({ exitCode: 0, stderr: "", stdout: "" }); const trackDisposable = vi.fn(); @@ -91,96 +76,39 @@ describe("compatible Anthropic inference switch setup", () => { ).rejects.toThrow("verification failed"); expect(command).toHaveBeenCalledTimes(2); - expect( - command.mock.calls.map(([program, args]) => [program, args?.[0], args?.at(-2), args?.at(-1)]), - ).toEqual([ - [ - "sudo", - "bash", - "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.original", - "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.mapped", - ], - [ - "sudo", - "bash", - "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.mapped", - "/tmp/nemoclaw-compatible-endpoint-hosts-test/hosts.original", - ], + expect(command.mock.calls.map(([program, args]) => [program, args?.[0], args?.[4]])).toEqual([ + ["sudo", "bash", "add"], + ["sudo", "bash", "remove"], ]); expect(trackDisposable.mock.invocationCallOrder[0]).toBeLessThan( command.mock.invocationCallOrder[0] ?? Number.POSITIVE_INFINITY, ); - expect(remove).toHaveBeenCalledWith("/tmp/nemoclaw-compatible-endpoint-hosts-test", { - force: true, - recursive: true, - }); - expect(close).toHaveBeenCalledWith(42); - expect(unlink).toHaveBeenCalledTimes(1); - }); - - it("does not discard a concurrent host resolver update", async () => { - const originalHosts = "127.0.0.1 localhost\n"; - const mappedHosts = hostVerificationHostsFile(originalHosts); - const concurrentHosts = `${mappedHosts}192.0.2.10 concurrent.example.test\n`; - const { close, remove, unlink } = mockHostsFixture([ - originalHosts, - mappedHosts, - mappedHosts, - concurrentHosts, - ]); - const command = vi.fn().mockResolvedValue({ exitCode: 0, stderr: "", stdout: "" }); - - await expect( - withHostVerificationLoopbackAlias( - { command } as unknown as HostCliClient, - { trackDisposable: vi.fn() }, - async () => undefined, - ), - ).rejects.toThrow("refusing to overwrite concurrent resolver state"); - - expect(command).toHaveBeenCalledTimes(1); - expect(remove).not.toHaveBeenCalled(); - expect(close).toHaveBeenCalledWith(42); - expect(unlink).toHaveBeenCalledTimes(1); }); - it("reports a failed owned resolver restoration", async () => { - const originalHosts = "127.0.0.1 localhost\n"; - const mappedHosts = hostVerificationHostsFile(originalHosts); - const { close, remove, unlink } = mockHostsFixture([ - originalHosts, - mappedHosts, - mappedHosts, - mappedHosts, - ]); + it("retries a failed owned-alias removal through tracked cleanup", async () => { + mockOwnerStartTime(); const command = vi .fn() .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "" }) - .mockResolvedValueOnce({ exitCode: 1, stderr: "permission denied", stdout: "" }); + .mockResolvedValueOnce({ exitCode: 1, stderr: "permission denied", stdout: "" }) + .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "" }); + const trackDisposable = vi.fn(); await expect( withHostVerificationLoopbackAlias( { command } as unknown as HostCliClient, - { trackDisposable: vi.fn() }, + { trackDisposable }, async () => undefined, ), - ).rejects.toThrow("could not restore /etc/hosts: permission denied"); + ).rejects.toThrow("could not remove the host verifier alias: permission denied"); - expect(command).toHaveBeenCalledTimes(2); - expect(remove).not.toHaveBeenCalled(); - expect(close).toHaveBeenCalledWith(42); - expect(unlink).toHaveBeenCalledTimes(1); + const trackedCleanup = trackDisposable.mock.calls[0]?.[1] as () => Promise; + await expect(trackedCleanup()).resolves.toBeUndefined(); + expect(command.mock.calls.map(([, args]) => args?.[4])).toEqual(["add", "remove", "remove"]); }); - it("restores an observed mapping after the mapping runner fails", async () => { - const originalHosts = "127.0.0.1 localhost\n"; - const mappedHosts = hostVerificationHostsFile(originalHosts); - const { close, remove, unlink } = mockHostsFixture([ - originalHosts, - mappedHosts, - mappedHosts, - originalHosts, - ]); + it("removes a possible alias after the mapping runner disconnects", async () => { + mockOwnerStartTime(); const command = vi .fn() .mockRejectedValueOnce(new Error("mapping runner disconnected")) @@ -195,12 +123,7 @@ describe("compatible Anthropic inference switch setup", () => { ).rejects.toThrow("mapping runner disconnected"); expect(command).toHaveBeenCalledTimes(2); - expect(remove).toHaveBeenCalledWith("/tmp/nemoclaw-compatible-endpoint-hosts-test", { - force: true, - recursive: true, - }); - expect(close).toHaveBeenCalledWith(42); - expect(unlink).toHaveBeenCalledTimes(1); + expect(command.mock.calls.map(([, args]) => args?.[4])).toEqual(["add", "remove"]); }); it("requires the direct provider to be absent before inference set owns its creation", async () => { @@ -252,3 +175,121 @@ describe("compatible Anthropic inference switch setup", () => { ); }); }); + +const linuxIt = process.platform === "linux" ? it : it.skip; + +describe("host verifier alias file ownership", () => { + function processStartTime(pid: number): string { + const stat = fs.readFileSync(`/proc/${pid}/stat`, "utf8"); + const close = stat.lastIndexOf(") "); + const fields = stat.slice(close + 2).trim().split(/\s+/u); + const startTime = fields[19]; + if (!startTime) throw new Error(`could not read process start time for ${pid}`); + return startTime; + } + + function runAliasScript( + operation: "add" | "remove", + hostsPath: string, + lockPath: string, + owner: { pid: number; startTime: string; token: string }, + ): void { + const result = spawnSync( + "bash", + [ + "-ceu", + HOST_VERIFICATION_ALIAS_SCRIPT, + "host-verifier-alias-test", + operation, + hostsPath, + lockPath, + String(owner.pid), + owner.startTime, + owner.token, + ], + { encoding: "utf8" }, + ); + expect(result.status, result.stderr).toBe(0); + } + + function testFiles(): { directory: string; hostsPath: string; lockPath: string } { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-host-verifier-test-")); + const hostsPath = path.join(directory, "hosts"); + const lockPath = path.join(directory, "hosts.lock"); + fs.writeFileSync(hostsPath, "127.0.0.1 localhost\n", { mode: 0o644 }); + return { directory, hostsPath, lockPath }; + } + + linuxIt("preserves a concurrent resolver update while removing its owned alias", () => { + const files = testFiles(); + const owner = { + pid: process.pid, + startTime: processStartTime(process.pid), + token: "a".repeat(32), + }; + try { + runAliasScript("add", files.hostsPath, files.lockPath, owner); + fs.appendFileSync(files.hostsPath, "192.0.2.10 concurrent.example.test\n"); + runAliasScript("remove", files.hostsPath, files.lockPath, owner); + + expect(fs.readFileSync(files.hostsPath, "utf8")).toBe( + "127.0.0.1 localhost\n192.0.2.10 concurrent.example.test\n", + ); + } finally { + fs.rmSync(files.directory, { force: true, recursive: true }); + } + }); + + linuxIt("serializes active owners through a persistent kernel-lock file", () => { + const files = testFiles(); + const startTime = processStartTime(process.pid); + const first = { pid: process.pid, startTime, token: "b".repeat(32) }; + const second = { pid: process.pid, startTime, token: "c".repeat(32) }; + try { + fs.writeFileSync(files.lockPath, "stale lock inode\n", { mode: 0o600 }); + runAliasScript("add", files.hostsPath, files.lockPath, first); + runAliasScript("add", files.hostsPath, files.lockPath, second); + runAliasScript("remove", files.hostsPath, files.lockPath, first); + expect(fs.readFileSync(files.hostsPath, "utf8")).toContain(second.token); + runAliasScript("remove", files.hostsPath, files.lockPath, second); + expect(fs.readFileSync(files.hostsPath, "utf8")).not.toContain( + "host.openshell.internal", + ); + } finally { + fs.rmSync(files.directory, { force: true, recursive: true }); + } + }); + + linuxIt("removes an alias whose owner process was killed", async () => { + const files = testFiles(); + const killed = spawn("sleep", ["30"], { stdio: "ignore" }); + if (!killed.pid) throw new Error("could not start killed-owner fixture"); + const killedExit = new Promise((resolve) => killed.once("exit", () => resolve())); + const killedOwner = { + pid: killed.pid, + startTime: processStartTime(killed.pid), + token: "d".repeat(32), + }; + const currentOwner = { + pid: process.pid, + startTime: processStartTime(process.pid), + token: "e".repeat(32), + }; + try { + runAliasScript("add", files.hostsPath, files.lockPath, killedOwner); + killed.kill("SIGKILL"); + await killedExit; + runAliasScript("add", files.hostsPath, files.lockPath, currentOwner); + const recovered = fs.readFileSync(files.hostsPath, "utf8"); + expect(recovered).not.toContain(killedOwner.token); + expect(recovered).toContain(currentOwner.token); + runAliasScript("remove", files.hostsPath, files.lockPath, currentOwner); + expect(fs.readFileSync(files.hostsPath, "utf8")).not.toContain( + "host.openshell.internal", + ); + } finally { + killed.kill("SIGKILL"); + fs.rmSync(files.directory, { force: true, recursive: true }); + } + }); +}); From f8b48408fa7e19b6461291df0beadcd026c207eb Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 18:51:35 -0700 Subject: [PATCH 08/23] test(e2e): keep alias recovery fixtures linear --- test/e2e/support/compatible-anthropic-switch.test.ts | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/test/e2e/support/compatible-anthropic-switch.test.ts b/test/e2e/support/compatible-anthropic-switch.test.ts index e8584414198..35a941a9226 100644 --- a/test/e2e/support/compatible-anthropic-switch.test.ts +++ b/test/e2e/support/compatible-anthropic-switch.test.ts @@ -184,8 +184,8 @@ describe("host verifier alias file ownership", () => { const close = stat.lastIndexOf(") "); const fields = stat.slice(close + 2).trim().split(/\s+/u); const startTime = fields[19]; - if (!startTime) throw new Error(`could not read process start time for ${pid}`); - return startTime; + expect(startTime, `process start time for ${pid}`).toBeDefined(); + return startTime as string; } function runAliasScript( @@ -263,11 +263,12 @@ describe("host verifier alias file ownership", () => { linuxIt("removes an alias whose owner process was killed", async () => { const files = testFiles(); const killed = spawn("sleep", ["30"], { stdio: "ignore" }); - if (!killed.pid) throw new Error("could not start killed-owner fixture"); + expect(killed.pid, "killed-owner fixture PID").toBeDefined(); + const killedPid = killed.pid as number; const killedExit = new Promise((resolve) => killed.once("exit", () => resolve())); const killedOwner = { - pid: killed.pid, - startTime: processStartTime(killed.pid), + pid: killedPid, + startTime: processStartTime(killedPid), token: "d".repeat(32), }; const currentOwner = { From ed685f4fb8f72ffcaf47e59965a7b6d90f8a4f77 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 21:07:30 -0700 Subject: [PATCH 09/23] fix(e2e): scope host resolver alias Signed-off-by: Carlos Villela --- .../fixtures/compatible-anthropic-switch.ts | 196 ++++---------- test/e2e/live/hermes-inference-switch.test.ts | 6 +- .../live/openclaw-inference-switch.test.ts | 12 +- .../compatible-anthropic-switch.test.ts | 247 ++++++------------ 4 files changed, 146 insertions(+), 315 deletions(-) diff --git a/test/e2e/fixtures/compatible-anthropic-switch.ts b/test/e2e/fixtures/compatible-anthropic-switch.ts index 53cd84b887f..06c914783a2 100644 --- a/test/e2e/fixtures/compatible-anthropic-switch.ts +++ b/test/e2e/fixtures/compatible-anthropic-switch.ts @@ -1,9 +1,6 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { randomBytes } from "node:crypto"; -import fs from "node:fs"; - import type { HostCliClient } from "./clients/host.ts"; import { resultText } from "./clients/index.ts"; @@ -11,85 +8,37 @@ export const COMPATIBLE_ANTHROPIC_PROVIDER = "compatible-anthropic-endpoint"; export const COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV = "COMPATIBLE_ANTHROPIC_API_KEY"; const DEFAULT_COMPATIBLE_ANTHROPIC_CREDENTIAL = "test-compatible-anthropic-key"; const HOST_VERIFICATION_HOSTS_PATH = "/etc/hosts"; -const HOST_VERIFICATION_LOCK_PATH = "/run/lock/nemoclaw-compatible-endpoint-hosts.lock"; -const HOST_VERIFICATION_COMMAND_TIMEOUT_MS = 60_000; -export const HOST_VERIFICATION_ALIAS_SCRIPT = [ +export const HOST_VERIFICATION_NAMESPACE_SCRIPT = [ "set -euo pipefail", "", - 'operation="$1"', - 'hosts_path="$2"', - 'lock_path="$3"', - 'owner_pid="$4"', - 'owner_start="$5"', - 'owner_token="$6"', + 'hosts_path="$1"', + 'run_uid="$2"', + 'run_gid="$3"', + "shift 3", 'alias_name="host.openshell.internal"', "", - 'case "$operation" in', - " add | remove) ;;", - ' *) echo "unsupported host verifier alias operation: $operation" >&2; exit 2 ;;', - "esac", - '[[ "$owner_pid" =~ ^[1-9][0-9]*$ ]] || { echo "invalid host verifier owner PID" >&2; exit 2; }', - '[[ "$owner_start" =~ ^[1-9][0-9]*$ ]] || { echo "invalid host verifier owner start time" >&2; exit 2; }', - '[[ "$owner_token" =~ ^[a-f0-9]{32}$ ]] || { echo "invalid host verifier owner token" >&2; exit 2; }', + '[[ "$run_uid" =~ ^[0-9]+$ ]] || { echo "invalid host verifier user ID" >&2; exit 2; }', + '[[ "$run_gid" =~ ^[0-9]+$ ]] || { echo "invalid host verifier group ID" >&2; exit 2; }', + '[[ "$#" -gt 0 ]] || { echo "host verifier command is required" >&2; exit 2; }', '[[ -f "$hosts_path" && ! -L "$hosts_path" ]] || { echo "host resolver file is not a regular file" >&2; exit 2; }', - '[[ ! -L "$lock_path" ]] || { echo "host resolver lock path must not be a symbolic link" >&2; exit 2; }', - 'command -v flock >/dev/null 2>&1 || { echo "flock is required for host resolver fixture ownership" >&2; exit 2; }', - "", - "owner_is_alive() {", - ' local pid="$1" expected_start="$2" stat remainder', - ' [[ -r "/proc/${pid}/stat" ]] || return 1', - ' IFS= read -r stat < "/proc/${pid}/stat"', - ' remainder="${stat##*) }"', - " set -- $remainder", - ' [[ "${20:-}" == "$expected_start" ]]', - "}", - "", - 'if [[ "$operation" == "add" ]] && ! owner_is_alive "$owner_pid" "$owner_start"; then', - ' echo "host verifier owner process is not alive" >&2', - " exit 2", - "fi", + 'command -v mount >/dev/null 2>&1 || { echo "mount is required for scoped host resolution" >&2; exit 2; }', + 'command -v setpriv >/dev/null 2>&1 || { echo "setpriv is required for scoped host resolution" >&2; exit 2; }', "", - "umask 077", - 'exec 9>>"$lock_path"', - 'chmod 0600 "$lock_path"', - 'flock -x -w 30 9 || { echo "timed out waiting for host resolver fixture ownership" >&2; exit 3; }', + 'private_hosts="$(mktemp)"', + "trap 'rm -f \"$private_hosts\"' EXIT", + 'cp --preserve=mode,ownership,timestamps -- "$hosts_path" "$private_hosts"', + "printf '\\n127.0.0.1 %s\\n' \"$alias_name\" >> \"$private_hosts\"", "", - 'marker="# nemoclaw-host-verifier:${owner_pid}:${owner_start}:${owner_token}"', - 'owned_line="127.0.0.1 ${alias_name} ${marker}"', - 'snapshot="$(mktemp "${hosts_path}.nemoclaw-snapshot.XXXXXX")"', - 'replacement="$(mktemp "${hosts_path}.nemoclaw-replacement.XXXXXX")"', - "trap 'rm -f \"$snapshot\" \"$replacement\"' EXIT", - 'cp --preserve=all -- "$hosts_path" "$snapshot"', - 'cp --preserve=all -- "$hosts_path" "$replacement"', - ': > "$replacement"', - "", - 'while IFS= read -r line || [[ -n "$line" ]]; do', - ' if [[ "$line" == "$owned_line" ]]; then', - " continue", - " fi", - ' if [[ "$line" =~ ^127\\.0\\.0\\.1[[:space:]]+host\\.openshell\\.internal[[:space:]]+#[[:space:]]nemoclaw-host-verifier:([1-9][0-9]*):([1-9][0-9]*):([a-f0-9]{32})$ ]]; then', - ' if owner_is_alive "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}"; then', - " printf '%s\\n' \"$line\" >> \"$replacement\"", - " fi", - " continue", - " fi", - " printf '%s\\n' \"$line\" >> \"$replacement\"", - 'done < "$snapshot"', - "", - 'if [[ "$operation" == "add" ]]; then', - " printf '%s\\n' \"$owned_line\" >> \"$replacement\"", - "fi", + "# The private mount keeps the fixture alias inside this command's mount namespace.", + "# The host resolver file remains available to unrelated writers.", + 'mount --make-rprivate /', + 'mount --bind "$private_hosts" "$hosts_path"', "", - "# The kernel lock serializes every cooperative fixture writer. Replacing the", - "# file in one rename prevents readers from observing a partial resolver file.", - 'mv -f -- "$replacement" "$hosts_path"', - "trap 'rm -f \"$snapshot\"' EXIT", - "", - 'if [[ "$operation" == "add" ]]; then', - ' grep -Fqx -- "$owned_line" "$hosts_path" || { echo "host verifier alias was not installed" >&2; exit 4; }', + 'if [[ "$(id -u)" == "$run_uid" && "$(id -g)" == "$run_gid" ]]; then', + ' "$@"', "else", - ' ! grep -Fqx -- "$owned_line" "$hosts_path" || { echo "host verifier alias was not removed" >&2; exit 4; }', + ' setpriv --reuid="$run_uid" --regid="$run_gid" --init-groups -- "$@"', "fi", ].join("\n"); @@ -124,80 +73,43 @@ export function compatibleAnthropicSwitchEnv( return binding ? { [COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV]: binding.credentialValue } : {}; } -function hostVerificationOwnerStartTime(): string { - const stat = fs.readFileSync(`/proc/${process.pid}/stat`, "utf8"); - const close = stat.lastIndexOf(") "); - const fields = close >= 0 ? stat.slice(close + 2).trim().split(/\s+/u) : []; - const startTime = fields[19]; - if (!startTime || !/^[1-9][0-9]*$/u.test(startTime)) { - throw new Error("could not read the host verifier owner process start time"); - } - return startTime; -} - -interface HostVerificationOwner { - pid: number; - startTime: string; - token: string; -} - -function createHostVerificationOwner(): HostVerificationOwner { - return { - pid: process.pid, - startTime: hostVerificationOwnerStartTime(), - token: randomBytes(16).toString("hex"), - }; -} - -async function updateHostVerificationAlias( - host: HostCliClient, - operation: "add" | "remove", - owner: HostVerificationOwner, -): Promise { - const artifactName = `${operation === "add" ? "map" : "restore"}-host-verifier-alias`; - const result = await host.command( - "sudo", - [ - "bash", - "-ceu", - HOST_VERIFICATION_ALIAS_SCRIPT, - artifactName, - operation, - HOST_VERIFICATION_HOSTS_PATH, - HOST_VERIFICATION_LOCK_PATH, - String(owner.pid), - owner.startTime, - owner.token, - ], - { artifactName, timeoutMs: HOST_VERIFICATION_COMMAND_TIMEOUT_MS }, - ); - if (result.exitCode !== 0) { - throw new Error( - `could not ${operation === "add" ? "install" : "remove"} the host verifier alias: ${resultText(result)}`, - ); - } -} - export async function withHostVerificationLoopbackAlias( host: HostCliClient, - cleanup: { trackDisposable(name: string, run: () => Promise | void): void }, - run: () => Promise, + run: (scopedHost: HostCliClient) => Promise, ): Promise { - const owner = createHostVerificationOwner(); - let restored = false; - const restore = async (): Promise => { - if (restored) return; - await updateHostVerificationAlias(host, "remove", owner); - restored = true; - }; - cleanup.trackDisposable("restore the host verifier alias mapping", restore); - - try { - await updateHostVerificationAlias(host, "add", owner); - return await run(); - } finally { - await restore(); + const uid = process.getuid?.(); + const gid = process.getgid?.(); + if (uid === undefined || gid === undefined) { + throw new Error("scoped host verification requires Linux user and group IDs"); } + const scopedHost = { + command: ( + command: string, + args: string[] = [], + options: Parameters[2] = {}, + ) => + host.command( + "sudo", + [ + "--preserve-env", + "unshare", + "--mount", + "--fork", + "--", + "bash", + "-ceu", + HOST_VERIFICATION_NAMESPACE_SCRIPT, + "host-verifier-namespace", + HOST_VERIFICATION_HOSTS_PATH, + String(uid), + String(gid), + command, + ...args, + ], + options, + ), + } as HostCliClient; + return await run(scopedHost); } export async function requireCompatibleAnthropicProviderAbsent( diff --git a/test/e2e/live/hermes-inference-switch.test.ts b/test/e2e/live/hermes-inference-switch.test.ts index ba839f378c5..2419a1b86e8 100644 --- a/test/e2e/live/hermes-inference-switch.test.ts +++ b/test/e2e/live/hermes-inference-switch.test.ts @@ -191,14 +191,14 @@ test("Hermes inference set updates route/config and preserves live runtime", { progress.phase("switch Hermes inference provider"); const compatibleMetadataArgs = compatibleAnthropicMetadataArgs(switchEndpointUrl); - const switchInference = () => - runHermesInferenceSetWithRetry(host, redactionValues, compatibleMetadataArgs, { + const switchInference = (commandHost: HostCliClient = host) => + runHermesInferenceSetWithRetry(commandHost, redactionValues, compatibleMetadataArgs, { artifacts, compatibleBinding: switchBinding, }); const switched = switchBinding && mockAnthropicSwitchEnabled() - ? await withHostVerificationLoopbackAlias(host, cleanup, switchInference) + ? await withHostVerificationLoopbackAlias(host, switchInference) : await switchInference(); expect(switched.exitCode, resultText(switched)).toBe(0); expect(resultText(switched)).not.toContain("writing the in-sandbox config failed"); diff --git a/test/e2e/live/openclaw-inference-switch.test.ts b/test/e2e/live/openclaw-inference-switch.test.ts index 06a37154ed1..6312e0e8317 100644 --- a/test/e2e/live/openclaw-inference-switch.test.ts +++ b/test/e2e/live/openclaw-inference-switch.test.ts @@ -1087,10 +1087,16 @@ test("openclaw-inference-switch: switches route and preserves live OpenClaw beha gatewayRestartExpected ? "anthropic-messages" : "openai-completions", ); const pidBefore = await openclawGatewayPid(sandbox, home); - const switchInference = () => - runOpenClawInferenceSetWithRetry(host, home, redactionValues, switchBinding, artifacts); + const switchInference = (commandHost: HostCliClient = host) => + runOpenClawInferenceSetWithRetry( + commandHost, + home, + redactionValues, + switchBinding, + artifacts, + ); const switchResult = mockProvider - ? await withHostVerificationLoopbackAlias(host, cleanup, switchInference) + ? await withHostVerificationLoopbackAlias(host, switchInference) : await switchInference(); expect(switchResult.exitCode, resultText(switchResult)).toBe(0); expect( diff --git a/test/e2e/support/compatible-anthropic-switch.test.ts b/test/e2e/support/compatible-anthropic-switch.test.ts index 35a941a9226..d1c7a219f8a 100644 --- a/test/e2e/support/compatible-anthropic-switch.test.ts +++ b/test/e2e/support/compatible-anthropic-switch.test.ts @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { spawn, spawnSync } from "node:child_process"; +import { spawnSync } from "node:child_process"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -14,7 +14,7 @@ import { COMPATIBLE_ANTHROPIC_PROVIDER, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, - HOST_VERIFICATION_ALIAS_SCRIPT, + HOST_VERIFICATION_NAMESPACE_SCRIPT, requireCompatibleAnthropicProviderAbsent, withHostVerificationLoopbackAlias, } from "../fixtures/compatible-anthropic-switch.ts"; @@ -22,13 +22,6 @@ import { describe("compatible Anthropic inference switch setup", () => { afterEach(() => vi.restoreAllMocks()); - function mockOwnerStartTime(): void { - const fields = ["S", ...Array.from({ length: 18 }, () => "0"), "12345"]; - vi.spyOn(fs, "readFileSync").mockReturnValue( - `${process.pid} (node fixture) ${fields.join(" ")}`, - ); - } - it("passes the direct binding credential only to the inference-set command", () => { const binding = compatibleAnthropicSwitchBinding("http://host.openshell.internal:18766", { COMPATIBLE_ANTHROPIC_API_KEY: "fixture-key", @@ -60,70 +53,45 @@ describe("compatible Anthropic inference switch setup", () => { ).toThrow("COMPATIBLE_ANTHROPIC_API_KEY is required"); }); - it("removes its host alias when verification fails", async () => { - mockOwnerStartTime(); - const command = vi.fn().mockResolvedValue({ exitCode: 0, stderr: "", stdout: "" }); - const trackDisposable = vi.fn(); + it("runs host verification inside a private resolver mount namespace", async () => { + const result = { exitCode: 0, stderr: "", stdout: "verified" }; + const command = vi.fn().mockResolvedValue(result); + const commandEnv = { COMPATIBLE_ANTHROPIC_API_KEY: "fixture-key" }; await expect( withHostVerificationLoopbackAlias( { command } as unknown as HostCliClient, - { trackDisposable }, - async () => { - throw new Error("verification failed"); - }, + (scopedHost) => + scopedHost.command("node", ["nemoclaw.js", "inference", "set"], { + artifactName: "inference-set", + env: commandEnv, + redactionValues: ["fixture-key"], + }), ), - ).rejects.toThrow("verification failed"); - - expect(command).toHaveBeenCalledTimes(2); - expect(command.mock.calls.map(([program, args]) => [program, args?.[0], args?.[4]])).toEqual([ - ["sudo", "bash", "add"], - ["sudo", "bash", "remove"], - ]); - expect(trackDisposable.mock.invocationCallOrder[0]).toBeLessThan( - command.mock.invocationCallOrder[0] ?? Number.POSITIVE_INFINITY, + ).resolves.toBe(result); + + expect(command).toHaveBeenCalledOnce(); + const [program, args, options] = command.mock.calls[0] ?? []; + expect(program).toBe("sudo"); + expect(args).toEqual( + expect.arrayContaining([ + "--preserve-env", + "unshare", + "--mount", + "--fork", + "/etc/hosts", + "node", + "nemoclaw.js", + "inference", + "set", + ]), ); - }); - - it("retries a failed owned-alias removal through tracked cleanup", async () => { - mockOwnerStartTime(); - const command = vi - .fn() - .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "" }) - .mockResolvedValueOnce({ exitCode: 1, stderr: "permission denied", stdout: "" }) - .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "" }); - const trackDisposable = vi.fn(); - - await expect( - withHostVerificationLoopbackAlias( - { command } as unknown as HostCliClient, - { trackDisposable }, - async () => undefined, - ), - ).rejects.toThrow("could not remove the host verifier alias: permission denied"); - - const trackedCleanup = trackDisposable.mock.calls[0]?.[1] as () => Promise; - await expect(trackedCleanup()).resolves.toBeUndefined(); - expect(command.mock.calls.map(([, args]) => args?.[4])).toEqual(["add", "remove", "remove"]); - }); - - it("removes a possible alias after the mapping runner disconnects", async () => { - mockOwnerStartTime(); - const command = vi - .fn() - .mockRejectedValueOnce(new Error("mapping runner disconnected")) - .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "" }); - - await expect( - withHostVerificationLoopbackAlias( - { command } as unknown as HostCliClient, - { trackDisposable: vi.fn() }, - async () => undefined, - ), - ).rejects.toThrow("mapping runner disconnected"); - - expect(command).toHaveBeenCalledTimes(2); - expect(command.mock.calls.map(([, args]) => args?.[4])).toEqual(["add", "remove"]); + expect(args).not.toContain("fixture-key"); + expect(options).toEqual({ + artifactName: "inference-set", + env: commandEnv, + redactionValues: ["fixture-key"], + }); }); it("requires the direct provider to be absent before inference set owns its creation", async () => { @@ -178,118 +146,63 @@ describe("compatible Anthropic inference switch setup", () => { const linuxIt = process.platform === "linux" ? it : it.skip; -describe("host verifier alias file ownership", () => { - function processStartTime(pid: number): string { - const stat = fs.readFileSync(`/proc/${pid}/stat`, "utf8"); - const close = stat.lastIndexOf(") "); - const fields = stat.slice(close + 2).trim().split(/\s+/u); - const startTime = fields[19]; - expect(startTime, `process start time for ${pid}`).toBeDefined(); - return startTime as string; - } - - function runAliasScript( - operation: "add" | "remove", - hostsPath: string, - lockPath: string, - owner: { pid: number; startTime: string; token: string }, - ): void { - const result = spawnSync( - "bash", - [ - "-ceu", - HOST_VERIFICATION_ALIAS_SCRIPT, - "host-verifier-alias-test", - operation, - hostsPath, - lockPath, - String(owner.pid), - owner.startTime, - owner.token, - ], - { encoding: "utf8" }, - ); - expect(result.status, result.stderr).toBe(0); - } - - function testFiles(): { directory: string; hostsPath: string; lockPath: string } { +describe("host verifier resolver namespace", () => { + function testFiles(): { directory: string; hostsPath: string; capturedPath: string } { const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-host-verifier-test-")); const hostsPath = path.join(directory, "hosts"); - const lockPath = path.join(directory, "hosts.lock"); + const capturedPath = path.join(directory, "hosts.private"); fs.writeFileSync(hostsPath, "127.0.0.1 localhost\n", { mode: 0o644 }); - return { directory, hostsPath, lockPath }; + return { directory, hostsPath, capturedPath }; } - linuxIt("preserves a concurrent resolver update while removing its owned alias", () => { + linuxIt("preserves an unrelated resolver write during private mount setup (#9166)", () => { const files = testFiles(); - const owner = { - pid: process.pid, - startTime: processStartTime(process.pid), - token: "a".repeat(32), - }; + const fakeBin = path.join(files.directory, "bin"); + const fakeMount = path.join(fakeBin, "mount"); try { - runAliasScript("add", files.hostsPath, files.lockPath, owner); - fs.appendFileSync(files.hostsPath, "192.0.2.10 concurrent.example.test\n"); - runAliasScript("remove", files.hostsPath, files.lockPath, owner); - - expect(fs.readFileSync(files.hostsPath, "utf8")).toBe( - "127.0.0.1 localhost\n192.0.2.10 concurrent.example.test\n", + fs.mkdirSync(fakeBin); + fs.writeFileSync( + fakeMount, + [ + "#!/usr/bin/env bash", + "set -euo pipefail", + '[[ "$1" == "--make-rprivate" ]] && exit 0', + '[[ "$1" == "--bind" ]]', + "printf '192.0.2.10 concurrent.example.test\\n' >> \"$NEMOCLAW_TEST_RESOLVER_SOURCE\"", + 'cp -- "$2" "$NEMOCLAW_TEST_RESOLVER_COPY"', + ].join("\n"), + { mode: 0o755 }, ); - } finally { - fs.rmSync(files.directory, { force: true, recursive: true }); - } - }); - - linuxIt("serializes active owners through a persistent kernel-lock file", () => { - const files = testFiles(); - const startTime = processStartTime(process.pid); - const first = { pid: process.pid, startTime, token: "b".repeat(32) }; - const second = { pid: process.pid, startTime, token: "c".repeat(32) }; - try { - fs.writeFileSync(files.lockPath, "stale lock inode\n", { mode: 0o600 }); - runAliasScript("add", files.hostsPath, files.lockPath, first); - runAliasScript("add", files.hostsPath, files.lockPath, second); - runAliasScript("remove", files.hostsPath, files.lockPath, first); - expect(fs.readFileSync(files.hostsPath, "utf8")).toContain(second.token); - runAliasScript("remove", files.hostsPath, files.lockPath, second); - expect(fs.readFileSync(files.hostsPath, "utf8")).not.toContain( - "host.openshell.internal", + const result = spawnSync( + "bash", + [ + "-ceu", + HOST_VERIFICATION_NAMESPACE_SCRIPT, + "host-verifier-namespace-test", + files.hostsPath, + String(process.getuid?.()), + String(process.getgid?.()), + "true", + ], + { + encoding: "utf8", + env: { + ...process.env, + NEMOCLAW_TEST_RESOLVER_COPY: files.capturedPath, + NEMOCLAW_TEST_RESOLVER_SOURCE: files.hostsPath, + PATH: `${fakeBin}:${process.env.PATH ?? ""}`, + }, + }, ); - } finally { - fs.rmSync(files.directory, { force: true, recursive: true }); - } - }); + expect(result.status, result.stderr).toBe(0); - linuxIt("removes an alias whose owner process was killed", async () => { - const files = testFiles(); - const killed = spawn("sleep", ["30"], { stdio: "ignore" }); - expect(killed.pid, "killed-owner fixture PID").toBeDefined(); - const killedPid = killed.pid as number; - const killedExit = new Promise((resolve) => killed.once("exit", () => resolve())); - const killedOwner = { - pid: killedPid, - startTime: processStartTime(killedPid), - token: "d".repeat(32), - }; - const currentOwner = { - pid: process.pid, - startTime: processStartTime(process.pid), - token: "e".repeat(32), - }; - try { - runAliasScript("add", files.hostsPath, files.lockPath, killedOwner); - killed.kill("SIGKILL"); - await killedExit; - runAliasScript("add", files.hostsPath, files.lockPath, currentOwner); - const recovered = fs.readFileSync(files.hostsPath, "utf8"); - expect(recovered).not.toContain(killedOwner.token); - expect(recovered).toContain(currentOwner.token); - runAliasScript("remove", files.hostsPath, files.lockPath, currentOwner); - expect(fs.readFileSync(files.hostsPath, "utf8")).not.toContain( - "host.openshell.internal", + expect(fs.readFileSync(files.hostsPath, "utf8")).toBe( + "127.0.0.1 localhost\n192.0.2.10 concurrent.example.test\n", ); + const privateResolver = fs.readFileSync(files.capturedPath, "utf8"); + expect(privateResolver).toContain("127.0.0.1 host.openshell.internal"); + expect(privateResolver).not.toContain("concurrent.example.test"); } finally { - killed.kill("SIGKILL"); fs.rmSync(files.directory, { force: true, recursive: true }); } }); From 0167b860af623c7f757d87d2e06de3c50f367eec Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 21:08:51 -0700 Subject: [PATCH 10/23] fix(e2e): import scoped host type Signed-off-by: Carlos Villela --- test/e2e/live/hermes-inference-switch.test.ts | 1 + 1 file changed, 1 insertion(+) diff --git a/test/e2e/live/hermes-inference-switch.test.ts b/test/e2e/live/hermes-inference-switch.test.ts index 2419a1b86e8..80c788aebbf 100644 --- a/test/e2e/live/hermes-inference-switch.test.ts +++ b/test/e2e/live/hermes-inference-switch.test.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; +import type { HostCliClient } from "../fixtures/clients/host.ts"; import { resultText } from "../fixtures/clients/index.ts"; import { trustedSandboxShellScript } from "../fixtures/clients/sandbox.ts"; import { withHostVerificationLoopbackAlias } from "../fixtures/compatible-anthropic-switch.ts"; From bc7c9c9906be2ae62fef0fee6cee75a42981a9d7 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 21:11:55 -0700 Subject: [PATCH 11/23] fix(e2e): constrain resolver command environment Signed-off-by: Carlos Villela --- .../fixtures/compatible-anthropic-switch.ts | 33 ++++++++++++++----- .../compatible-anthropic-switch.test.ts | 5 +-- 2 files changed, 28 insertions(+), 10 deletions(-) diff --git a/test/e2e/fixtures/compatible-anthropic-switch.ts b/test/e2e/fixtures/compatible-anthropic-switch.ts index 06c914783a2..172d6e95437 100644 --- a/test/e2e/fixtures/compatible-anthropic-switch.ts +++ b/test/e2e/fixtures/compatible-anthropic-switch.ts @@ -15,7 +15,8 @@ export const HOST_VERIFICATION_NAMESPACE_SCRIPT = [ 'hosts_path="$1"', 'run_uid="$2"', 'run_gid="$3"', - "shift 3", + 'command_path="$4"', + "shift 4", 'alias_name="host.openshell.internal"', "", '[[ "$run_uid" =~ ^[0-9]+$ ]] || { echo "invalid host verifier user ID" >&2; exit 2; }', @@ -36,12 +37,22 @@ export const HOST_VERIFICATION_NAMESPACE_SCRIPT = [ 'mount --bind "$private_hosts" "$hosts_path"', "", 'if [[ "$(id -u)" == "$run_uid" && "$(id -g)" == "$run_gid" ]]; then', - ' "$@"', + ' /usr/bin/env PATH="$command_path" "$@"', "else", - ' setpriv --reuid="$run_uid" --regid="$run_gid" --init-groups -- "$@"', + ' /usr/bin/setpriv --reuid="$run_uid" --regid="$run_gid" --init-groups -- /usr/bin/env PATH="$command_path" "$@"', "fi", ].join("\n"); +function hostVerificationPreservedEnvNames(env: NodeJS.ProcessEnv | undefined): string[] { + return Object.keys(env ?? {}) + .filter((name) => + /^(?:CI|COMPATIBLE_ANTHROPIC_API_KEY|GITHUB_ACTIONS|HOME|LANG|LC_ALL|LC_CTYPE|LOGNAME|NEMOCLAW_[A-Z0-9_]+|OPENSHELL_GATEWAY|RUNNER_OS|RUNNER_TEMP|TERM|TMPDIR|TZ|USER)$/u.test( + name, + ), + ) + .sort(); +} + export interface CompatibleAnthropicSwitchBinding { endpointUrl: string; credentialValue: string; @@ -87,11 +98,15 @@ export async function withHostVerificationLoopbackAlias( command: string, args: string[] = [], options: Parameters[2] = {}, - ) => - host.command( + ) => { + const preservedEnvNames = hostVerificationPreservedEnvNames(options.env); + const preserveEnvOption = `--preserve-env=${preservedEnvNames.join(",")}`; + const commandPath = options.env?.PATH ?? process.env.PATH ?? "/usr/bin:/bin"; + const scopedCommand = command === "node" ? process.execPath : command; + return host.command( "sudo", [ - "--preserve-env", + preserveEnvOption, "unshare", "--mount", "--fork", @@ -103,11 +118,13 @@ export async function withHostVerificationLoopbackAlias( HOST_VERIFICATION_HOSTS_PATH, String(uid), String(gid), - command, + commandPath, + scopedCommand, ...args, ], options, - ), + ); + }, } as HostCliClient; return await run(scopedHost); } diff --git a/test/e2e/support/compatible-anthropic-switch.test.ts b/test/e2e/support/compatible-anthropic-switch.test.ts index d1c7a219f8a..f789f562abe 100644 --- a/test/e2e/support/compatible-anthropic-switch.test.ts +++ b/test/e2e/support/compatible-anthropic-switch.test.ts @@ -75,17 +75,17 @@ describe("compatible Anthropic inference switch setup", () => { expect(program).toBe("sudo"); expect(args).toEqual( expect.arrayContaining([ - "--preserve-env", "unshare", "--mount", "--fork", "/etc/hosts", - "node", + process.execPath, "nemoclaw.js", "inference", "set", ]), ); + expect(args).toContain("--preserve-env=COMPATIBLE_ANTHROPIC_API_KEY"); expect(args).not.toContain("fixture-key"); expect(options).toEqual({ artifactName: "inference-set", @@ -182,6 +182,7 @@ describe("host verifier resolver namespace", () => { files.hostsPath, String(process.getuid?.()), String(process.getgid?.()), + process.env.PATH ?? "/usr/bin:/bin", "true", ], { From d7d804232d65d0b19c5a588886db705c8c02c979 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 21:20:52 -0700 Subject: [PATCH 12/23] fix(e2e): keep auth failures terminal Signed-off-by: Carlos Villela --- .../e2e/live/cloud-inference-provider-skip.ts | 11 +++++++--- .../cloud-inference-provider-skip.test.ts | 20 +++++++++++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/test/e2e/live/cloud-inference-provider-skip.ts b/test/e2e/live/cloud-inference-provider-skip.ts index 3a265c57b64..7a00233a087 100644 --- a/test/e2e/live/cloud-inference-provider-skip.ts +++ b/test/e2e/live/cloud-inference-provider-skip.ts @@ -51,10 +51,15 @@ export function classifyCloudChatFailure( error: unknown, timedOut = false, ): RetryFailureClass { - if (timedOut) return "transient-external"; + const hasNoResponseStatus = httpStatus === "" || httpStatus === "000"; const thrownDetail = error instanceof Error ? error.message : ""; - if (httpStatus === "429" || /^5\d{2}$/u.test(httpStatus)) return "transient-external"; - if (TRANSIENT_CHAT_FAILURE.test(`${transportOutput}\n${thrownDetail}`)) { + if (httpStatus === "408" || httpStatus === "429" || /^5\d{2}$/u.test(httpStatus)) { + return "transient-external"; + } + if ( + hasNoResponseStatus && + (timedOut || TRANSIENT_CHAT_FAILURE.test(`${transportOutput}\n${thrownDetail}`)) + ) { return "transient-external"; } return failure === "response was not parseable JSON" ? "malformed-input" : "deterministic"; diff --git a/test/e2e/support/cloud-inference-provider-skip.test.ts b/test/e2e/support/cloud-inference-provider-skip.test.ts index ef5a1f9ed67..2efe3fa5da6 100644 --- a/test/e2e/support/cloud-inference-provider-skip.test.ts +++ b/test/e2e/support/cloud-inference-provider-skip.test.ts @@ -22,6 +22,12 @@ describe("cloud inference pre-contract provider skip classifier", () => { expect(classifyCloudChatFailure("503", "", "expected PONG", undefined)).toBe( "transient-external", ); + expect(classifyCloudChatFailure("408", "", "expected PONG", undefined)).toBe( + "transient-external", + ); + expect(classifyCloudChatFailure("429", "", "expected PONG", undefined)).toBe( + "transient-external", + ); expect(classifyCloudChatFailure("", "", "", new Error("request ETIMEDOUT"))).toBe( "transient-external", ); @@ -38,6 +44,20 @@ describe("cloud inference pre-contract provider skip classifier", () => { ).toBe("deterministic"); }); + it("keeps authentication statuses terminal when transport text is present (#9166)", () => { + expect( + classifyCloudChatFailure("401", "request timed out", "expected PONG", undefined), + ).toBe("deterministic"); + expect( + classifyCloudChatFailure( + "403", + "failed to connect after ECONNRESET", + "expected PONG", + undefined, + ), + ).toBe("deterministic"); + }); + it("separates the curl status trailer from provider content", () => { const output = `{"message":"rate limit and HTTP 503"}${cloudChatWriteOutArg().replace("%{http_code}", "200")}`; expect(parseCloudChatResponse(output)).toEqual({ From 2219fa36448ec86faa5cb928ec6e74fa83d38102 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 21:38:22 -0700 Subject: [PATCH 13/23] fix(e2e): route mock verification through loopback Signed-off-by: Carlos Villela --- .../fixtures/compatible-anthropic-switch.ts | 94 +-------------- .../live/hermes-inference-switch-helpers.ts | 11 +- test/e2e/live/hermes-inference-switch.test.ts | 16 ++- .../live/openclaw-inference-switch.test.ts | 22 ++-- .../compatible-anthropic-switch.test.ts | 114 +----------------- ...mes-inference-switch-command-shape.test.ts | 10 -- 6 files changed, 25 insertions(+), 242 deletions(-) diff --git a/test/e2e/fixtures/compatible-anthropic-switch.ts b/test/e2e/fixtures/compatible-anthropic-switch.ts index 172d6e95437..eff457000b9 100644 --- a/test/e2e/fixtures/compatible-anthropic-switch.ts +++ b/test/e2e/fixtures/compatible-anthropic-switch.ts @@ -7,57 +7,16 @@ import { resultText } from "./clients/index.ts"; export const COMPATIBLE_ANTHROPIC_PROVIDER = "compatible-anthropic-endpoint"; export const COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV = "COMPATIBLE_ANTHROPIC_API_KEY"; const DEFAULT_COMPATIBLE_ANTHROPIC_CREDENTIAL = "test-compatible-anthropic-key"; -const HOST_VERIFICATION_HOSTS_PATH = "/etc/hosts"; - -export const HOST_VERIFICATION_NAMESPACE_SCRIPT = [ - "set -euo pipefail", - "", - 'hosts_path="$1"', - 'run_uid="$2"', - 'run_gid="$3"', - 'command_path="$4"', - "shift 4", - 'alias_name="host.openshell.internal"', - "", - '[[ "$run_uid" =~ ^[0-9]+$ ]] || { echo "invalid host verifier user ID" >&2; exit 2; }', - '[[ "$run_gid" =~ ^[0-9]+$ ]] || { echo "invalid host verifier group ID" >&2; exit 2; }', - '[[ "$#" -gt 0 ]] || { echo "host verifier command is required" >&2; exit 2; }', - '[[ -f "$hosts_path" && ! -L "$hosts_path" ]] || { echo "host resolver file is not a regular file" >&2; exit 2; }', - 'command -v mount >/dev/null 2>&1 || { echo "mount is required for scoped host resolution" >&2; exit 2; }', - 'command -v setpriv >/dev/null 2>&1 || { echo "setpriv is required for scoped host resolution" >&2; exit 2; }', - "", - 'private_hosts="$(mktemp)"', - "trap 'rm -f \"$private_hosts\"' EXIT", - 'cp --preserve=mode,ownership,timestamps -- "$hosts_path" "$private_hosts"', - "printf '\\n127.0.0.1 %s\\n' \"$alias_name\" >> \"$private_hosts\"", - "", - "# The private mount keeps the fixture alias inside this command's mount namespace.", - "# The host resolver file remains available to unrelated writers.", - 'mount --make-rprivate /', - 'mount --bind "$private_hosts" "$hosts_path"', - "", - 'if [[ "$(id -u)" == "$run_uid" && "$(id -g)" == "$run_gid" ]]; then', - ' /usr/bin/env PATH="$command_path" "$@"', - "else", - ' /usr/bin/setpriv --reuid="$run_uid" --regid="$run_gid" --init-groups -- /usr/bin/env PATH="$command_path" "$@"', - "fi", -].join("\n"); - -function hostVerificationPreservedEnvNames(env: NodeJS.ProcessEnv | undefined): string[] { - return Object.keys(env ?? {}) - .filter((name) => - /^(?:CI|COMPATIBLE_ANTHROPIC_API_KEY|GITHUB_ACTIONS|HOME|LANG|LC_ALL|LC_CTYPE|LOGNAME|NEMOCLAW_[A-Z0-9_]+|OPENSHELL_GATEWAY|RUNNER_OS|RUNNER_TEMP|TERM|TMPDIR|TZ|USER)$/u.test( - name, - ), - ) - .sort(); -} export interface CompatibleAnthropicSwitchBinding { endpointUrl: string; credentialValue: string; } +export function compatibleAnthropicMockEndpointUrl(port: number): string { + return `http://127.0.0.1:${port}`; +} + export function compatibleAnthropicSwitchBinding( endpointUrl: string, runtimeEnv: NodeJS.ProcessEnv = process.env, @@ -84,51 +43,6 @@ export function compatibleAnthropicSwitchEnv( return binding ? { [COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV]: binding.credentialValue } : {}; } -export async function withHostVerificationLoopbackAlias( - host: HostCliClient, - run: (scopedHost: HostCliClient) => Promise, -): Promise { - const uid = process.getuid?.(); - const gid = process.getgid?.(); - if (uid === undefined || gid === undefined) { - throw new Error("scoped host verification requires Linux user and group IDs"); - } - const scopedHost = { - command: ( - command: string, - args: string[] = [], - options: Parameters[2] = {}, - ) => { - const preservedEnvNames = hostVerificationPreservedEnvNames(options.env); - const preserveEnvOption = `--preserve-env=${preservedEnvNames.join(",")}`; - const commandPath = options.env?.PATH ?? process.env.PATH ?? "/usr/bin:/bin"; - const scopedCommand = command === "node" ? process.execPath : command; - return host.command( - "sudo", - [ - preserveEnvOption, - "unshare", - "--mount", - "--fork", - "--", - "bash", - "-ceu", - HOST_VERIFICATION_NAMESPACE_SCRIPT, - "host-verifier-namespace", - HOST_VERIFICATION_HOSTS_PATH, - String(uid), - String(gid), - commandPath, - scopedCommand, - ...args, - ], - options, - ); - }, - } as HostCliClient; - return await run(scopedHost); -} - export async function requireCompatibleAnthropicProviderAbsent( host: HostCliClient, options: { diff --git a/test/e2e/live/hermes-inference-switch-helpers.ts b/test/e2e/live/hermes-inference-switch-helpers.ts index 5ed48777aee..7887cd1d641 100644 --- a/test/e2e/live/hermes-inference-switch-helpers.ts +++ b/test/e2e/live/hermes-inference-switch-helpers.ts @@ -18,6 +18,7 @@ import { } from "../fixtures/clients/sandbox.ts"; import { type CompatibleAnthropicSwitchBinding, + compatibleAnthropicMockEndpointUrl, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, requireCompatibleAnthropicProviderAbsent, @@ -81,14 +82,6 @@ export function compatibleAnthropicMetadataArgs(endpointUrl: string | null): str : []; } -export function mockAnthropicEndpointUrl( - port: number, - runtimeEnv: NodeJS.ProcessEnv = process.env, -): string { - const host = runtimeEnv.NEMOCLAW_SWITCH_MOCK_HOST ?? "host.openshell.internal"; - return `http://${host}:${port}`; -} - export function openAiSurfaceEndpointUrl(endpointUrl: string): string { const trimmed = endpointUrl.replace(/\/+$/u, ""); return trimmed.endsWith("/v1") ? trimmed : `${trimmed}/v1`; @@ -536,7 +529,7 @@ async function startMockAnthropicProvider(): Promise closeServer(server), }; } diff --git a/test/e2e/live/hermes-inference-switch.test.ts b/test/e2e/live/hermes-inference-switch.test.ts index 80c788aebbf..1303125d228 100644 --- a/test/e2e/live/hermes-inference-switch.test.ts +++ b/test/e2e/live/hermes-inference-switch.test.ts @@ -2,10 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; -import type { HostCliClient } from "../fixtures/clients/host.ts"; import { resultText } from "../fixtures/clients/index.ts"; import { trustedSandboxShellScript } from "../fixtures/clients/sandbox.ts"; -import { withHostVerificationLoopbackAlias } from "../fixtures/compatible-anthropic-switch.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { startFakeOpenAiCompatibleServer } from "../fixtures/fake-openai-compatible.ts"; import { DEFAULT_HOSTED_INFERENCE_BASE_URL } from "../fixtures/hosted-inference.ts"; @@ -192,15 +190,15 @@ test("Hermes inference set updates route/config and preserves live runtime", { progress.phase("switch Hermes inference provider"); const compatibleMetadataArgs = compatibleAnthropicMetadataArgs(switchEndpointUrl); - const switchInference = (commandHost: HostCliClient = host) => - runHermesInferenceSetWithRetry(commandHost, redactionValues, compatibleMetadataArgs, { + const switched = await runHermesInferenceSetWithRetry( + host, + redactionValues, + compatibleMetadataArgs, + { artifacts, compatibleBinding: switchBinding, - }); - const switched = - switchBinding && mockAnthropicSwitchEnabled() - ? await withHostVerificationLoopbackAlias(host, switchInference) - : await switchInference(); + }, + ); expect(switched.exitCode, resultText(switched)).toBe(0); expect(resultText(switched)).not.toContain("writing the in-sandbox config failed"); expect(resultText(switched)).toContain(`Inference route synced for '${SANDBOX_NAME}'`); diff --git a/test/e2e/live/openclaw-inference-switch.test.ts b/test/e2e/live/openclaw-inference-switch.test.ts index 6312e0e8317..f0e86971259 100644 --- a/test/e2e/live/openclaw-inference-switch.test.ts +++ b/test/e2e/live/openclaw-inference-switch.test.ts @@ -26,10 +26,10 @@ import { } from "../fixtures/clients/sandbox.ts"; import { type CompatibleAnthropicSwitchBinding, + compatibleAnthropicMockEndpointUrl, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, requireCompatibleAnthropicProviderAbsent, - withHostVerificationLoopbackAlias, } from "../fixtures/compatible-anthropic-switch.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { @@ -428,7 +428,7 @@ async function startMockAnthropicProvider(): Promise { } const port = (address as AddressInfo).port; return { - endpointUrl: `http://host.openshell.internal:${port}`, + endpointUrl: compatibleAnthropicMockEndpointUrl(port), close: () => closeServer(server), }; } @@ -1087,17 +1087,13 @@ test("openclaw-inference-switch: switches route and preserves live OpenClaw beha gatewayRestartExpected ? "anthropic-messages" : "openai-completions", ); const pidBefore = await openclawGatewayPid(sandbox, home); - const switchInference = (commandHost: HostCliClient = host) => - runOpenClawInferenceSetWithRetry( - commandHost, - home, - redactionValues, - switchBinding, - artifacts, - ); - const switchResult = mockProvider - ? await withHostVerificationLoopbackAlias(host, switchInference) - : await switchInference(); + const switchResult = await runOpenClawInferenceSetWithRetry( + host, + home, + redactionValues, + switchBinding, + artifacts, + ); expect(switchResult.exitCode, resultText(switchResult)).toBe(0); expect( resultText(switchResult).includes( diff --git a/test/e2e/support/compatible-anthropic-switch.test.ts b/test/e2e/support/compatible-anthropic-switch.test.ts index f789f562abe..460de62fed7 100644 --- a/test/e2e/support/compatible-anthropic-switch.test.ts +++ b/test/e2e/support/compatible-anthropic-switch.test.ts @@ -1,22 +1,16 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { spawnSync } from "node:child_process"; -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; - import { afterEach, describe, expect, it, vi } from "vitest"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import { COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV, COMPATIBLE_ANTHROPIC_PROVIDER, + compatibleAnthropicMockEndpointUrl, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, - HOST_VERIFICATION_NAMESPACE_SCRIPT, requireCompatibleAnthropicProviderAbsent, - withHostVerificationLoopbackAlias, } from "../fixtures/compatible-anthropic-switch.ts"; describe("compatible Anthropic inference switch setup", () => { @@ -53,45 +47,8 @@ describe("compatible Anthropic inference switch setup", () => { ).toThrow("COMPATIBLE_ANTHROPIC_API_KEY is required"); }); - it("runs host verification inside a private resolver mount namespace", async () => { - const result = { exitCode: 0, stderr: "", stdout: "verified" }; - const command = vi.fn().mockResolvedValue(result); - const commandEnv = { COMPATIBLE_ANTHROPIC_API_KEY: "fixture-key" }; - - await expect( - withHostVerificationLoopbackAlias( - { command } as unknown as HostCliClient, - (scopedHost) => - scopedHost.command("node", ["nemoclaw.js", "inference", "set"], { - artifactName: "inference-set", - env: commandEnv, - redactionValues: ["fixture-key"], - }), - ), - ).resolves.toBe(result); - - expect(command).toHaveBeenCalledOnce(); - const [program, args, options] = command.mock.calls[0] ?? []; - expect(program).toBe("sudo"); - expect(args).toEqual( - expect.arrayContaining([ - "unshare", - "--mount", - "--fork", - "/etc/hosts", - process.execPath, - "nemoclaw.js", - "inference", - "set", - ]), - ); - expect(args).toContain("--preserve-env=COMPATIBLE_ANTHROPIC_API_KEY"); - expect(args).not.toContain("fixture-key"); - expect(options).toEqual({ - artifactName: "inference-set", - env: commandEnv, - redactionValues: ["fixture-key"], - }); + it("advertises the mock on the gateway host loopback (#9166)", () => { + expect(compatibleAnthropicMockEndpointUrl(18_766)).toBe("http://127.0.0.1:18766"); }); it("requires the direct provider to be absent before inference set owns its creation", async () => { @@ -143,68 +100,3 @@ describe("compatible Anthropic inference switch setup", () => { ); }); }); - -const linuxIt = process.platform === "linux" ? it : it.skip; - -describe("host verifier resolver namespace", () => { - function testFiles(): { directory: string; hostsPath: string; capturedPath: string } { - const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-host-verifier-test-")); - const hostsPath = path.join(directory, "hosts"); - const capturedPath = path.join(directory, "hosts.private"); - fs.writeFileSync(hostsPath, "127.0.0.1 localhost\n", { mode: 0o644 }); - return { directory, hostsPath, capturedPath }; - } - - linuxIt("preserves an unrelated resolver write during private mount setup (#9166)", () => { - const files = testFiles(); - const fakeBin = path.join(files.directory, "bin"); - const fakeMount = path.join(fakeBin, "mount"); - try { - fs.mkdirSync(fakeBin); - fs.writeFileSync( - fakeMount, - [ - "#!/usr/bin/env bash", - "set -euo pipefail", - '[[ "$1" == "--make-rprivate" ]] && exit 0', - '[[ "$1" == "--bind" ]]', - "printf '192.0.2.10 concurrent.example.test\\n' >> \"$NEMOCLAW_TEST_RESOLVER_SOURCE\"", - 'cp -- "$2" "$NEMOCLAW_TEST_RESOLVER_COPY"', - ].join("\n"), - { mode: 0o755 }, - ); - const result = spawnSync( - "bash", - [ - "-ceu", - HOST_VERIFICATION_NAMESPACE_SCRIPT, - "host-verifier-namespace-test", - files.hostsPath, - String(process.getuid?.()), - String(process.getgid?.()), - process.env.PATH ?? "/usr/bin:/bin", - "true", - ], - { - encoding: "utf8", - env: { - ...process.env, - NEMOCLAW_TEST_RESOLVER_COPY: files.capturedPath, - NEMOCLAW_TEST_RESOLVER_SOURCE: files.hostsPath, - PATH: `${fakeBin}:${process.env.PATH ?? ""}`, - }, - }, - ); - expect(result.status, result.stderr).toBe(0); - - expect(fs.readFileSync(files.hostsPath, "utf8")).toBe( - "127.0.0.1 localhost\n192.0.2.10 concurrent.example.test\n", - ); - const privateResolver = fs.readFileSync(files.capturedPath, "utf8"); - expect(privateResolver).toContain("127.0.0.1 host.openshell.internal"); - expect(privateResolver).not.toContain("concurrent.example.test"); - } finally { - fs.rmSync(files.directory, { force: true, recursive: true }); - } - }); -}); diff --git a/test/e2e/support/hermes-inference-switch-command-shape.test.ts b/test/e2e/support/hermes-inference-switch-command-shape.test.ts index fdced5ad593..7e60f458c4b 100644 --- a/test/e2e/support/hermes-inference-switch-command-shape.test.ts +++ b/test/e2e/support/hermes-inference-switch-command-shape.test.ts @@ -22,7 +22,6 @@ import { hostedInstallModel, inferenceLocalMaxTokens, installHermes, - mockAnthropicEndpointUrl, mockAnthropicSwitchEnabled, openAiSurfaceEndpointUrl, openshellGatewayName, @@ -110,15 +109,6 @@ describe("Hermes inference switch command shape", () => { ).toBe("initial-hosted-model"); }); - it("advertises the mock through the OpenShell host alias", () => { - expect(mockAnthropicEndpointUrl(18_766, {})).toBe("http://host.openshell.internal:18766"); - expect( - mockAnthropicEndpointUrl(18_766, { - NEMOCLAW_SWITCH_MOCK_HOST: "host.openshell.internal", - }), - ).toBe("http://host.openshell.internal:18766"); - }); - it("uses authenticated model inventory as baseline readiness evidence", () => { expect(() => expectAuthenticatedBaselineInventoryRequest({ From 6c9ef08d5b9b9a4f6c27d352579949b1d8b0eaeb Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 21:58:01 -0700 Subject: [PATCH 14/23] fix(e2e): scope mock resolver to gateway Signed-off-by: Carlos Villela --- .../fixtures/compatible-anthropic-switch.ts | 156 +++++++++++++++++- .../live/hermes-inference-switch-helpers.ts | 2 + .../live/openclaw-inference-switch.test.ts | 2 + .../compatible-anthropic-switch.test.ts | 124 +++++++++++++- 4 files changed, 281 insertions(+), 3 deletions(-) diff --git a/test/e2e/fixtures/compatible-anthropic-switch.ts b/test/e2e/fixtures/compatible-anthropic-switch.ts index eff457000b9..4ad22662fbb 100644 --- a/test/e2e/fixtures/compatible-anthropic-switch.ts +++ b/test/e2e/fixtures/compatible-anthropic-switch.ts @@ -1,12 +1,82 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { randomBytes } from "node:crypto"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + import type { HostCliClient } from "./clients/host.ts"; import { resultText } from "./clients/index.ts"; export const COMPATIBLE_ANTHROPIC_PROVIDER = "compatible-anthropic-endpoint"; export const COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV = "COMPATIBLE_ANTHROPIC_API_KEY"; const DEFAULT_COMPATIBLE_ANTHROPIC_CREDENTIAL = "test-compatible-anthropic-key"; +const OPENSHELL_HOST_ALIAS = "host.openshell.internal"; +const GATEWAY_SERVICE_NAMES = ["nemoclaw-openshell-gateway", "openshell-gateway"] as const; + +export const GATEWAY_HOST_VERIFICATION_MOUNT_SCRIPT = [ + "set -euo pipefail", + 'operation="$1"', + 'resolver_source="$2"', + 'owner_token="$3"', + 'hosts_path="$4"', + `alias_name="${OPENSHELL_HOST_ALIAS}"`, + 'owned_line="127.0.0.1 ${alias_name} # nemoclaw-gateway-host-verifier:${owner_token}"', + "", + 'case "$operation" in', + " add | remove) ;;", + ' *) echo "unsupported gateway resolver operation: $operation" >&2; exit 2 ;;', + "esac", + '[[ "$owner_token" =~ ^[a-f0-9]{32}$ ]] || { echo "invalid gateway resolver owner token" >&2; exit 2; }', + '[[ -f "$hosts_path" ]] || { echo "gateway resolver path is not a regular file" >&2; exit 2; }', + "", + 'if [[ "$operation" == "add" ]]; then', + ' [[ -f "$resolver_source" && ! -L "$resolver_source" ]] || { echo "gateway resolver source is not a regular file" >&2; exit 2; }', + ' grep -Fqx -- "$owned_line" "$resolver_source" || { echo "gateway resolver source lacks its ownership marker" >&2; exit 2; }', + " mount --make-rprivate /", + ' mount --bind "$resolver_source" "$hosts_path"', + ' grep -Fqx -- "$owned_line" "$hosts_path" || { echo "gateway resolver mount was not installed" >&2; exit 4; }', + " exit 0", + "fi", + "", + '# A restarted gateway has already released the owned mount namespace.', + 'grep -Fqx -- "$owned_line" "$hosts_path" || exit 0', + 'umount "$hosts_path"', + 'if grep -Fqx -- "$owned_line" "$hosts_path"; then', + ' echo "gateway resolver mount was not removed" >&2', + " exit 4", + "fi", +].join("\n"); + +const GATEWAY_HOST_VERIFICATION_NAMESPACE_SCRIPT = [ + "set -euo pipefail", + 'operation="$1"', + 'target_pid="$2"', + 'resolver_source="$3"', + 'owner_token="$4"', + 'mount_script="$5"', + "", + '[[ "$target_pid" =~ ^[1-9][0-9]*$ ]] || { echo "invalid OpenShell gateway PID" >&2; exit 2; }', + "gateway_is_alive() {", + ' local executable=""', + ' [[ -r "/proc/${target_pid}/stat" ]] || return 1', + ' executable="$(readlink -f "/proc/${target_pid}/exe" 2>/dev/null || true)"', + ' [[ "${executable##*/}" == "openshell-gateway" ]]', + "}", + "", + 'if ! gateway_is_alive; then', + ' [[ "$operation" == "remove" ]] && exit 0', + ' echo "active OpenShell gateway process is unavailable" >&2', + " exit 3", + "fi", + 'command -v nsenter >/dev/null 2>&1 || { echo "nsenter is required for scoped gateway resolution" >&2; exit 2; }', + 'current_namespace="$(readlink /proc/self/ns/mnt)"', + 'target_namespace="$(readlink "/proc/${target_pid}/ns/mnt")"', + '[[ "$current_namespace" != "$target_namespace" ]] || { echo "OpenShell gateway does not have a private mount namespace" >&2; exit 3; }', + "", + 'exec nsenter --target "$target_pid" --mount -- bash -ceu "$mount_script" gateway-resolver-mount "$operation" "$resolver_source" "$owner_token" /etc/hosts', +].join("\n"); export interface CompatibleAnthropicSwitchBinding { endpointUrl: string; @@ -14,7 +84,91 @@ export interface CompatibleAnthropicSwitchBinding { } export function compatibleAnthropicMockEndpointUrl(port: number): string { - return `http://127.0.0.1:${port}`; + return `http://${OPENSHELL_HOST_ALIAS}:${port}`; +} + +async function activeOpenShellGatewayPid(host: HostCliClient): Promise { + for (const serviceName of GATEWAY_SERVICE_NAMES) { + const result = await host.command( + "systemctl", + [ + "--user", + "show", + serviceName, + "--property=ActiveState", + "--property=MainPID", + ], + { artifactName: `inspect-${serviceName}`, timeoutMs: 30_000 }, + ); + if (result.exitCode !== 0) continue; + const properties = new Map( + result.stdout + .split(/\r?\n/u) + .map((line) => line.split("=", 2)) + .filter((entry): entry is [string, string] => entry.length === 2), + ); + const pid = Number(properties.get("MainPID")); + if (properties.get("ActiveState") === "active" && Number.isSafeInteger(pid) && pid > 0) { + return pid; + } + } + throw new Error("could not find an active OpenShell gateway user service"); +} + +export async function installGatewayHostVerificationAlias( + host: HostCliClient, + cleanup: { add(name: string, run: () => Promise | void): void }, +): Promise { + const gatewayPid = await activeOpenShellGatewayPid(host); + const ownerToken = randomBytes(16).toString("hex"); + const fixtureDirectory = fs.mkdtempSync( + path.join(os.homedir(), ".nemoclaw-gateway-resolver-"), + ); + const resolverSource = path.join(fixtureDirectory, "hosts"); + const ownedLine = `127.0.0.1 ${OPENSHELL_HOST_ALIAS} # nemoclaw-gateway-host-verifier:${ownerToken}`; + fs.chmodSync(fixtureDirectory, 0o700); + fs.writeFileSync(resolverSource, `${ownedLine}\n${fs.readFileSync("/etc/hosts", "utf8")}`, { + mode: 0o600, + }); + + const updateMount = async (operation: "add" | "remove"): Promise => { + const result = await host.command( + "sudo", + [ + "bash", + "-ceu", + GATEWAY_HOST_VERIFICATION_NAMESPACE_SCRIPT, + `gateway-resolver-${operation}`, + operation, + String(gatewayPid), + resolverSource, + ownerToken, + GATEWAY_HOST_VERIFICATION_MOUNT_SCRIPT, + ], + { artifactName: `${operation}-gateway-host-verifier-alias`, timeoutMs: 60_000 }, + ); + if (result.exitCode !== 0) { + throw new Error( + `could not ${operation === "add" ? "install" : "remove"} the gateway host verifier alias: ${resultText(result)}`, + ); + } + }; + + let restored = false; + const restore = async (): Promise => { + if (restored) return; + await updateMount("remove"); + restored = true; + fs.rmSync(fixtureDirectory, { force: true, recursive: true }); + }; + cleanup.add("remove the OpenShell gateway resolver mount", restore); + + try { + await updateMount("add"); + } catch (error) { + await restore(); + throw error; + } } export function compatibleAnthropicSwitchBinding( diff --git a/test/e2e/live/hermes-inference-switch-helpers.ts b/test/e2e/live/hermes-inference-switch-helpers.ts index 7887cd1d641..3b0f8db1f34 100644 --- a/test/e2e/live/hermes-inference-switch-helpers.ts +++ b/test/e2e/live/hermes-inference-switch-helpers.ts @@ -21,6 +21,7 @@ import { compatibleAnthropicMockEndpointUrl, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, + installGatewayHostVerificationAlias, requireCompatibleAnthropicProviderAbsent, } from "../fixtures/compatible-anthropic-switch.ts"; import { expect } from "../fixtures/e2e-test.ts"; @@ -542,6 +543,7 @@ export async function prepareCompatibleAnthropicSwitchBinding( return null; const mock = mockAnthropicSwitchEnabled() ? await startMockAnthropicProvider() : undefined; mock && cleanup.add("close compatible Anthropic switch mock", () => mock.close()); + if (mock) await installGatewayHostVerificationAlias(host, cleanup); const binding = compatibleAnthropicSwitchBinding( process.env.NEMOCLAW_SWITCH_ENDPOINT_URL ?? mock?.endpointUrl ?? "", ); diff --git a/test/e2e/live/openclaw-inference-switch.test.ts b/test/e2e/live/openclaw-inference-switch.test.ts index f0e86971259..0fa95a5728e 100644 --- a/test/e2e/live/openclaw-inference-switch.test.ts +++ b/test/e2e/live/openclaw-inference-switch.test.ts @@ -29,6 +29,7 @@ import { compatibleAnthropicMockEndpointUrl, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, + installGatewayHostVerificationAlias, requireCompatibleAnthropicProviderAbsent, } from "../fixtures/compatible-anthropic-switch.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; @@ -1067,6 +1068,7 @@ test("openclaw-inference-switch: switches route and preserves live OpenClaw beha if (SWITCH_PROVIDER === "compatible-anthropic-endpoint" && SWITCH_MOCK_ANTHROPIC === "1") { mockProvider = await startMockAnthropicProvider(); + await installGatewayHostVerificationAlias(host, cleanup); await artifacts.writeJson("mock-anthropic-provider.json", { endpointUrl: mockProvider.endpointUrl, }); diff --git a/test/e2e/support/compatible-anthropic-switch.test.ts b/test/e2e/support/compatible-anthropic-switch.test.ts index 460de62fed7..530f605f840 100644 --- a/test/e2e/support/compatible-anthropic-switch.test.ts +++ b/test/e2e/support/compatible-anthropic-switch.test.ts @@ -1,15 +1,23 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + import { afterEach, describe, expect, it, vi } from "vitest"; +import { normalizeCustomEndpointUrl } from "../../../src/lib/actions/inference-set.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import { COMPATIBLE_ANTHROPIC_CREDENTIAL_ENV, COMPATIBLE_ANTHROPIC_PROVIDER, + GATEWAY_HOST_VERIFICATION_MOUNT_SCRIPT, compatibleAnthropicMockEndpointUrl, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, + installGatewayHostVerificationAlias, requireCompatibleAnthropicProviderAbsent, } from "../fixtures/compatible-anthropic-switch.ts"; @@ -47,8 +55,43 @@ describe("compatible Anthropic inference switch setup", () => { ).toThrow("COMPATIBLE_ANTHROPIC_API_KEY is required"); }); - it("advertises the mock on the gateway host loopback (#9166)", () => { - expect(compatibleAnthropicMockEndpointUrl(18_766)).toBe("http://127.0.0.1:18766"); + it("passes the mock bridge through endpoint validation without DNS rewriting (#9166)", async () => { + const endpointUrl = compatibleAnthropicMockEndpointUrl(18_766); + const rewrite = vi.fn(); + + await expect(normalizeCustomEndpointUrl(endpointUrl, rewrite)).resolves.toBe(endpointUrl); + expect(rewrite).not.toHaveBeenCalled(); + }); + + it("mounts the resolver alias only inside the active gateway namespace (#9166)", async () => { + const command = vi + .fn() + .mockResolvedValueOnce({ + exitCode: 0, + stderr: "", + stdout: "ActiveState=active\nMainPID=4242\n", + }) + .mockResolvedValue({ exitCode: 0, stderr: "", stdout: "" }); + const add = vi.fn(); + + await installGatewayHostVerificationAlias({ command } as unknown as HostCliClient, { add }); + const cleanupMount = add.mock.calls[0]?.[1] as () => Promise; + await cleanupMount(); + + expect(command.mock.calls[0]?.slice(0, 2)).toEqual([ + "systemctl", + [ + "--user", + "show", + "nemoclaw-openshell-gateway", + "--property=ActiveState", + "--property=MainPID", + ], + ]); + for (const call of command.mock.calls.slice(1)) { + expect(call[0]).toBe("sudo"); + expect(call[1]).toEqual(expect.arrayContaining(["4242", GATEWAY_HOST_VERIFICATION_MOUNT_SCRIPT])); + } }); it("requires the direct provider to be absent before inference set owns its creation", async () => { @@ -100,3 +143,80 @@ describe("compatible Anthropic inference switch setup", () => { ); }); }); + +const linuxIt = process.platform === "linux" ? it : it.skip; + +describe("gateway resolver mount", () => { + linuxIt("preserves a resolver write that overlaps mount installation (#9166)", () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-resolver-test-")); + const hostsPath = path.join(directory, "hosts"); + const underlayPath = path.join(directory, "hosts.underlay"); + const resolverSource = path.join(directory, "resolver-source"); + const fakeBin = path.join(directory, "bin"); + const token = "a".repeat(32); + const ownedLine = + `127.0.0.1 host.openshell.internal # nemoclaw-gateway-host-verifier:${token}`; + try { + fs.mkdirSync(fakeBin); + fs.writeFileSync(hostsPath, "127.0.0.1 localhost\n", { mode: 0o644 }); + fs.writeFileSync(resolverSource, `${ownedLine}\n127.0.0.1 localhost\n`, { mode: 0o600 }); + fs.writeFileSync( + path.join(fakeBin, "mount"), + [ + "#!/usr/bin/env bash", + "set -euo pipefail", + '[[ "$1" == "--make-rprivate" ]] && exit 0', + '[[ "$1" == "--bind" ]]', + "printf '192.0.2.10 concurrent.example.test\\n' >> \"$3\"", + 'mv -- "$3" "$NEMOCLAW_TEST_RESOLVER_UNDERLAY"', + 'ln -s -- "$2" "$3"', + ].join("\n"), + { mode: 0o755 }, + ); + fs.writeFileSync( + path.join(fakeBin, "umount"), + [ + "#!/usr/bin/env bash", + "set -euo pipefail", + 'rm -- "$1"', + 'mv -- "$NEMOCLAW_TEST_RESOLVER_UNDERLAY" "$1"', + ].join("\n"), + { mode: 0o755 }, + ); + const run = (operation: "add" | "remove") => + spawnSync( + "bash", + [ + "-ceu", + GATEWAY_HOST_VERIFICATION_MOUNT_SCRIPT, + "gateway-resolver-mount-test", + operation, + resolverSource, + token, + hostsPath, + ], + { + encoding: "utf8", + env: { + ...process.env, + NEMOCLAW_TEST_RESOLVER_UNDERLAY: underlayPath, + PATH: `${fakeBin}:${process.env.PATH ?? ""}`, + }, + }, + ); + + const added = run("add"); + expect(added.status, added.stderr).toBe(0); + expect(fs.readFileSync(hostsPath, "utf8")).toContain(ownedLine); + expect(fs.readFileSync(underlayPath, "utf8")).toContain("concurrent.example.test"); + + const removed = run("remove"); + expect(removed.status, removed.stderr).toBe(0); + expect(fs.readFileSync(hostsPath, "utf8")).toBe( + "127.0.0.1 localhost\n192.0.2.10 concurrent.example.test\n", + ); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); +}); From d25899b0603857fc0ab535214652f6b5e99c7fe0 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 22:09:41 -0700 Subject: [PATCH 15/23] fix(e2e): retain agent retry evidence Signed-off-by: Carlos Villela --- test/e2e/live/common-egress-agent-helpers.ts | 70 ++++++ test/e2e/live/common-egress-agent.test.ts | 223 ++++++++++-------- .../common-egress-agent-helpers.test.ts | 77 +++++- 3 files changed, 272 insertions(+), 98 deletions(-) diff --git a/test/e2e/live/common-egress-agent-helpers.ts b/test/e2e/live/common-egress-agent-helpers.ts index b862f7c7df4..fb75ce5491f 100644 --- a/test/e2e/live/common-egress-agent-helpers.ts +++ b/test/e2e/live/common-egress-agent-helpers.ts @@ -8,6 +8,12 @@ // classification without gating on NEMOCLAW_RUN_LIVE_E2E=1. import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; +import { + runBoundedRetry, + type BoundedRetryResult, + type RetryEvidence, + type RetryFailureClass, +} from "../fixtures/retry-policy.ts"; import { isTransientProviderValidationFailure } from "./network-policy-transient-provider.ts"; export const COMMON_EGRESS_TEST_TIMEOUT_MS = 40 * 60_000; @@ -34,6 +40,70 @@ export interface CommonEgressProviderValidationSkip { transientProviderValidationFailure: boolean; } +export interface AgentAssertionAttempt { + failureClass?: RetryFailureClass; + passed: boolean; + recoveryRequired?: boolean; +} + +interface AgentAssertionRetryOptions { + attempts: number; + delayMs: (attempt: number) => number; + onEvidence: (evidence: RetryEvidence) => Promise | void; + run: (attempt: number) => Promise; + sleep?: (milliseconds: number) => Promise; +} + +export function runHermesAgentAssertionRetry( + options: AgentAssertionRetryOptions, +): Promise> { + return runBoundedRetry({ + operation: "common-egress.hermes-agent", + owner: "hermes-agent", + idempotence: "read-only", + maxAttempts: options.attempts, + delayMs: options.delayMs, + onEvidence: options.onEvidence, + run: options.run, + sleep: options.sleep, + classify: (value, error) => { + if (error !== undefined) return { outcome: "failed", failureClass: "deterministic" }; + if (value?.passed) return { outcome: "passed" }; + return { outcome: "failed", failureClass: value?.failureClass ?? "deterministic" }; + }, + }); +} + +export function runOpenClawAgentAssertionRetry( + options: AgentAssertionRetryOptions & { + reconcile: (attempt: AgentAssertionAttempt, attemptNumber: number) => Promise; + }, +): Promise> { + return runBoundedRetry({ + operation: "common-egress.openclaw-agent", + owner: "openclaw-agent", + idempotence: "reconciled-mutation", + maxAttempts: options.attempts, + delayMs: options.delayMs, + onEvidence: options.onEvidence, + run: options.run, + sleep: options.sleep, + classify: (value, error) => { + if (error !== undefined) return { outcome: "failed", failureClass: "deterministic" }; + if (value?.passed) return { outcome: "passed" }; + return { outcome: "failed", failureClass: value?.failureClass ?? "deterministic" }; + }, + reconcile: async (value, _error, attemptNumber) => { + if (!value) return false; + try { + return await options.reconcile(value, attemptNumber); + } catch { + return false; + } + }, + }); +} + export function text(result: Pick): string { return [result.stdout, result.stderr].filter(Boolean).join("\n"); } diff --git a/test/e2e/live/common-egress-agent.test.ts b/test/e2e/live/common-egress-agent.test.ts index 63eee5dabaa..c1d4256665b 100644 --- a/test/e2e/live/common-egress-agent.test.ts +++ b/test/e2e/live/common-egress-agent.test.ts @@ -33,6 +33,8 @@ import { isHermesTransientAgentFailure, parseChatContent, parseOpenClawAgentText, + runHermesAgentAssertionRetry, + runOpenClawAgentAssertionRetry, } from "./common-egress-agent-helpers.ts"; import { stripAnsi } from "./json-envelope.ts"; @@ -447,73 +449,89 @@ async function runOpenClawAgentAssertion( ); let lastFailure = ""; - for (let attempt = 1; attempt <= OPENCLAW_AGENT_ATTEMPTS; attempt += 1) { - const sessionId = `e2e-common-egress-${Date.now()}-${process.pid}-${attempt}`; - const sessionRoot = "/sandbox/.openclaw/agents/main/sessions"; - const remoteCommand = [ - `rm -f ${shellQuote(`${sessionRoot}/${sessionId}.jsonl.lock`)} ${shellQuote( - `${sessionRoot}/${sessionId}.trajectory.jsonl`, - )} 2>/dev/null || true`, - `openclaw agent --agent main --json --thinking off --session-id ${shellQuote( - sessionId, - )} -m ${shellQuote(args.prompt)}`, - ].join("; "); - const agent = await host.command( - "ssh", - [ - "-F", - sshConfigPath, - "-o", - "StrictHostKeyChecking=no", - "-o", - "UserKnownHostsFile=/dev/null", - "-o", - "ConnectTimeout=10", - "-o", - "LogLevel=ERROR", - `openshell-${args.sandboxName}.default`, - remoteCommand, - ], - { - artifactName: `${args.label}-openclaw-agent-attempt-${attempt}`, - env: commandEnv(), - redactionValues: [args.apiKey], - timeoutMs: AGENT_TURN_TIMEOUT_MS, - }, - ); - const combined = text(agent); - if (isOpenClawPolicyBlock(combined)) { - throw new Error(`${args.label}: agent hit policy block: ${combined.slice(0, 600)}`); - } - - const reply = parseOpenClawAgentText(agent.stdout); - if (agent.exitCode === 0 && agentReplyContainsToken(reply, args.expected)) { - return; - } - lastFailure = `reply='${reply.slice(0, 240)}' exit=${agent.exitCode} stdout='${agent.stdout.slice( - 0, - 240, - )}' stderr='${agent.stderr.slice(0, 240)}'`; - - if (attempt < OPENCLAW_AGENT_ATTEMPTS && isOpenClawScopeUpgradePending(combined)) { + const execution = await runOpenClawAgentAssertionRetry({ + attempts: OPENCLAW_AGENT_ATTEMPTS, + delayMs: (attempt) => attempt * 15_000, + onEvidence: async (evidence) => { + await artifacts.writeJson(`retry/${args.label}-agent-retry-evidence.json`, evidence); + }, + run: async (attempt) => { + const sessionId = `e2e-common-egress-${Date.now()}-${process.pid}-${attempt}`; + const sessionRoot = "/sandbox/.openclaw/agents/main/sessions"; + const remoteCommand = [ + `rm -f ${shellQuote(`${sessionRoot}/${sessionId}.jsonl.lock`)} ${shellQuote( + `${sessionRoot}/${sessionId}.trajectory.jsonl`, + )} 2>/dev/null || true`, + `openclaw agent --agent main --json --thinking off --session-id ${shellQuote( + sessionId, + )} -m ${shellQuote(args.prompt)}`, + ].join("; "); + const agent = await host.command( + "ssh", + [ + "-F", + sshConfigPath, + "-o", + "StrictHostKeyChecking=no", + "-o", + "UserKnownHostsFile=/dev/null", + "-o", + "ConnectTimeout=10", + "-o", + "LogLevel=ERROR", + `openshell-${args.sandboxName}.default`, + remoteCommand, + ], + { + artifactName: `${args.label}-openclaw-agent-attempt-${attempt}`, + env: commandEnv(), + redactionValues: [args.apiKey], + timeoutMs: AGENT_TURN_TIMEOUT_MS, + }, + ); + const combined = text(agent); + const reply = parseOpenClawAgentText(agent.stdout); + if (agent.exitCode === 0 && agentReplyContainsToken(reply, args.expected)) { + return { passed: true }; + } + lastFailure = `reply='${reply.slice(0, 240)}' exit=${agent.exitCode} stdout='${agent.stdout.slice( + 0, + 240, + )}' stderr='${agent.stderr.slice(0, 240)}'`; + if (isOpenClawPolicyBlock(combined)) { + return { passed: false, failureClass: "policy-denial" }; + } + if (/\b401\b|unauthorized|authentication failed|invalid api key/iu.test(combined)) { + return { passed: false, failureClass: "authentication" }; + } + if (/\b403\b|forbidden/iu.test(combined)) { + return { passed: false, failureClass: "authorization" }; + } + const recoveryRequired = isOpenClawScopeUpgradePending(combined); + return { + passed: false, + failureClass: + recoveryRequired || isOpenClawTransientAgentError(combined) + ? "transient-external" + : "deterministic", + recoveryRequired, + }; + }, + reconcile: async (attempt, attemptNumber) => { + if (!attempt.recoveryRequired) return true; const recover = await host.command("node", [CLI_ENTRYPOINT, args.sandboxName, "recover"], { - artifactName: `${args.label}-recover-after-attempt-${attempt}`, + artifactName: `${args.label}-recover-after-attempt-${attemptNumber}`, env: commandEnv(), timeoutMs: 120_000, }); - if (recover.exitCode !== 0) break; - await sleep(attempt * 15_000); - continue; - } - - if (attempt < OPENCLAW_AGENT_ATTEMPTS && isOpenClawTransientAgentError(combined)) { - await sleep(attempt * 15_000); - continue; - } - - break; - } - + if (recover.exitCode !== 0) { + lastFailure = `recovery exit=${recover.exitCode}`; + return false; + } + return true; + }, + }); + if (execution.outcome === "passed") return; throw new Error(`${args.label}: expected ${args.expected}, got ${lastFailure}`); } @@ -534,6 +552,7 @@ After the command completes, reply exactly HERMES_REFERENCE_AGENT_OK if that exa async function runHermesAgentAssertion( sandbox: SandboxClient, + artifacts: ArtifactSink, args: { expected: string; label: string; @@ -564,39 +583,49 @@ async function runHermesAgentAssertion( ].join("; "); let lastFailure = ""; - for (let attempt = 1; attempt <= HERMES_AGENT_ATTEMPTS; attempt += 1) { - const agent = await sandbox.execShell(args.sandboxName, trustedSandboxShellScript(remote), { - artifactName: `${args.label}-hermes-agent-attempt-${attempt}`, - env: commandEnv(), - timeoutMs: HERMES_AGENT_TIMEOUT_MS, - }); - const response = text(agent); - const httpStatus = httpStatusFromResponse(response); - const body = httpBodyFromResponse(response); - let reply = ""; - try { - reply = parseChatContent(body); - } catch { - reply = ""; - } - if ( - agent.exitCode === 0 && - httpStatus === "200" && - agentReplyContainsToken(reply, args.expected) - ) { - return; - } - lastFailure = `exit=${agent.exitCode} http=${httpStatus} reply='${reply.slice( - 0, - 240, - )}' body='${body.slice(0, 240)}'`; - if (attempt < HERMES_AGENT_ATTEMPTS && isHermesTransientAgentFailure(httpStatus, response)) { - await sleep(attempt * 5_000); - continue; - } - break; - } - + const execution = await runHermesAgentAssertionRetry({ + attempts: HERMES_AGENT_ATTEMPTS, + delayMs: () => 5_000, + onEvidence: async (evidence) => { + await artifacts.writeJson(`retry/${args.label}-agent-retry-evidence.json`, evidence); + }, + run: async (attempt) => { + const agent = await sandbox.execShell(args.sandboxName, trustedSandboxShellScript(remote), { + artifactName: `${args.label}-hermes-agent-attempt-${attempt}`, + env: commandEnv(), + timeoutMs: HERMES_AGENT_TIMEOUT_MS, + }); + const response = text(agent); + const httpStatus = httpStatusFromResponse(response); + const body = httpBodyFromResponse(response); + let reply = ""; + try { + reply = parseChatContent(body); + } catch { + reply = ""; + } + if ( + agent.exitCode === 0 && + httpStatus === "200" && + agentReplyContainsToken(reply, args.expected) + ) { + return { passed: true }; + } + lastFailure = `exit=${agent.exitCode} http=${httpStatus} reply='${reply.slice( + 0, + 240, + )}' body='${body.slice(0, 240)}'`; + if (httpStatus === "401") return { passed: false, failureClass: "authentication" }; + if (httpStatus === "403") return { passed: false, failureClass: "authorization" }; + return { + passed: false, + failureClass: isHermesTransientAgentFailure(httpStatus, response) + ? "transient-external" + : "deterministic", + }; + }, + }); + if (execution.outcome === "passed") return; throw new Error(`${args.label}: expected ${args.expected}, got ${lastFailure}`); } @@ -855,7 +884,7 @@ After web_fetch returns, reply exactly REFERENCE_AGENT_OK if the fetched respons "/modal", ]); progress.phase("fetch Wikidata with Hermes agent"); - await runHermesAgentAssertion(sandbox, { + await runHermesAgentAssertion(sandbox, artifacts, { expected: "HERMES_REFERENCE_AGENT_OK", label: "c3-agent-reference", prompt: buildHermesReferencePrompt(), diff --git a/test/e2e/support/common-egress-agent-helpers.test.ts b/test/e2e/support/common-egress-agent-helpers.test.ts index 560359e3734..0127d4644e8 100644 --- a/test/e2e/support/common-egress-agent-helpers.test.ts +++ b/test/e2e/support/common-egress-agent-helpers.test.ts @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { agentReplyContainsToken, @@ -9,6 +9,8 @@ import { isHermesTransientAgentFailure, parseChatContent, parseOpenClawAgentText, + runHermesAgentAssertionRetry, + runOpenClawAgentAssertionRetry, } from "../live/common-egress-agent-helpers.ts"; describe("common-egress agent parsing and classification helpers", () => { @@ -55,6 +57,79 @@ describe("common-egress agent parsing and classification helpers", () => { expect(isHermesTransientAgentFailure("200", "reply mentions fetch failed")).toBe(false); }); + it("records recovered OpenClaw success after reconciliation", async () => { + const onEvidence = vi.fn(); + const reconcile = vi.fn().mockResolvedValue(true); + const run = vi + .fn() + .mockResolvedValueOnce({ + passed: false, + failureClass: "transient-external", + recoveryRequired: true, + }) + .mockResolvedValueOnce({ passed: true }); + + const result = await runOpenClawAgentAssertionRetry({ + attempts: 3, + delayMs: () => 0, + onEvidence, + reconcile, + run, + }); + + expect(result.outcome).toBe("passed"); + expect(onEvidence).toHaveBeenCalledWith({ + schemaVersion: 1, + operation: "common-egress.openclaw-agent", + owner: "openclaw-agent", + idempotence: "reconciled-mutation", + maxAttempts: 3, + outcome: "passed-after-retry", + attempts: [ + { + attempt: 1, + outcome: "failed", + failureClass: "transient-external", + reconciled: true, + retryScheduled: true, + }, + { attempt: 2, outcome: "passed", retryScheduled: false }, + ], + }); + expect(reconcile).toHaveBeenCalledWith(expect.objectContaining({ recoveryRequired: true }), 1); + }); + + it("records a deterministic Hermes failure without retrying", async () => { + const onEvidence = vi.fn(); + const run = vi.fn().mockResolvedValue({ passed: false, failureClass: "deterministic" }); + + const result = await runHermesAgentAssertionRetry({ + attempts: 3, + delayMs: () => 0, + onEvidence, + run, + }); + + expect(result.outcome).toBe("failed"); + expect(run).toHaveBeenCalledOnce(); + expect(onEvidence).toHaveBeenCalledWith({ + schemaVersion: 1, + operation: "common-egress.hermes-agent", + owner: "hermes-agent", + idempotence: "read-only", + maxAttempts: 3, + outcome: "failed-no-retry", + attempts: [ + { + attempt: 1, + outcome: "failed", + failureClass: "deterministic", + retryScheduled: false, + }, + ], + }); + }); + it("classifies pre-contract provider validation skips", () => { expect( classifyPreContractProviderValidationSkip({ From cb9628e507e0c859352e1cf08173931d78eb2620 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Fri, 14 Aug 2026 22:15:35 -0700 Subject: [PATCH 16/23] fix(e2e): satisfy retry evidence guardrail Signed-off-by: Carlos Villela --- test/e2e/RETRY_INVENTORY.md | 2 +- test/e2e/live/common-egress-agent-helpers.ts | 77 +++++++++++++++++++ test/e2e/live/common-egress-agent.test.ts | 69 ++++------------- .../common-egress-agent-helpers.test.ts | 73 ++++++++++++++++++ 4 files changed, 166 insertions(+), 55 deletions(-) diff --git a/test/e2e/RETRY_INVENTORY.md b/test/e2e/RETRY_INVENTORY.md index 26c3805b0dd..e58fcb8ca66 100644 --- a/test/e2e/RETRY_INVENTORY.md +++ b/test/e2e/RETRY_INVENTORY.md @@ -25,7 +25,7 @@ Exhaustion remains failed. | `provider-ci-skip` | Cloud-provider validation skip; `test/e2e/live/cloud-inference-provider-skip.ts` | Same transient provider allowlist | No retry; one CI-only skip | Read-only provider validation | Inference provider | Explicit transient skip versus terminal failure | Skip reason and command artifact | Intentional transient skip; release conclusion unchanged | | `hosted-inference-budget` | Hosted inference response-budget escalation; `test/e2e/live/hermes-e2e.test.ts` | Reasoning consumed a smaller response budget | No retry; one request with the sufficient bounded budget | Read-only inference request | Test owner | Deterministic response assertion | Direct and sandbox chat artifacts | Ineligible deterministic retry removed | | `cloud-inference-probe` | Hosted cloud inference request; `test/e2e/live/cloud-inference.test.ts` | Curl transport timeout/reset/DNS/connect error, HTTP 429, or HTTP 5xx | 1-10 attempts (default 3); fixed 5s delay | Read-only inference request | Inference provider | Explicit external transient versus terminal response/semantic assertion | Per-request artifacts and aggregate attempt evidence | Eligible bounded read; HTTP status is separated from provider content and deterministic assertions do not retry | -| `agent-turn-probe` | OpenClaw and Hermes agent turns; `test/e2e/live/common-egress-agent.test.ts`, `test/e2e/live/common-egress-agent-helpers.ts` | Exact OpenClaw scope-upgrade signal followed by successful recovery, or non-200 transport/provider timeout, reset, DNS, connection, 408/429/5xx signature | 3 attempts; OpenClaw linear 15s, Hermes fixed 5s | Read-only prompt; scope-upgrade retry requires successful recovery reconciliation | Agent runtime or inference provider | Explicit transient/reconciled convergence versus terminal product response | Per-attempt result, stdout, stderr, recovery artifact, and progress | Existing bounded path; HTTP 200 product replies and unconditional fallback retry remain terminal | +| `agent-turn-probe` | OpenClaw and Hermes agent turns; `test/e2e/live/common-egress-agent.test.ts`, `test/e2e/live/common-egress-agent-helpers.ts` | Exact OpenClaw scope-upgrade signal followed by successful recovery, or non-200 transport/provider timeout, reset, DNS, connection, 408/429/5xx signature | 3 attempts; OpenClaw linear 15s, Hermes fixed 5s | Read-only prompt; scope-upgrade retry requires successful recovery reconciliation | Agent runtime or inference provider | Explicit transient/reconciled convergence versus terminal product response | Per-attempt result, stdout, stderr, recovery artifact, and progress; aggregate credential-free attempt evidence in `retry/