Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
5bf10d6
fix(e2e): retry transient OpenClaw provider turns
cjagwani Sep 4, 2026
2c6c687
test(e2e): register provider retry parity
cjagwani Sep 4, 2026
65677ba
fix(e2e): structure provider retry signals
cjagwani Sep 4, 2026
5c8b008
merge: update provider retry repair from main
prekshivyas Sep 5, 2026
8c240fb
fix(forward): settle service startup safely
prekshivyas Sep 5, 2026
e916534
Merge remote-tracking branch 'origin/main' into pr-11077
prekshivyas Sep 5, 2026
b180da4
fix(e2e): make forward retries process-safe
prekshivyas Sep 5, 2026
5571a9b
fix(forward): verify process identity before cleanup
prekshivyas Sep 5, 2026
3668662
fix(forward): verify supported-host listener ownership
prekshivyas Sep 5, 2026
fdcd474
Merge remote-tracking branch 'origin/main' into pr-11077
prekshivyas Sep 5, 2026
f9fa066
fix(e2e): close forward retry validation gaps
prekshivyas Sep 5, 2026
57f938e
fix(forward): avoid activating readiness probes
prekshivyas Sep 5, 2026
741c539
fix(forward): restore readiness handoff retries
prekshivyas Sep 5, 2026
7de1f17
fix(forward): normalize wrapped readiness errors
prekshivyas Sep 5, 2026
3cdb354
fix(forward): normalize diagnostic layout
prekshivyas Sep 5, 2026
45c5a18
fix(e2e): require provider outage evidence
prekshivyas Sep 5, 2026
2d0df5e
fix(e2e): preserve retry cleanup budget
prekshivyas Sep 5, 2026
7ebb655
test(e2e): model empty provider turns
prekshivyas Sep 5, 2026
312b2ee
fix(forward): report scoped recovery state
prekshivyas Sep 5, 2026
03be604
fix(forward): preserve OpenShell config authority
prekshivyas Sep 5, 2026
0d1cfc7
fix(e2e): keep provider retries fail closed
prekshivyas Sep 5, 2026
26933c7
fix(forward): verify recovered listener identity
prekshivyas Sep 5, 2026
e1fc111
test(forward): use platform test gate
prekshivyas Sep 5, 2026
d4b602c
fix(forward): bound listener stability
prekshivyas Sep 5, 2026
4f9126d
fix(forward): verify launch listener authority
prekshivyas Sep 5, 2026
cebe90d
fix(e2e): block retry after cleanup failure
prekshivyas Sep 5, 2026
4302882
fix(e2e): narrow provider outage retry
prekshivyas Sep 8, 2026
d0bd3f4
fix(e2e): fail closed before provider retry
prekshivyas Sep 8, 2026
5154b00
chore: merge latest main into provider retry
prekshivyas Sep 8, 2026
7eb235e
test(e2e): close provider retry evidence gaps
prekshivyas Sep 8, 2026
4f3705a
chore: merge latest main into provider retry
prekshivyas Sep 8, 2026
f5817d1
fix(e2e): delimit provider retry marker
prekshivyas Sep 8, 2026
a357622
chore: merge latest main into provider retry
prekshivyas Sep 8, 2026
639a46e
chore: merge latest main into provider retry
prekshivyas Sep 8, 2026
86a13cc
fix(e2e): trust structured provider failures
prekshivyas Sep 8, 2026
254a16a
chore: merge main into provider availability fix
prekshivyas Sep 8, 2026
8892c34
fix(ci): preserve macOS E2E timeout headroom
prekshivyas Sep 8, 2026
2813f54
chore: merge latest main into provider availability fix
prekshivyas Sep 8, 2026
d730fe4
refactor(e2e): centralize full E2E timeout budget
prekshivyas Sep 8, 2026
ced63a2
test(e2e): stage timeout contract in advisor fixture
prekshivyas Sep 9, 2026
1d293d9
chore: merge latest main into provider availability fix
prekshivyas Sep 9, 2026
ea428b6
Merge origin/main into codex/fix-10978-provider-availability-20260904
prekshivyas Sep 9, 2026
ea2d353
fix(deps): require patched js-yaml
prekshivyas Sep 9, 2026
428db9a
Merge origin/main into codex/fix-10978-provider-availability-20260904
prekshivyas Sep 9, 2026
4c7c717
test(e2e): prove exhausted provider handoff
prekshivyas Sep 9, 2026
005a9a0
test(e2e): keep provider proof within guardrails
prekshivyas Sep 9, 2026
f6a376a
test(e2e): accept bounded provider diagnostics
prekshivyas Sep 9, 2026
fe8072e
Merge origin/main into codex/fix-10978-provider-availability-20260904
prekshivyas Sep 9, 2026
b7414a1
ci(e2e): bind portable timeout ownership
prekshivyas Sep 9, 2026
512f326
Merge origin/main into codex/fix-10978-provider-availability-20260904
prekshivyas Sep 9, 2026
a7dce15
fix(e2e): constrain provider retries and budgets
prekshivyas Sep 9, 2026
651937c
Merge origin/main into codex/fix-10978-provider-availability-20260904
prekshivyas Sep 9, 2026
80cb7c1
test(e2e): use typed standard timeout contract
prekshivyas Sep 9, 2026
b5a0c8c
Merge remote-tracking branch 'origin/main' into pr-11077-narrow
prekshivyas Sep 9, 2026
66cd7e0
fix(e2e): route timeout contract evidence
prekshivyas Sep 9, 2026
18befcf
test(e2e): cover accepted provider variants
prekshivyas Sep 9, 2026
8e8fdc9
chore(deps): drop unrelated lockfile drift
prekshivyas Sep 9, 2026
b6eaf1c
Merge remote-tracking branch 'origin/main' into pr-11077-narrow
prekshivyas Sep 9, 2026
4fe9f99
fix(e2e): complete timeout evidence routing
prekshivyas Sep 9, 2026
8e0f75b
fix(e2e): validate provider retry evidence
prekshivyas Sep 9, 2026
49e8658
fix(e2e): preserve fail-closed provider retries
prekshivyas Sep 9, 2026
399bef7
Merge origin/main into codex/fix-10978-provider-availability-20260904
prekshivyas Sep 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions .github/workflows/platform-vitest-main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,8 @@ jobs:
matrix:
include:
- shard: 1
timeout_minutes: 60
# The live E2E has a 70m timeout; every other step shares the remaining job time.
timeout_minutes: 150
- shard: 2
timeout_minutes: 30
- shard: 3
Expand Down Expand Up @@ -192,9 +193,7 @@ jobs:
NEMOCLAW_SANDBOX_NAME: "e2e-macos"
NEMOCLAW_RUN_LIVE_E2E: "1"
run: >-
npx vitest run --project e2e-live
test/e2e/live/full-e2e.test.ts
--silent=false --reporter=default
npx vitest run --project e2e-live test/e2e/live/full-e2e.test.ts --silent=false --reporter=default

- name: Explain skipped macOS live E2E
if: ${{ matrix.shard == 1 && (steps.macos_docker.outputs.docker_ok != 'true' || github.ref != 'refs/heads/main') }}
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/portable-profile-e2e.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ on:
- "test/e2e/live/portable-profile-gateway-proof.ts"
- "test/e2e/live/portable-profile-rootless-linux.test.ts"
- "tools/e2e/check-semantic-phases.mts"
- "tools/e2e/full-e2e-timeout-contract.mts"

permissions:
contents: read
Expand Down Expand Up @@ -155,7 +156,8 @@ jobs:
portable-launch:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: ubuntu-26.04
timeout-minutes: 75
# The live E2E has a 70m timeout; every other step shares the remaining job time.
timeout-minutes: 135
env:
E2E_JOB: "1"
E2E_TARGET_ID: portable-launch
Expand Down
1 change: 1 addition & 0 deletions test/automation/e2e/e2e-recommendations.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ describe("E2E recommendation normalizer", () => {
"tools/advisors/risk-plan.mts",
"tools/e2e/credential-free-tests.mts",
"tools/e2e/execution-coverage.mts",
"tools/e2e/full-e2e-timeout-contract.mts",
"tools/e2e/gateway-runtime.mts",
"tools/e2e/onboard-timeout-contract.mts",
"tools/e2e/openshell-gateway-upgrade-fixture.mts",
Expand Down
16 changes: 14 additions & 2 deletions test/automation/e2e/platform-vitest-main-workflow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,24 @@ describe("platform evidence workflow", () => {
},
])("limits credentialed $job E2E to the first main-branch shard", (workflowCase) => {
const live = step(workflowCase.job, workflowCase.step);
expect(live.if).toContain("matrix.shard == 1");
expect(live.if).toContain(workflowCase.dockerOutput);
expect([
live.if?.includes("matrix.shard == 1"),
live.if?.includes(workflowCase.dockerOutput),
]).toEqual([true, true]);
expect(live.if).toContain("github.ref == 'refs/heads/main'");
expect(live.env).toMatchObject({
GITHUB_TOKEN: "${{ github.token }}",
NVIDIA_INFERENCE_API_KEY: "${{ secrets.NVIDIA_INFERENCE_API_KEY }}",
});
});

it("reserves macOS cleanup and artifact time beyond the full E2E deadline", () => {
const macosShards = job("macos-vitest").strategy?.matrix?.include as Array<{
shard: number;
timeout_minutes: number;
}>;
const firstShard = macosShards.find(({ shard }) => shard === 1);

expect(firstShard?.timeout_minutes).toBe(150);
});
});
11 changes: 11 additions & 0 deletions test/e2e-runtime/brev-launchable-e2e.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import fs from "node:fs";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";

import { FULL_E2E_TEST_TIMEOUT_MINUTES } from "../../tools/e2e/full-e2e-timeout-contract.mts";
import {
candidateSha,
cleanupFixtures,
Expand All @@ -28,6 +29,16 @@ function identitySmokeEnv(env: NodeJS.ProcessEnv): NodeJS.ProcessEnv {
}

describe("focused staging Brev Launchable lane", () => {
it("keeps the staging SSH wrapper outside the full E2E deadline", () => {
const source = fs.readFileSync(
path.resolve(import.meta.dirname, "../../tools/e2e/brev-launchable-e2e.sh"),
"utf8",
);
const defaultTimeout = source.match(/FULL_E2E_TIMEOUT_SECONDS:-([0-9]+)\}/u)?.[1];

expect(Number(defaultTimeout)).toBe(FULL_E2E_TEST_TIMEOUT_MINUTES * 60 + 300);
});

it("runs the strict lane without inherited lane controls (#9925)", () => {
vi.stubEnv("BREV_CREATE_RECONCILE_SECONDS", "0");
vi.stubEnv("NEMOCLAW_BREV_DEFER_CLEANUP", "1");
Expand Down
18 changes: 12 additions & 6 deletions test/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,9 @@ boundaries are the behavior under test.
`.github/workflows/platform-vitest-main.yaml` publishes the `CI / Platform Compatibility` workflow.
It runs the Ubuntu 26.04 compatibility contracts and the full Vitest suite in four shards on macOS and WSL.
The matrix disables `fail-fast`.
The first macOS shard has a 60-minute budget for live E2E; the other shards have 30 minutes.
The first macOS shard has a 150-minute job timeout. Its live E2E has a
70-minute timeout, and every other step shares the remaining job time. The
other shards have 30 minutes.
The first WSL shard has a 180-minute budget for root-required contracts and live E2E; the other shards have 90 minutes.

On shard 1, the workflow runs focused macOS and WSL live E2E only when the run tests `main` and Docker is available.
Expand Down Expand Up @@ -497,11 +499,15 @@ The host must provide that user a secure, independently writable OS runtime auth
The host must provide the util-linux `script` command and GNU `timeout` command.

The helper rebuilds the candidate CLI, runs `connect --probe-only`, and then
runs two `launch` sessions during the same fixed lease.
Each real pseudo-terminal session sends two distinct messages and `/exit`, then
requires process exit status `0`. The OpenClaw session store must append two
nonempty `user` and `assistant` record pairs in one session. The helper does not
compare message content. Terminal output is a bounded failure diagnostic only.
runs two logical `launch` sessions during the same fixed lease. Each logical
session may retry once with a fresh run ID and input only when the OpenClaw
session store contains a structured transient provider-unavailability record
and cleanup succeeds. Authentication, authorization, policy, malformed-response,
cleanup, and unknown failures stop the acceptance test without retrying.
Each successful real pseudo-terminal attempt sends two distinct messages and
`/exit`, then requires process exit status `0`. The OpenClaw session store must
append two nonempty `user` and `assistant` record pairs in one session. The helper
does not compare message content. Terminal output is a bounded failure diagnostic only.
Deterministic unit tests separately prove selection of the complete preflight
and lease paths, stale-producer exclusion, the fixed time-unsafe quarantine,
refusal to recover when prior evidence cannot be durably fenced, and the named
Expand Down
3 changes: 2 additions & 1 deletion test/e2e/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,8 @@ test/e2e/
- `.github/workflows/platform-vitest-main.yaml` publishes `CI / Platform Compatibility`.
It runs the Ubuntu 26.04 compatibility contracts and four full-suite Vitest shards on each of macOS and WSL.
Each macOS shard installs the pinned OpenShell formula.
Shard 1 has a 60-minute budget for live E2E; the other shards have 30 minutes.
Shard 1 has a 150-minute job timeout. Its live E2E has a 70-minute timeout, and every other step shares the remaining job time.
The other shards have 30 minutes.
WSL shard 1 has a 180-minute budget for root-required contracts and live E2E; the other shards have 90 minutes.
On shard 1, the workflow runs focused macOS and WSL live E2E only when the run tests `main` and Docker is available.
Otherwise, those live tests skip and the platform contracts remain as evidence.
Expand Down
5 changes: 4 additions & 1 deletion test/e2e/live/full-e2e.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,17 @@ import {
import { readFullE2eColdWorkloadEvidence } from "./full-e2e-workload-evidence.ts";
import { runOpenClawLaunchReadinessLeaseTurns } from "./launch-agent-turn.ts";
import { bindApprovedPrBaseForBaseImageComparison } from "./pr-base-comparison.ts";
import { FULL_E2E_TEST_TIMEOUT_MS } from "../../../tools/e2e/full-e2e-timeout-contract.mts";
import { parseOpenClawJsonDocuments } from "../../../src/lib/openclaw/agent-json-provenance.ts";

const SANDBOX_NAME = process.env.NEMOCLAW_SANDBOX_NAME ?? "e2e-full";
const FULL_E2E_TARGET_ID = process.env.E2E_TARGET_ID ?? "full-e2e";
const SETUP_MODE = process.env.NEMOCLAW_E2E_SETUP_MODE ?? "source-install";
const USE_PREINSTALLED_LAUNCHABLE = SETUP_MODE === "preinstalled-launchable";
const PORTABLE_PROFILE = process.env.NEMOCLAW_EXPERIMENTAL_PROFILE === "portable";
const LIVE_TIMEOUT_MS = testTimeout(50 * 60_000);
// Cold install plus two launch turns with one provider retry each can consume
// almost 50 minutes before the remaining assertions and cleanup run.
const LIVE_TIMEOUT_MS = testTimeout(FULL_E2E_TEST_TIMEOUT_MS);
const INSTALL_TIMEOUT_MS = execTimeout(25 * 60_000);
const FIRST_TURN_TIMEOUT_MS = 240_000;
const MAX_SILENCE_SECS = 60;
Expand Down
Loading
Loading