Skip to content

test(e2e): synchronize fixture request proofs - #8014

Merged
cv merged 3 commits into
mainfrom
codex/fix-e2e-fixture-request-contracts
Jul 31, 2026
Merged

test(e2e): synchronize fixture request proofs#8014
cv merged 3 commits into
mainfrom
codex/fix-e2e-fixture-request-contracts

Conversation

@prekshivyas

@prekshivyas prekshivyas commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

The device-auth and inference-switch live E2E targets now create and await the authenticated inference request that each test proves before reading the request ledger. Assertions inspect only rows produced after that phase begins, and the new evidence artifacts retain at most 20 redacted request projections. This removes dependence on incidental startup traffic and keeps failure diagnostics content-free.

Related Issue

Fixes #8002

Changes

  • Capture a ledger offset immediately before each request-proof phase.
  • Send and await an authenticated /v1/chat/completions request through inference.local.
  • Assert only the new ledger slice and record its phase, count, truncation state, and bounded auth/method/model/path projection.
  • Keep the inference-switch expectation scoped to mock-baseline selections.
  • Product scope is approved for this test-only reliability fix. It changes no supported product surface, ownership, lifecycle, compatibility contract, or production security boundary.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: This changes live E2E fixture orchestration and evidence only. It does not change a user-facing API, CLI, configuration, workflow, default, error, or supported behavior.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: A nine-category security review of exact head dcceb35094b1a2b3334f469bf71e5d2e7ca05989 against base 57f73a509cd7e950e01af1423b2514ef352bec82 passed all categories. The tests use fixed fixture values, quote the JSON payload, keep provider credentials in the existing managed route, cap evidence at 20 projected rows, and publish through the canonical redacting artifact sink. No production authentication, authorization, network-policy, or sandbox boundary changes. CodeRabbit confirmed and resolved its diagnostic-data finding, and the exact-head PR Review Advisor reports no blocking findings.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: The complete diff against upstream/main changes only live E2E fixture proof orchestration, bounded credential-free request-evidence artifacts, assertion diagnostics, comments, and test contract wording. It does not change a user-facing API, CLI, configuration, workflow, default, error, or supported product behavior. Changed explanatory text was reviewed against the NemoClaw writing and documentation rules.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit: Not applicable; scripts/prepare-dgx-station-host.sh is unchanged.
  • Station profile/scenario: Not applicable.
  • Result: Not applicable.
  • Supporting evidence: Not applicable.

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — exact head dcceb35094b1a2b3334f469bf71e5d2e7ca05989, base 57f73a509cd7e950e01af1423b2514ef352bec82, diff SHA-256 880192aa008776938c09665b83990d3bdfca958ea7a5be6b77171842a56b5128: Biome passed; semantic E2E phase coverage passed for 114 tests across 71 files; focused helper tests passed 6/6; Vitest project membership passed for 1,956 files across seven projects; the touched inference-switch file remains at 41 conditionals; CLI build and type-check passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not applicable to this two-file live E2E fixture synchronization. Exact-head required CI and trusted E2E provide the broad repository evidence.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Prekshi Vyas prekshiv@nvidia.com

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The live E2E tests now issue explicit authenticated inference requests, capture requests after each probe, and verify the expected fixture POST instead of relying on incidental traffic.

Changes

Authenticated inference probes

Layer / File(s) Summary
Device authentication probe
test/e2e/live/device-auth-health.test.ts
The test runs an authenticated chat-completions probe after installation, records post-probe requests, and validates the expected fixture POST.
Inference switch baseline probe
test/e2e/live/openclaw-inference-switch.test.ts
The test runs an authenticated baseline probe through the sandbox and validates the recorded POST instead of inspecting incidental traffic.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#6621: Updates live E2E tests with authenticated inference probes against a fake inference endpoint.
  • NVIDIA/NemoClaw#6687: Modifies live inference E2E tests with authenticated sandbox probes.
  • NVIDIA/NemoClaw#6989: Changes the baseline fixture network binding used by a related live inference test.

Suggested labels: area: e2e, integration: openclaw, bug-fix

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Both tests explicitly trigger and await authenticated requests, record post-probe evidence, and report phase-specific request data.
Out of Scope Changes check ✅ Passed The changes are limited to the two tests named in issue #8002 and directly support deterministic fixture request assertions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes synchronizing fixture request proofs, which is the primary change in both live E2E tests.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-e2e-fixture-request-contracts

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit dcceb35 in the codex/fix-e2e-fixtur... branch remains at 96%, unchanged from commit 57f73a5 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit dcceb35 in the codex/fix-e2e-fixtur... branch remains at 81%, unchanged from commit 57f73a5 in the main branch.

Show a code coverage summary of the most impacted files.
File main 57f73a5 codex/fix-e2e-fixtur... dcceb35 +/-
src/lib/onboard/docker-cdi.ts 80% 70% -10%
src/lib/onboard...host-anchors.ts 94% 90% -4%
src/lib/onboard/preflight.ts 80% 80% 0%
src/lib/actions...orchestrator.ts 83% 87% +4%
src/lib/onboard...box-gpu-mode.ts 92% 97% +5%
src/lib/domain/.../connect-env.ts 89% 97% +8%
src/lib/actions...confirmation.ts 69% 79% +10%
src/lib/actions...ocker-health.ts 65% 82% +17%
src/lib/onboard...box-prebuild.ts 74% 92% +18%
src/lib/onboard...flight-ports.ts 33% 67% +34%

Updated July 31, 2026 17:00 UTC

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/live/device-auth-health.test.ts`:
- Around line 133-143: In test/e2e/live/device-auth-health.test.ts lines
133-143, update the phase-1 artifact and related assertions to use a capped,
redacted projection of authenticatedRequests, and limit assertion diagnostics to
the phase and artifact name; do not include request records or
resultText(authenticatedProbe). Apply the same capped redaction and content-free
diagnostics to the corresponding request persistence and assertions in
test/e2e/live/openclaw-inference-switch.test.ts lines 168-184.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2c46a142-a833-4c18-82a9-2748b0875657

📥 Commits

Reviewing files that changed from the base of the PR and between 57f73a5 and a8d7e71.

📒 Files selected for processing (2)
  • test/e2e/live/device-auth-health.test.ts
  • test/e2e/live/openclaw-inference-switch.test.ts

Comment thread test/e2e/live/device-auth-health.test.ts Outdated
@prekshivyas

Copy link
Copy Markdown
Collaborator Author

Cross-issue sweep progress:

  • Step 1: Extract fingerprint (files, symbols, error strings, primary issue)
  • Step 2: Search candidate issues (capped at 30, primary excluded)
  • Step 3: Classify each candidate (4-class with evidence)
  • Step 4: Apply reverse-link boost
  • Step 5: Filter (drop UNRELATED, SAME_ISSUE_DIFF, low-confidence)
  • Step 6: Render report using the repository template

Cross-issue scan — PR #8014

No adjacent fixes or contradictions found above the medium confidence floor.

Suppressed: 0 unrelated, 0 same-issue duplicates.

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections match; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: device-auth-health, inference-routing, network-policy, openclaw-inference-switch

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@prekshivyas

Copy link
Copy Markdown
Collaborator Author

Addressed the actionable codebase-growth-guardrails failure at exact head 2c4bb600b7d9b3c8f4c3c34cd9b6f8db8b8a25a1.

  • Extracted the selected mock-baseline request proof into a named helper; request behavior is unchanged.
  • npm run test-conditionals:scan -- --top 25: the changed file is back to the base count of 41 if statements.
  • E2E semantic phase validation: 114 tests across 71 files.
  • Biome and CLI type-checking pass.
  • npm run validate:pr passes on the new head.
  • The new commit is GitHub Verified (valid).
  • Independent documentation writer review: no-docs-needed, head 2c4bb600b, AGENTS.md blob c052d60aa; the PR receipt has been refreshed.

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>

@senthilr-nv senthilr-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the Files changed for PR #8014 at head dcceb35 against base 57f73a5. I approve this exact head for squash merge.

@cv
cv merged commit 9b0d219 into main Jul 31, 2026
113 of 120 checks passed
@cv
cv deleted the codex/fix-e2e-fixture-request-contracts branch July 31, 2026 17:13
@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance labels Jul 31, 2026
@senthilr-nv senthilr-nv mentioned this pull request Aug 1, 2026
23 tasks
senthilr-nv added a commit that referenced this pull request Aug 1, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the canonical dated changelog entry for `v0.0.100` so the
maintainer release plan can verify the pre-tag documentation
prerequisite. The entry summarizes the user-facing changes merged since
`v0.0.99` and links to the relevant guides.

## Changes

- Add `docs/changelog/2026-07-31.mdx` with the exact `## v0.0.100`
heading.
- Cover restored OpenClaw pairing, transactional replacement, Deep
Agents Code, onboarding recovery, lifecycle cleanup, Hermes builds, host
provenance, documentation, and trusted E2E evidence.
- Distinguish active Docker and Kubernetes runtime-bundle enforcement
from the still-inactive managed shared-state transaction foundation.

## Source Coverage

The release entry maps the doc-impacting merged PRs in the
`v0.0.99..main` release range to `docs/changelog/2026-07-31.mdx`: #8021,
#8024, #7973, #8028, #7947, #7788, #7884, #8023, #7969, #8020, #7989,
#8000, #7907, #7942, #7567, #8013, #7955, #8017, #8014, #8015, #7629,
#7644, #7821, #7971, and #7991.

PR #7974 was reviewed after the final rebase and excluded because it
changes internal maintainer-skill attribution policy and tests only; it
does not change a user-facing product or documentation surface.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification: the
changelog contract test validates the dated entry, version heading, SPDX
form, and route constraints.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/changelog/2026-07-31.mdx`; exact-head review passed
for `6093f44f`; writing rules and documentation style reviewed; `npx
vitest run test/changelog-docs.test.ts` passed 6/6; `npm run docs`
passed with zero Fern errors and two generic Fern upgrade notices.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 6093f44 -->
<!-- docs-review-agents-blob-sha: 3dd7c24 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable; no DGX Station host script changed.
- Station profile/scenario: Not applicable.
- Result: Not applicable.
- Supporting evidence: Not applicable.

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npx
vitest run test/changelog-docs.test.ts` passed 6/6 at `6093f44f`.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to a dated
prose-only release entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
validation passed with zero errors; Fern emitted two generic upgrade
notices.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— the changelog entry has the required parser-safe MDX SPDX header;
dated changelog entries intentionally do not use page frontmatter.

---

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Added release notes for v0.0.100.
* Documented improvements to restore pairing, sandbox replacement,
onboarding recovery, lifecycle cleanup, runtime handling, build support,
host readiness, and end-to-end validation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(e2e): Stabilize nondeterministic fixture request assertions in device-auth-health and openclaw-inference-switch

4 participants