Skip to content

test(e2e): migrate gateway health honesty to Vitest - #5141

Closed
cv wants to merge 18 commits into
mainfrom
codex/e2e-migrate-gateway-health-honest
Closed

test(e2e): migrate gateway health honesty to Vitest#5141
cv wants to merge 18 commits into
mainfrom
codex/e2e-migrate-gateway-health-honest

Conversation

@cv

@cv cv commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Migrates the gateway health honesty regression from its legacy shell wrapper into a Vitest live scenario. The regression workflow now runs the scenario directly through e2e-scenarios-live and keeps artifacts under the shared fixture layout.

Related Issue

Refs #4941

Changes

  • Add test/e2e-scenario/live/gateway-health-honest.test.ts for gateway health failure and recovery assertions.
  • Replace test/e2e/test-gateway-health-honest.sh in regression-e2e.yaml with direct Vitest execution.
  • Extend test/regression-e2e-workflow.test.ts so the workflow contract locks the Vitest scenario and retired shell path.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

@cv cv self-assigned this Jun 10, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jun 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 62371b93-3525-49cb-827f-c79fb090b40e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/e2e-migrate-gateway-health-honest

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: gateway-health-honest-e2e
Optional E2E: None

Dispatch hint: gateway-health-honest-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/codex/e2e-simplify-migration-tracking
Head: HEAD
Confidence: high

Required E2E

  • gateway-health-honest-e2e (low): This PR directly changes the gateway-health-honest regression E2E job and migrates its implementation from a shell script to a Vitest live scenario. Run the targeted lane to verify the workflow selector, dependency setup, Vitest project invocation, fake OpenShell/gateway harness, failure assertions, and artifact upload path all work in CI.

Optional E2E

  • None.

New E2E recommendations

  • None.

Dispatch hint

  • Workflow: .github/workflows/regression-e2e.yaml
  • jobs input: gateway-health-honest-e2e

@github-actions

github-actions Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

E2E Scenario Advisor Recommendation

Required scenario E2E: e2e-scenarios-all
Optional scenario E2E: None

Dispatch required scenario E2E:

  • gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/codex/e2e-simplify-migration-tracking
Head: HEAD
Confidence: medium

Required scenario E2E

  • e2e-scenarios-all: PR adds a live Vitest scenario test under test/e2e-scenario/live. The new gateway-health-honest test is not a trusted-main live-supported typed registry scenario ID, so a targeted scenario dispatch is not valid; run the scenario fan-out through the canonical Vitest scenario workflow.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Optional scenario E2E

  • None.

Relevant changed files

  • test/e2e-scenario/live/gateway-health-honest.test.ts

@github-actions

github-actions Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 0 nice ideas
Since last review: 0 prior items resolved, 0 still apply, 0 new items found

Consider writing more tests for
  • **Runtime validation** — Run the gateway-health-honest-e2e workflow lane selection and verify it invokes test/e2e-scenario/live/gateway-health-honest.test.ts rather than the deleted shell script.. Static workflow and Vitest contract coverage is strong, but the changed GitHub Actions lane, YAML execution semantics, npm/Vitest invocation, artifact upload behavior, and runner PATH/env isolation are best validated at runtime.
  • **Runtime validation** — Validate the gateway-health-honest artifact upload contains the Vitest fixture layout under e2e-artifacts/vitest/gateway-health-honest/<test-slug>/ on both pass and failure paths.. Static workflow and Vitest contract coverage is strong, but the changed GitHub Actions lane, YAML execution semantics, npm/Vitest invocation, artifact upload behavior, and runner PATH/env isolation are best validated at runtime.
  • **Runtime validation** — Exercise the Vitest scenario on a clean runner without a real OpenShell installation to confirm the fake PATH/env boundary is sufficient and the sabotage pre-assertion proves the fake gateway binary was invoked.. Static workflow and Vitest contract coverage is strong, but the changed GitHub Actions lane, YAML execution semantics, npm/Vitest invocation, artifact upload behavior, and runner PATH/env isolation are best validated at runtime.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure refactor PR restructures code without intended behavior change labels Jun 10, 2026
@wscurran

Copy link
Copy Markdown
Contributor

Base automatically changed from codex/e2e-simplify-migration-tracking to main June 10, 2026 20:53
@jyaunches

Copy link
Copy Markdown
Contributor

Closing this draft codex migration PR as superseded by the simplified #5098 per-script migration process. The branch carries broad/stale framework-era diff noise; if this test is still needed, we'll salvage the useful core in a fresh/simple one-script PR using nemoclaw-e2e-legacy-migrate.

@jyaunches jyaunches closed this Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure refactor PR restructures code without intended behavior change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants