test(e2e): handle satisfied 4462 legacy repro - #5412
Merged
Conversation
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Contributor
📝 WalkthroughWalkthroughThe E2E test script for issue ChangesLegacy scope-upgrade gateway-mode handling
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related issues
Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in the Show a code coverage summary of the most covered files.
TypeScript / code-coverage/cliThe overall coverage in the Show a code coverage summary of the most covered files.
Updated |
Contributor
E2E Advisor RecommendationRequired E2E: None Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
|
Contributor
Vitest E2E Scenario RecommendationRequired Vitest E2E scenarios: None Full Vitest E2E advisor summaryVitest E2E Scenario AdvisorBase: Required Vitest E2E scenarios
Optional Vitest E2E scenarios
Relevant changed files
|
Contributor
PR Review AdvisorFindings: 0 needs attention, 3 worth checking, 0 nice ideas Review findings🛠️ Needs attention
🔎 Worth checking
🌱 Nice ideas
Consider writing more tests for
This is an automated advisory review. A human maintainer must make the final merge decision. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
test/e2e/test-issue-4462-scope-upgrade-approval.sh (1)
1002-1014:⚠️ Potential issue | 🟠 Major | ⚡ Quick winDon't exit the reproduced legacy path before Phase 5.
This branch still calls
finish_successimmediately after characterization, so the legacy-repro path that does reproduce the pending failure never executes the final gateway-path agent check. That lets this mode pass without the verification the PR description says is preserved.Suggested change
if [ "$TEST_MODE" = "legacy-repro" ] && [ "$LEGACY_SCOPE_UPGRADE_NOT_REPRODUCED" != "1" ]; then legacy_gateway_pinned_approval_characterization "$scope_request_id" || exit 1 if [ "$FAIL" -gt 0 ]; then section "Summary" echo "" printf ' Total: %d | \033[32mPass: %d\033[0m | \033[31mFail: %d\033[0m\n' \ "$TOTAL" "$PASS" "$FAIL" echo "" echo "RESULT: FAILED - ${FAIL} test(s) failed" exit 1 fi - finish_success "RESULT: PASSED - `#4462` legacy gateway-pinned approval behavior characterized and final state handled" + info "Legacy gateway-pinned approval behavior characterized; continuing to final gateway-path verification" fiBased on PR objectives, this path is still supposed to run the final gateway-mode agent verification before reporting success.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/test-issue-4462-scope-upgrade-approval.sh` around lines 1002 - 1014, The issue is that the legacy-repro path calls finish_success immediately after the characterization phase, which causes an early exit and prevents Phase 5 (the final gateway-path agent verification) from executing. To fix this, remove the finish_success call from within the legacy-repro conditional block so that control flow continues to execute Phase 5. The finish_success call for the legacy-repro path should either be moved to after Phase 5 completes or restructured so that the legacy-repro verification logic does not exit prematurely but allows the subsequent gateway-mode agent check to run before reporting success.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@test/e2e/test-issue-4462-scope-upgrade-approval.sh`:
- Around line 1002-1014: The issue is that the legacy-repro path calls
finish_success immediately after the characterization phase, which causes an
early exit and prevents Phase 5 (the final gateway-path agent verification) from
executing. To fix this, remove the finish_success call from within the
legacy-repro conditional block so that control flow continues to execute Phase
5. The finish_success call for the legacy-repro path should either be moved to
after Phase 5 completes or restructured so that the legacy-repro verification
logic does not exit prematurely but allows the subsequent gateway-mode agent
check to run before reporting success.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: de6a20c3-972d-47ff-9bcf-8f6105fa356f
📒 Files selected for processing (1)
test/e2e/test-issue-4462-scope-upgrade-approval.sh
Contributor
Selective E2E Results — ✅ All requested jobs passedRun: 27486841808
|
21 tasks
This was referenced Jun 14, 2026
nightly-e2e: issue-4462 gateway-pinned approval E2E flakes on transient gateway unreachability
#5377
Closed
13 tasks
This was referenced Jun 25, 2026
jyaunches
added a commit
that referenced
this pull request
Jun 25, 2026
## Summary Restore the Kimi-specific issue #5800 parity work for package `P0-D`; existing recovery and scope-upgrade package rows are explicitly mapped as pre-existing coverage and revalidated context, but not changed acceptance scope in this PR. ## Related Issues Refs #5800 Refs #5098 Refs #5342 Refs #5401 Refs #5406 Refs #5412 Refs #5413 Refs #5625 Refs #5760 ## Scope gate - Package: `P0-D — Recovery, Kimi, and scope-upgrade parity` - Included PRs all merged and touched `test/e2e`: yes - Changed acceptance scope in this PR: Kimi public-NVIDIA/mock parity (`D2`, `D3`) - Existing package rows revalidated without diff changes: recovery (`D1`) and scope-upgrade (`D4`) - Out of scope: unmerged/non-bash PRs; shell lane retirement / PR #5756 cleanup ## Parity map | ID | Source PR | Contract | Inference classification | Vitest assertion / waiver | Status | | --- | --- | --- | --- | --- | --- | | D1 | #5342, #5401 | Recovery proxy env sourcing, missing proxy-env warning, guard retention, ciao/networkInterfaces preload, and crash-loop stability are pre-existing package coverage. | `hermetic-default` | Existing `test/e2e-scenario/live/issue-2478-crash-loop-recovery.test.ts`, `test/e2e-scenario/support-tests/e2e-recovery-helpers.test.ts`; selective run `28186561267` job `issue-2478-crash-loop-recovery-vitest` passed. No diff changes here. | existing / revalidated context | | D2 | #5401 | Kimi remains a public-NVIDIA model/provider contract when run in trusted selective CI, while retaining mock fallback for local/untrusted validation. | `public-nvidia required` | `.github/workflows/e2e-vitest-scenarios.yaml`, `test/e2e-scenario/live/kimi-inference-compat.test.ts`, `test/e2e-scenario/support-tests/kimi-inference-compat-helpers.test.ts`, `test/e2e-script-workflow.test.ts` | covered / changed | | D3 | #5413, #5625 | Kimi multiturn tool calls split `hostname; date; uptime`, preserve tool-result flow, reject abandoned/continue traces, and normalize final punctuation. | `public-nvidia required` with mock fallback | `test/e2e-scenario/live/kimi-inference-compat-helpers.ts` trajectory assertions; selective run `28190216767` job `kimi-inference-compat-vitest` passed on the previous head; latest run `28193896380` passed on `f36fef6da`. | covered / changed | | D4 | #5406, #5412, #5760 | Scope-upgrade approval tolerates preapproved / not-reproduced states, denies `operator.admin` leakage, stays on gateway/no embedded fallback, and accepts whitespace-normalized `42`; this is pre-existing package coverage. | `hosted-compatible capable` | Existing `test/e2e-scenario/live/issue-4462-scope-upgrade-approval.test.ts`; selective run `28186561267` job `issue-4462-scope-upgrade-approval-vitest` passed. No diff changes here. | existing / revalidated context | ## Inference mode support - Default mode for touched live target: Kimi `mock` unless workflow selects `public-nvidia`. - Real inference support preserved: yes for Kimi public NVIDIA; yes for existing scope-upgrade hosted-compatible; not required for recovery. - Modes validated in this PR: Kimi public NVIDIA via selective workflows `28188683830`, `28190216767`; latest follow-up validation `28193896380` is running for head `f36fef6da`. Kimi helper/mock behavior via local support tests. - Source-of-truth contract: `NEMOCLAW_E2E_INFERENCE_MODE` is the canonical selector; absent selector defaults to mock for local/untrusted validation; unknown explicit values now fail closed; legacy `NEMOCLAW_KIMI_USE_MOCK=0` remains only as a temporary shell-lane compatibility alias until shell retirement. - Secret boundary: public Kimi workflow passes only `NVIDIA_API_KEY`; helper probe envs are secret-free by default; raw public NVIDIA key handoff is limited to onboard; sandbox `openclaw agent` now runs with a secret-free env and uses the configured `nvidia-prod` route. ## Validation - [x] `npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/kimi-inference-compat-helpers.test.ts` - [x] `npx vitest run test/e2e-script-workflow.test.ts` - [x] `npm run typecheck:cli` - [x] `npm run test-conditionals:scan -- --top 25` - [x] `npx prek run --all-files --stage pre-push --skip tsc-plugin --skip tsc-js --skip tsc-cli --skip version-tag-sync --skip test-cli --skip test-plugin --skip source-shape-test-budget --skip test-file-size-budget --skip test-skills-yaml` - [x] `git diff --check` - [x] Kimi selective E2E / Vitest Scenarios on previous head: https://github.com/NVIDIA/NemoClaw/actions/runs/28190216767 - [x] Kimi selective E2E / Vitest Scenarios after review-gap fixes: https://github.com/NVIDIA/NemoClaw/actions/runs/28193896380 - [x] Existing recovery/scope rows revalidated in selective run: https://github.com/NVIDIA/NemoClaw/actions/runs/28186561267 (`issue-2478-crash-loop-recovery-vitest` ✅, `issue-4462-scope-upgrade-approval-vitest` ✅; Kimi in that stale run was superseded) - [ ] Local live mock Kimi: attempted but blocked by local Docker daemon unavailable (`Cannot connect to the Docker daemon at unix:///Users/jyaunches/.docker/run/docker.sock`). CI selective run is the live validation path for this head. ## Follow-ups / waivers - None. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for running Kimi compatibility e2e checks in either mock or public NVIDIA mode. * The live scenario now adapts its setup, redaction, and traffic validation based on the selected mode. * **Bug Fixes** * Improved handling of API key propagation so public NVIDIA runs use the expected credentials without exposing secrets in other paths. * **Tests** * Added coverage for mode selection, API key validation, workflow environment wiring, and the new public NVIDIA Vitest lane. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
## Summary Restore the Kimi-specific issue NVIDIA#5800 parity work for package `P0-D`; existing recovery and scope-upgrade package rows are explicitly mapped as pre-existing coverage and revalidated context, but not changed acceptance scope in this PR. ## Related Issues Refs NVIDIA#5800 Refs NVIDIA#5098 Refs NVIDIA#5342 Refs NVIDIA#5401 Refs NVIDIA#5406 Refs NVIDIA#5412 Refs NVIDIA#5413 Refs NVIDIA#5625 Refs NVIDIA#5760 ## Scope gate - Package: `P0-D — Recovery, Kimi, and scope-upgrade parity` - Included PRs all merged and touched `test/e2e`: yes - Changed acceptance scope in this PR: Kimi public-NVIDIA/mock parity (`D2`, `D3`) - Existing package rows revalidated without diff changes: recovery (`D1`) and scope-upgrade (`D4`) - Out of scope: unmerged/non-bash PRs; shell lane retirement / PR NVIDIA#5756 cleanup ## Parity map | ID | Source PR | Contract | Inference classification | Vitest assertion / waiver | Status | | --- | --- | --- | --- | --- | --- | | D1 | NVIDIA#5342, NVIDIA#5401 | Recovery proxy env sourcing, missing proxy-env warning, guard retention, ciao/networkInterfaces preload, and crash-loop stability are pre-existing package coverage. | `hermetic-default` | Existing `test/e2e-scenario/live/issue-2478-crash-loop-recovery.test.ts`, `test/e2e-scenario/support-tests/e2e-recovery-helpers.test.ts`; selective run `28186561267` job `issue-2478-crash-loop-recovery-vitest` passed. No diff changes here. | existing / revalidated context | | D2 | NVIDIA#5401 | Kimi remains a public-NVIDIA model/provider contract when run in trusted selective CI, while retaining mock fallback for local/untrusted validation. | `public-nvidia required` | `.github/workflows/e2e-vitest-scenarios.yaml`, `test/e2e-scenario/live/kimi-inference-compat.test.ts`, `test/e2e-scenario/support-tests/kimi-inference-compat-helpers.test.ts`, `test/e2e-script-workflow.test.ts` | covered / changed | | D3 | NVIDIA#5413, NVIDIA#5625 | Kimi multiturn tool calls split `hostname; date; uptime`, preserve tool-result flow, reject abandoned/continue traces, and normalize final punctuation. | `public-nvidia required` with mock fallback | `test/e2e-scenario/live/kimi-inference-compat-helpers.ts` trajectory assertions; selective run `28190216767` job `kimi-inference-compat-vitest` passed on the previous head; latest run `28193896380` passed on `f36fef6da`. | covered / changed | | D4 | NVIDIA#5406, NVIDIA#5412, NVIDIA#5760 | Scope-upgrade approval tolerates preapproved / not-reproduced states, denies `operator.admin` leakage, stays on gateway/no embedded fallback, and accepts whitespace-normalized `42`; this is pre-existing package coverage. | `hosted-compatible capable` | Existing `test/e2e-scenario/live/issue-4462-scope-upgrade-approval.test.ts`; selective run `28186561267` job `issue-4462-scope-upgrade-approval-vitest` passed. No diff changes here. | existing / revalidated context | ## Inference mode support - Default mode for touched live target: Kimi `mock` unless workflow selects `public-nvidia`. - Real inference support preserved: yes for Kimi public NVIDIA; yes for existing scope-upgrade hosted-compatible; not required for recovery. - Modes validated in this PR: Kimi public NVIDIA via selective workflows `28188683830`, `28190216767`; latest follow-up validation `28193896380` is running for head `f36fef6da`. Kimi helper/mock behavior via local support tests. - Source-of-truth contract: `NEMOCLAW_E2E_INFERENCE_MODE` is the canonical selector; absent selector defaults to mock for local/untrusted validation; unknown explicit values now fail closed; legacy `NEMOCLAW_KIMI_USE_MOCK=0` remains only as a temporary shell-lane compatibility alias until shell retirement. - Secret boundary: public Kimi workflow passes only `NVIDIA_API_KEY`; helper probe envs are secret-free by default; raw public NVIDIA key handoff is limited to onboard; sandbox `openclaw agent` now runs with a secret-free env and uses the configured `nvidia-prod` route. ## Validation - [x] `npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/kimi-inference-compat-helpers.test.ts` - [x] `npx vitest run test/e2e-script-workflow.test.ts` - [x] `npm run typecheck:cli` - [x] `npm run test-conditionals:scan -- --top 25` - [x] `npx prek run --all-files --stage pre-push --skip tsc-plugin --skip tsc-js --skip tsc-cli --skip version-tag-sync --skip test-cli --skip test-plugin --skip source-shape-test-budget --skip test-file-size-budget --skip test-skills-yaml` - [x] `git diff --check` - [x] Kimi selective E2E / Vitest Scenarios on previous head: https://github.com/NVIDIA/NemoClaw/actions/runs/28190216767 - [x] Kimi selective E2E / Vitest Scenarios after review-gap fixes: https://github.com/NVIDIA/NemoClaw/actions/runs/28193896380 - [x] Existing recovery/scope rows revalidated in selective run: https://github.com/NVIDIA/NemoClaw/actions/runs/28186561267 (`issue-2478-crash-loop-recovery-vitest` ✅, `issue-4462-scope-upgrade-approval-vitest` ✅; Kimi in that stale run was superseded) - [ ] Local live mock Kimi: attempted but blocked by local Docker daemon unavailable (`Cannot connect to the Docker daemon at unix:///Users/jyaunches/.docker/run/docker.sock`). CI selective run is the live validation path for this head. ## Follow-ups / waivers - None. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for running Kimi compatibility e2e checks in either mock or public NVIDIA mode. * The live scenario now adapts its setup, redaction, and traffic validation based on the selected mode. * **Bug Fixes** * Improved handling of API key propagation so public NVIDIA runs use the expected credentials without exposing secrets in other paths. * **Tests** * Added coverage for mode selection, API key validation, workflow environment wiring, and the new public NVIDIA Vitest lane. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
wscurran
added
area: ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stabilizes the #4462 legacy characterization E2E when the modern sandbox no longer reproduces a pending gateway-pinned scope-upgrade failure. If the trigger agent completes through gateway mode without fallback markers and no pending request appears, the script now treats that as an explicit non-repro/satisfied outcome, still verifies no
operator.admingrant, and continues to the final gateway-path agent proof.Related Issue
Related to #4462.
Changes
LEGACY_SCOPE_UPGRADE_NOT_REPRODUCEDstate fortest/e2e/test-issue-4462-scope-upgrade-approval.sh.legacy-repromode, accept a trigger-agent success with preserved gateway URL and no fallback/pairing markers as a non-reproducible legacy condition.operator.adminassertions, and still run the final gateway-mode agent verification before success.Type of Change
Verification
npx prek run --from-ref main --to-ref HEADpassesnpm testpasses (broad runtime changes only)npm run docsbuilds without warnings (doc changes only)Targeted verification:
bash -n test/e2e/test-issue-4462-scope-upgrade-approval.shNightly evidence motivating this patch:
issue-4462-gateway-pinned-approval-characterization-e2ebecause the legacy trigger agent completed successfully through gateway mode without producing a pending scope-upgrade request to characterize.Docs review: no user-facing docs changes needed; this is E2E harness stabilization only.
Signed-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit