Skip to content

fix(e2e): skip redundant inference verify in hermes-inference-switch - #4109

Closed
hunglp6d wants to merge 1 commit into
mainfrom
fix/nightly-e2e-infer-switch-no-verify-7c7f7a4
Closed

fix(e2e): skip redundant inference verify in hermes-inference-switch#4109
hunglp6d wants to merge 1 commit into
mainfrom
fix/nightly-e2e-infer-switch-no-verify-7c7f7a4

Conversation

@hunglp6d

@hunglp6d hunglp6d commented May 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

The hermes-inference-switch-e2e nightly job failed because nemohermes inference set timed out (60 s) verifying the z-ai/glm-5.1 model at https://integrate.api.nvidia.com/v1/chat/completions. The verification is redundant: Phase 4 of the test already validates live inference after the switch via check_inference_local and check_hermes_api_chat.

This PR passes --no-verify to skip the endpoint verification timeout without reducing coverage.

Related Issue

Fixes #4111

Changes

  • test/e2e/test-hermes-inference-switch.sh: Added --no-verify to the nemohermes inference set call, with a comment explaining why verification is redundant in this test.

Validation

The custom-e2e validation branch could not be pushed because the PAT lacks workflow scope (GitHub blocks pushes of .github/workflows/ files without it). To validate manually:

gh workflow run nightly-e2e.yaml --repo NVIDIA/NemoClaw \
  --ref fix/nightly-e2e-infer-switch-no-verify-7c7f7a4 \
  -f jobs=hermes-inference-switch-e2e
  • Original failing run: 26318179268 on 7c7f7a428624ad72082d7b11395e25d7ae43daad
  • Targeted job: hermes-inference-switch-e2e (#77481693221)

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • No secrets, API keys, or credentials committed

AI Disclosure

  • AI-assisted — tool: Claude Code (nemoclaw-diagnosis skill)

Signed-off-by: Hung Le hple@nvidia.com

The nemohermes inference set call was timing out (60 s) during endpoint
verification against z-ai/glm-5.1 on NVIDIA Endpoints.  This step is
redundant because Phase 4 of the test already validates live inference
after the switch.  Pass --no-verify to eliminate the flaky timeout
without reducing coverage.

Fixes nightly-e2e run 26318179268 job hermes-inference-switch-e2e.

Signed-off-by: Hung Le <hple@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented May 23, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented May 23, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 34d344e6-986c-47de-ac6e-ee378d28af69

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/nightly-e2e-infer-switch-no-verify-7c7f7a4

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 0 nice ideas
Top item: No actionable code-review findings

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@github-actions

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: hermes-inference-switch-e2e

Dispatch hint: hermes-inference-switch-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. No required E2E is recommended because the PR only changes an existing E2E test script and cannot affect NemoClaw runtime behavior or real assistant user flows.

Optional E2E

  • hermes-inference-switch-e2e (high): Optional self-validation of the modified E2E script. This is not merge-blocking because the PR is tests-only, but running it would confirm that skipping upfront verification still leaves the live post-switch assertions sufficient and passing.

New E2E recommendations

  • None.

Dispatch hint

  • Workflow: .github/workflows/nightly-e2e.yaml
  • jobs input: hermes-inference-switch-e2e

@github-actions

Copy link
Copy Markdown
Contributor

E2E Scenario Advisor Recommendation

Required scenario E2E: None
Optional scenario E2E: None

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

  • None. No scenario workflow, scenario metadata, scenario runtime, or validation-suite files changed.

Optional scenario E2E

  • None.

Relevant changed files

  • None.

@hunglp6d

Copy link
Copy Markdown
Collaborator Author

Closed by #4152, which retries the transient verify and falls back to --no-verify — exactly the failure mode this issue hit. Today's nightly hermes-inference-switch-e2e is green.

@hunglp6d hunglp6d closed this May 25, 2026
@wscurran wscurran added the bug-fix PR fixes a bug or regression label Jun 8, 2026
@cv
cv deleted the fix/nightly-e2e-infer-switch-no-verify-7c7f7a4 branch June 28, 2026 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nightly-e2e: hermes-inference-switch-e2e fails — inference verify timeout on z-ai/glm-5.1

2 participants