fix(ci): fix H200 nightly benchmark model path, worker logs and CUDA errors

[key4ng]

Description

Problem

The H200 nightly benchmark tests had three issues:

1. Worker logs were not being collected for the H200 single-worker and H100 multi-worker jobs, making it difficult to diagnose worker failures. Logs were sent to /dev/null because E2E_LOG_DIR was not set.
2. The Llama-4-Maverick model was hitting cudaErrorIllegalAddress (illegal memory access) due to aggressive memory settings and incompatible FlashInfer MLA attention backend configuration.
3. The genai-bench docker container can't access the model in node

Solution

• Add E2E_LOG_DIR and mkdir -p to the H200 single-worker and H100 multi-worker jobs so worker logs are written to files and included in uploaded artifacts.
• Switch Llama-4-Maverick attention backend from flashinfer (with MLA-specific flags) to fa3, and reduce --mem-fraction-static from 0.9 to 0.82 to avoid out-of-bounds memory access.

Changes

• .github/workflows/nightly-benchmark.yml: Add E2E_LOG_DIR: nightly_gateway_logs and mkdir -p nightly_gateway_logs to multi-worker and single-worker-h200 jobs
• e2e_test/infra/model_specs.py: Replace FlashInfer MLA backend flags with --attention-backend=fa3 and lower static memory fraction to 0.82 for Llama-4-Maverick
• e2e_test/benchmarks/conftest.py: Mount ROUTER_LOCAL_MODEL_PATH into benchmark container
• .github/workflows/nightly-benchmark.yml: Temporary concurrency group rename

Test Plan

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets --all-features -- -D warnings passes
• (Optional) Documentation updated

Summary by CodeRabbit

• New Features

  • Nightly benchmark mode with dedicated gateway log directory and automated log creation for end-to-end runs
  • Option to mount local model directories into benchmark environments for easier local testing

• Chores

  • Updated model runtime configurations and startup arguments to improve performance and compatibility across execution modes

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR enables nightly E2E mode in CI (env + log dir), mounts local model paths into test Docker containers, removes vLLM's --enforce-eager flag, and conditionally adjusts model vllm_args and worker_args for nightly runs.

Changes

Cohort / File(s)	Summary
Nightly Workflow ​.github/workflows/nightly-benchmark.yml	Added E2E_NIGHTLY: "1" and E2E_LOG_DIR: nightly_gateway_logs to multiple jobs and create nightly_gateway_logs before running nightly tests.
E2E Test Benchmarks e2e_test/benchmarks/conftest.py	When ROUTER_LOCAL_MODEL_PATH is set, add a Docker volume mount (-v local_path:local_path) to the container command so local model dir is accessible inside tests.
Model Pool / vLLM Launch e2e_test/infra/model_pool.py	Removed the --enforce-eager flag from the vLLM gRPC command when runtime is vllm.
Model Specs e2e_test/infra/model_specs.py	Added nightly toggle (_is_nightly) to choose vllm_args; for Qwen3-30B-A3B set vllm_args to [] when nightly else ["--enforce-eager"]; updated Llama-4-Maverick worker_args and added vllm_args.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• fix(e2e): migrate genai-bench to Docker and fix router pipe hang  #403 — Related edits to CI workflow env/log-dir and test infra changes (conftest.py, model_pool.py).
• chore(e2e): overhaul nightly benchmark summary and trim model list #392 — Related updates to nightly benchmark workflow and logging directory plumbing.

Suggested reviewers

• CatherineSue
• slin1237
• XinyueZhang369

Poem

🐇 Nightly hops with logs in tow,
Volumes mounted row by row,
Flags trimmed down, configs aligned,
Models wake in eager kind,
A rabbit cheers for nightly flow 🥕

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: fixing H200 nightly benchmark issues including model path mounting, worker log collection, and CUDA memory errors.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Merge Conflict Detection	✅ Passed	✅ No merge conflicts detected when merging into main

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix-h200-test

---

No actionable comments were generated in the recent review. 🎉

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Summary of Changes

Hello @key4ng, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves an issue related to model path handling, specifically for H200 models, within the benchmark testing setup. It enhances the command builder to correctly mount local model directories and streamlines the process of passing necessary environment variables to the container, ensuring models are accessible for testing.

Highlights

• Local Model Path Mounting: Introduced functionality to mount local model directories into the benchmark container environment by checking for the ROUTER_LOCAL_MODEL_PATH environment variable.
• Environment Variable Handling Refinement: Refactored the passing of environment variables to explicitly include HF_TOKEN if set, while removing the previous generic loop that also included HF_HOME.

Changelog

• e2e_test/benchmarks/conftest.py
  • Added logic to mount a local model directory if the ROUTER_LOCAL_MODEL_PATH environment variable is defined.
  • Modified the environment variable passing mechanism to specifically include HF_TOKEN and remove HF_HOME from the list of variables to be passed.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to mount a local model directory into the benchmark container, which is useful for environments like H200 where models are stored locally. The implementation correctly uses the ROUTER_LOCAL_MODEL_PATH environment variable to configure the volume mount. However, the change also removes the pass-through of the HF_HOME environment variable, which could be an unintended breaking change. My review includes a suggestion to restore this functionality to avoid breaking existing workflows.

[key4ng]

waiting for finish: https://github.com/lightseekorg/smg/actions/runs/21957536020/job/63425914078

[key4ng]

gpt-oss grpc vllm test time reduced to 1 hr
https://github.com/lightseekorg/smg/actions/runs/21968856205/job/63465395566

[key4ng]

llama 4 grpc/http sglang: https://github.com/lightseekorg/smg/actions/runs/21957536020/job/63425914078
llama 4 grpc vllm: https://github.com/lightseekorg/smg/actions/runs/21967166149/job/63461105742

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@e2e_test/infra/model_specs.py`:
- Around line 105-106: Update the stale inline comment next to the
"--mem-fraction-static=0.82" argument in the list (near the
"--attention-backend=fa3" entry): change the comment text from "90% GPU memory
for static allocation" to reflect the actual value, e.g. "82% GPU memory for
static allocation", so the comment matches the "--mem-fraction-static=0.82"
setting.
- Line 111: The inline comment for the argument "--attention-backend=FLASHINFER"
is misleading (it says "fa3 attention backend"); update the comment to correctly
indicate that this flag selects the FLASHINFER attention backend (vLLM
equivalent) rather than fa3, so change the comment next to
"--attention-backend=FLASHINFER" in model_specs.py to reference FLASHINFER/vLLM.