Skip to content

[CI] Align extraction test with canonical auxiliary layer order - #55457

Merged
khluu merged 1 commit into
vllm-project:mainfrom
khluu:codex/fix-hidden-state-extraction-test-order
Sep 6, 2026
Merged

[CI] Align extraction test with canonical auxiliary layer order#55457
khluu merged 1 commit into
vllm-project:mainfrom
khluu:codex/fix-hidden-state-extraction-test-order

Conversation

@khluu

@khluu khluu commented Sep 5, 2026

Copy link
Copy Markdown
Member

The H200 MIG hidden-state extraction lane fails its predictable-model layer assertion and then cannot start the following Qwen3.5 test because the first test's engine retains GPU memory (main build 87376).

#50514 intentionally canonicalizes auxiliary layer IDs by sorting and deduplicating them. The integration test supplies [5, 2, 10] but still expects the output in that input order, while the model now returns [2, 5, 10]. Keep the unsorted input and check all three extraction scenarios against canonical order, consistent with the existing auxiliary-layer unit test. Run the predictable-model test in its own process using the existing helper so engine memory is released before the next test.

Duplicate checks: open-PR searches for extraction auxiliary hidden sorted and extract_hidden_states found no equivalent ordering/process-isolation fix; existing extraction PRs address separate cache layout, model coverage, and documentation changes.

Validation:

  • All applicable pre-commit hooks passed on commit; git diff --check passed.
  • VLLM_TARGET_DEVICE=cpu .venv/bin/python -m pytest -q tests/v1/worker/test_eagle3_aux_hidden_states_pp.py: 2 passed, confirming the established sorted/deduplicated contract.
  • H200 GPU 2 on h200-ci-1, existing main image 784cac7c424db2f2fdc879b672abad7e231f5698, with this test and current-main vllm/model_executor/models/interfaces.py mounted: /tmp/.venv/bin/python -m pytest -v tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py -k 'predictable or qwen35': 2 passed, 1 deselected in 132.69s. The venv was created with uv venv --system-site-packages /tmp/.venv.
  • This is a test-only change; no model behavior changed. The exact 18 GiB MIG lane still needs CI validation: the local check used a full H200 and an existing image with the current interface overlaid, rather than a complete rebuild of current main.

AI assistance was used for investigation, implementation, and validation. Draft for human review and exact-lane CI; human line-by-line review has not been represented as completed.

CI follow-up: Buildkite 87388 was triggered for this PR head. The exact regression job(s) are enabled: (H200 MIG 18GB) Extract Hidden States Integration. At 2026-09-05T10:52:05+00:00, they were waiting for image/dependency jobs; no successful end-to-end result is claimed.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: khluu <khluu000@gmail.com>
@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mergify mergify Bot added the kv-connector label Sep 5, 2026
@khluu

khluu commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #87388 for commit 514ae2a11147.

@AndreasKaratzas

Copy link
Copy Markdown
Member

@khluu Can we merge this? It also resolves a failure in AMD CI.

@yewentao256 yewentao256 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the work!

@khluu
khluu marked this pull request as ready for review September 6, 2026 18:22

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@khluu
khluu merged commit 722d169 into vllm-project:main Sep 6, 2026
31 checks passed
ItsRoy69 pushed a commit to ItsRoy69/vllm that referenced this pull request Sep 10, 2026
…-project#55457)

Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Jyotirmoy Roy <jyotirmoyroy649@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants