[Misc] Refactor aclgraph accuracy test to use logprob-based comparison

[Potabk]

What this PR does / why we need it?

Replace text-match assertions with a two-tier logprob accuracy check:

• Prefill (token 0): assert token ID is identical between eager baseline and compiled mode, then verify logprob matches within atol.
• Decode (tokens 1-2): if chosen tokens match, compare logprobs directly; if they differ, cross-lookup the baseline token in the compiled model's top-20 distribution and assert the assigned logprob is within decode_atol (defaults to 2x atol). This tolerates minor argmax drift caused by floating-point differences while still catching distribution divergence.

Does this PR introduce any user-facing change?

How was this patch tested?

• vLLM version: v0.17.0
• vLLM main: vllm-project/vllm@8a68046

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the robustness of aclgraph accuracy tests by transitioning from simple text-match assertions to a sophisticated two-tier log probability comparison. This change addresses the limitations of text-based comparisons, which can be overly sensitive to minor floating-point variations in model outputs. By comparing log probabilities with specific tolerances and implementing a cross-lookup mechanism for differing tokens, the tests now more accurately reflect the numerical consistency of the model's output distributions between eager and compiled execution modes, thereby improving the reliability of accuracy validation.

Highlights

• Refactored ACLGraph Accuracy Tests: The existing text-match assertions in aclgraph accuracy tests were replaced with a new two-tier logprob-based comparison mechanism.
• Prefill Token Validation: For the first token (prefill pass), the test now asserts that the token ID is identical between the eager baseline and compiled mode, and their log probabilities match within a specified absolute tolerance (atol).
• Decode Token Validation: For subsequent tokens (decode passes), if the chosen tokens match, their log probabilities are directly compared. If they differ, the baseline token's log probability is cross-referenced within the compiled model's top-20 distribution, asserting that the assigned log probability is within a decode_atol (defaults to 2x atol). This allows for minor argmax drift due to floating-point differences.
• Removed golden_answers: The golden_answers field was removed from LLMTestCase instances in the accuracy tests, as the new validation method no longer relies on pre-defined text outputs.
• Introduced compare_logprobs: A new utility function, compare_logprobs, was added to handle the detailed log probability comparison logic, replacing the previous gen_and_valid function.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

[gemini-code-assist]

Code Review

This pull request refactors the aclgraph accuracy test to use a logprob-based comparison instead of text-match assertions. The changes involve removing golden answers from test cases and introducing a compare_logprobs function to verify accuracy by comparing log probabilities against an eager-mode baseline. Additionally, the PR adds @wait_until_npu_memory_free(0.7) decorator to ensure sufficient NPU memory before running tests. The PR also removes the gen_and_valid function and replaces it with compare_logprobs calls.