Skip to content

[ROCm][CI] Fix spec decode profile assertion and logprob test determinism #35043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vllm-bot merged 4 commits into from
Feb 23, 2026
Merged

[ROCm][CI] Fix spec decode profile assertion and logprob test determinism #35043

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d52d0c3
Fix _dummy_run assertion to use max_num_tokens instead of max_num_bat…
AndreasKaratzas Feb 22, 2026
afc399f
[ROCm][CI] Introduce deterministic engine kwargs for spec decode logp…
AndreasKaratzas Feb 22, 2026
d40e632
Merge remote-tracking branch 'origin/main' into akaratza_fix_v1_others
AndreasKaratzas Feb 22, 2026
26656e1
Reverted gpu model runner modification in anticipatio of follow-up PR
AndreasKaratzas Feb 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions tests/v1/sample/test_logprobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
from vllm import SamplingParams
from vllm.config.model import LogprobsMode
from vllm.distributed import cleanup_dist_env_and_memory
from vllm.platforms import current_platform

from ...conftest import HfRunner, VllmRunner

Expand All @@ -31,6 +32,23 @@
PROMPT = BatchLogprobsComposition.PROMPT
SAMPLE_PROMPT = BatchLogprobsComposition.SAMPLE_PROMPT

# On ROCm, floating-point reductions in attention and GEMM kernels are
# non-associative and sensitive to batch geometry. The ref LLM (no spec
# decode, default scheduling) and the spec-decode LLM (chunked prefill,
# different effective batch sizes) follow different reduction orders,
# producing numerically divergent logprobs that get mis-attributed to
# spec-decode incorrectness.
#
# Force LLM instances into an identical, deterministic execution
# mode so the test isolates spec-decode correctness only:
ROCM_DETERMINISM_KWARGS: dict = (
dict(
max_num_seqs=1,
)
if current_platform.is_rocm()
else {}
)
Comment on lines +44 to +50

@gemini-code-assist gemini-code-assist Bot Feb 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The ROCM_DETERMINISM_KWARGS dictionary currently only sets max_num_seqs=1. The PR description mentions enforce_eager and async_scheduling=False as part of the determinism kwargs. These should also be included in the dictionary to fully align with the described fix and ensure consistent execution paths on ROCm.

Suggested change
ROCM_DETERMINISM_KWARGS: dict = (
dict(
max_num_seqs=1,
)
if current_platform.is_rocm()
else {}
)
ROCM_DETERMINISM_KWARGS: dict = (
dict(
enforce_eager=True,
async_scheduling=False,
max_num_seqs=1,
)
if current_platform.is_rocm()
else {}
)

@AndreasKaratzas AndreasKaratzas Feb 22, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the description already, apparently those args were unnecessary.



@pytest.fixture(
scope="module",
Expand Down Expand Up @@ -1035,6 +1053,7 @@ def test_spec_decode_logprobs(
logprobs_mode=logprobs_mode,
gpu_memory_utilization=0.4,
enable_prefix_caching=False,
**ROCM_DETERMINISM_KWARGS,
)
ref_results = ref_llm.generate(
[prompt, prompt], [sampling_params, penalty_sampling_params]
Expand Down Expand Up @@ -1064,6 +1083,7 @@ def test_spec_decode_logprobs(
enable_chunked_prefill=True,
max_num_batched_tokens=32,
enable_prefix_caching=False,
**ROCM_DETERMINISM_KWARGS,
)
spec_results = spec_llm.generate(
[prompt, prompt], [sampling_params, penalty_sampling_params]
Expand Down
4 changes: 2 additions & 2 deletions vllm/v1/worker/gpu_model_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -378,7 +378,7 @@
self.calculate_kv_scales = self.cache_config.calculate_kv_scales
self.dcp_world_size = self.parallel_config.decode_context_parallel_size
self.dcp_rank = 0 if self.dcp_world_size <= 1 else get_dcp_group().rank_in_group
self.max_num_tokens = scheduler_config.max_num_batched_tokens
self.max_num_tokens = vllm_config.max_num_tokens_per_forward_pass

Check failure on line 381 in vllm/v1/worker/gpu_model_runner.py

View workflow job for this annotation

GitHub Actions / pre-commit 

"VllmConfig" has no attribute "max_num_tokens_per_forward_pass"  [attr-defined]

Check failure on line 381 in vllm/v1/worker/gpu_model_runner.py

View workflow job for this annotation

GitHub Actions / pre-commit 

"VllmConfig" has no attribute "max_num_tokens_per_forward_pass"  [attr-defined]

Check failure on line 381 in vllm/v1/worker/gpu_model_runner.py

View workflow job for this annotation

GitHub Actions / pre-commit 

"VllmConfig" has no attribute "max_num_tokens_per_forward_pass"  [attr-defined]

Check failure on line 381 in vllm/v1/worker/gpu_model_runner.py

View workflow job for this annotation

GitHub Actions / pre-commit 

"VllmConfig" has no attribute "max_num_tokens_per_forward_pass"  [attr-defined]
self.max_num_reqs = scheduler_config.max_num_seqs

# Broadcast PP output for external_launcher (torchrun)
Expand Down Expand Up @@ -4678,7 +4678,7 @@
# Set num_scheduled_tokens based on num_tokens and max_num_seqs
# for dummy run with LoRA so that the num_reqs collectively
# has num_tokens in total.
assert num_tokens <= self.scheduler_config.max_num_batched_tokens
assert num_tokens <= self.max_num_tokens
max_num_reqs = self.scheduler_config.max_num_seqs
if create_mixed_batch:
assert not uniform_decode
Expand Down
Loading