Compatibility with inference other than vllm < 0.10.2 - #700
Closed
tdene wants to merge 1 commit into
Closed
Conversation
tdene
marked this pull request as draft
February 15, 2026 21:57
tdene
force-pushed
the
tde/new_vllm_compat
branch
from
February 15, 2026 23:01
9d30e72 to
94afba2
Compare
tdene
marked this pull request as ready for review
February 15, 2026 23:01
tdene
force-pushed
the
tde/new_vllm_compat
branch
from
February 15, 2026 23:08
94afba2 to
111ef6b
Compare
tdene
force-pushed
the
tde/new_vllm_compat
branch
from
March 31, 2026 19:48
08a7db2 to
6e604e2
Compare
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
tdene
force-pushed
the
tde/new_vllm_compat
branch
from
March 31, 2026 19:49
6e604e2 to
af3ebae
Compare
Merged
tdene
added a commit
to tdene/Gym
that referenced
this pull request
Apr 27, 2026
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
tdene
added a commit
to tdene/Gym
that referenced
this pull request
Apr 27, 2026
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
bxyu-nvidia
pushed a commit
that referenced
this pull request
Apr 27, 2026
github-actions
Bot
added
the
sla:review-overdue
ananthsub
added a commit
that referenced
this pull request
Aug 18, 2026
#1558) ## Summary - Consume a complete token bundle from `choice.message` without making a second `/tokenize` request. - Support opt-in vLLM response token IDs for endpoints that return top-level `prompt_token_ids` and choice-level `token_ids`. - Retain `/tokenize` for integrations that return neither inline representation. - Reject partial or conflicting sources. Token capture supports one completion choice per request (`n` omitted or set to `1`). ## Why Gym needs the prompt token IDs, generated token IDs, and selected-token log probabilities used during generation. The previous adapter reconstructed prompt IDs through a second awaited `/tokenize` request. That request adds inter-turn latency and can reproduce a different prompt when prompt-affecting inputs are omitted. NeMo RL [#3390](NVIDIA-NeMo/RL#3390) attaches the token bundle to the assistant message. Gym reads that bundle directly and does not request duplicate response-level IDs by default. Gym cannot require that message shape from every training integration. Other integrations, including Verl, can request vLLM's response token IDs or continue using the compatibility fallback. ## Source precedence 1. A complete token bundle on `choice.message`. 2. Top-level `prompt_token_ids`, choice-level `token_ids`, and `choice.logprobs` from the vLLM response. 3. Generation metadata from `choice.logprobs` and prompt IDs from `/tokenize`. `request_prompt_and_generation_token_ids: true` enables the second source by sending `return_token_ids=true` to compatible vLLM endpoints. The option defaults to `false`. If both inline sources are present, their token IDs must agree. The selected training message is validated once to avoid repeated list scans and copies at long sequence lengths. The fallback forwards `model`, `messages`, `tools`, `chat_template_kwargs`, `mm_processor_kwargs`, and `required_prefix_token_ids`. This keeps fallback tokenization aligned with the generation prompt. `prompt_logprobs` is not requested. vLLM returns prompt IDs independently when `return_token_ids` is enabled, and Gym does not consume prompt-token log probabilities. ## Related work - Gym [#700](#700) is superseded by this change. - Gym [#2576](#2576) is superseded by this change. - NeMo RL [#3390](NVIDIA-NeMo/RL#3390) produces the message-level token bundle consumed by the first source. - NeMo RL [#3581](NVIDIA-NeMo/RL#3581) reuses replayed token metadata across turns. - Gym [#1784](#1784) can add the Dynamo `nvext.engine_data` source independently. - Gym [#2324](#2324) can retain its video-specific schema, conversion, and adapter behavior independently. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inside
vllm_model/app.py, there is a comment bloc that says:The associated code always assumes that the user is using vLLM < 0.10.2. This removes that assumption, allowing NeMo Gym to work easily with vLLM >= 0.10.2 or any other inference framework.