Conversation
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
aoshen02
marked this pull request as ready for review
September 16, 2026 01:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review-only alternative: CPU execution-side assembly
Alternative to #13, not an additional upstream fix. Both target
r3_offload;do not merge the alternatives together. Duplicate-work checks found upstream
vllm-project#45635 and the existing fork #13; this alternative deliberately offers
a different lifecycle tradeoff for review, not a competing upstream PR.
Finish R3 processing inside
sample_tokens()before returning. The next workerRPC cannot clean request state until this completes. Keep the existing async
GPU-to-CPU copy, wait for it after draft proposal, then assemble on CPU. Remove
the pending-output event/wait handshake; output consumption no longer mutates
the connector. No new output type or no-forward wrapper.
Tradeoff: CPU assembly and its copy wait enter the execution critical path.
Background storage remains unchanged.
The runner calls connector-local
worker.finish_aux_output(); completionimplementation is no longer added to the public
AsyncOutputclass. Replace theretained step metadata object with a presence flag and reuse the computed end
position.
Validation
/home/aoshen/vllm/.venv/bin/python -m pytest tests/distributed/aux_output_connector -q: 58 passed (16.83s). Includes completion-order coverage.VLLM_GPU_SYNC_CHECK=error: 73 store/config tests and both HTTP R3 tests pass (Slurm 29066).AI assistance (Codex and Claude Code) was used. For human line-by-line
review and hardware validation; not ready to merge.