[WIP][4a/5][core]refactor communication layer: PR 4a of 5 Qwen3 Omni in async mode #4146
Open
natureofnature wants to merge 11 commits into
Open
[WIP][4a/5][core]refactor communication layer: PR 4a of 5 Qwen3 Omni in async mode #4146natureofnature wants to merge 11 commits into
natureofnature wants to merge 11 commits into
Conversation
1 task
natureofnature
force-pushed
the
pr/refactor/pr4
branch
from
58d906a to
3d97797
Compare
Introduce the coordinator-side selection, carried registration field, chunk-ready/finished consumption, late-ready retention, and scheduler completion guards needed before enabling async-chunk stages. Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Add runner-side async-chunk sends, duplicate-send guards, finish sentinel enqueue/consume paths, code2wav terminal flushing, deep metadata merge semantics, and quiet per-chunk transport logging. Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Flip qwen3_omni async-chunk stages onto the coordinator+mixin transport, preserve audio finish behavior, and keep allowlist-active branches safe for partial unit-test mocks. Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Carry segment-finished state across the legacy thinker to coordinator talker/code2wav boundary with a distinct connector signal so realtime async-chunk requests flush and finish correctly. Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Keep prefill and decode rows separate across the streaming handoff, then consume cached decode rows as the talker decode prefix before current decode rows. Signed-off-by: natureofnature <wzliu@connect.hku.hk>
natureofnature
force-pushed
the
pr/refactor/pr4
branch
from
e7bb493 to
4d46b5a
Compare
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
natureofnature
requested review from
Gaohan123,
ZeldaHuang,
gcanlin,
hsliuustc0106,
linyueqian,
princepride,
tzhouam,
yenuo26,
yuanheng-zhao and
ywang96
as code owners
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
hsliuustc0106
left a comment
hsliuustc0106
left a comment
Collaborator
There was a problem hiding this comment.
Posted one inline blocker found during review.
| If the request finished on a chunk boundary (no unsent tail), emit a | ||
| finish-only flag instead of re-sending the last full chunk. | ||
| """ | ||
| chunk_size_config, left_context_size_config = _code2wav_codec_config(transfer_manager) |
Collaborator
There was a problem hiding this comment.
This finish-sentinel path does not mirror the live send path when initial_codec_chunk_frames is configured. The bundled Qwen3 Omni configs set initial_codec_chunk_frames: 4, so a request that ends exactly at 4 frames, or at 4 + N * codec_chunk_frames, has already emitted the full boundary chunk in the normal path. Here length % codec_chunk_frames is non-zero, so the sentinel re-sends those frames as a tail, duplicating audio at the end. Can we include the same initial-chunk adjustment used below before deciding whether there is an unsent tail?
hsliuustc0106
added
ready
Collaborator
|
is there any benefit of this PR in terms of perf? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.
Purpose
For the Refactor of commnication layer, there are going to be 5~6 PRs in total.
4a: Qwen3 Omni only
4b: Other relevant models.
Refer to PR #1555 as the first PR.
PR:#2677 as the second PR.
PR:#3719 as the third PR.
Test Plan
test-ready, test-merge, test-nightly
Test Result
To be added.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model. Please runmkdocs serveto sync the documentation editions to./docs.BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)