Skip to content

Qwen3-Omni: one grouping for the prefill schedule and the KV layout - #1165

Merged
merceod merged 2 commits into
mainfrom
users/atindra/omni-prefill-steps
Nov 3, 2026
Merged

merceod merged 2 commits into
mainfrom
users/atindra/omni-prefill-steps

Conversation

@merceod

@merceod merceod commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #1110, the two points from its review.

Change

The schedule and the layout read one grouping. _thinker_steps now returns the Thinker's prefill steps as ThinkerStep(walk, parts): each attachment with the text before it, the last one also with the text after it. _build_thinker_prefill_schedule builds one entry per step from it, and _kv_layout sets opens_step on the first part of each step. Before, each worked out the grouping on its own, which is how the two-image layout drop in #1110 happened. Now the layout's steps are the steps that run.

The schedule's skips go: a part with no tensors, if entry:, and a text index past the list. process_prompt runs check_attachments and check_plan, then _kv_layout indexes every part's tensors with no skip, and the data worker is the only place a request reaches the conductor from (data_worker._prepare_input always hands process_prompt a tensors dict). So a request with a part to skip fails at intake and never gets here. The schedule can't come out empty either, since prefill_plan always ends with a text part, so the empty-schedule fallbacks in _get_thinker_initial_args go too. _get_thinker_prefill_inputs already indexed schedule[0] without checking.

A layout next_step can't walk fails where it's built. next_step only extends a step over spans of its own walk, so a span with opens_step=False after a span of another walk would end the step early, and the lookup would drop the layout with only a warning. LabelLayout.row() now raises ValueError for that, next to the span-kind check, naming the span index and both walks (or saying no span comes before it, for span 0). The next_step, Span and adding_models.rst docs say so. Only Qwen3-Omni sets the mark today, and its layouts all pass.

Tests

  • ruff check .: clean.
  • test_the_layout_says_where_each_step_the_schedule_runs_ends now covers 15 prompts: every layout the parity and prefix-cache tests use (text, image, image+text, audio, video, text around each, text then image, two images, image text image, image text image text, three images, image video text, image audio text). For each one it checks the walks and span count of every step against a hand-written table, and each span's slots against what the schedule entry's row writes. I checked it against four mutations (TEXT_BEFORE forced to 0, every span its own step, text order reversed, text after an attachment joined to it); each fails 2 to 14 of the 15 cases.
  • test_a_step_that_changes_walk_fails_where_its_layout_is_built (new): text then an image marked opens_step=False, and a first span marked so, both raise; the same layout with one walk passes. It fails on main.
  • Schedule unchanged: a script dumped the schedule (walks, entry keys in order, which signal each names), the Thinker's first forward args (walk, step metadata, input edges) and both KV layout rows, on main and on this branch, for the 15 prompts with text and with audio out. All 30 match.
  • The touched modules (test_qwen3_omni_prefix_cache.py, test_kv_prefix_layout_lease.py, test_multimodal_ordering.py, test_qwen3_omni_prompt_parity.py): 155 passed, 2 skipped.
  • CPU Core (the select_cpu_tests.py paths, OMP_NUM_THREADS=2): 11342 passed, 1034 skipped, 10 failed; main at the same base: 11331 passed, 1034 skipped, 10 failed, the same 10 by id (test_send_registration_failure 5, test_eval_compare_multigpu 2, test_client_error_text, test_qwen3_omni_encoders, test_serving_metrics). The 11 more are the new cases.

GPU check

1x B300 (sm_103), torch 2.12.1+cu132, qwen3omni_colocated on GPU 0 with -O2 and prefix caching on, over /v1/chat/completions so the parts keep their order. Each prompt sent four times, greedy, on two image sets, after a warm pass on other files. TTFT in ms:

prompt set 1 set 2 layout warnings
image, image, text 119 94 85 88 112 92 89 86 0
image, "and", image, text 111 87 88 79 113 83 81 73 0
image, audio, text 110 89 84 84 166 68 85 77 0
image, video, text 407 412 394 394 401 370 378 387 0

Repeats come in 25-30 ms under the first send, as in #1110, and the video prompt stays level, since only its leading text is keyed. No "its chain ends here" line in the whole server log. All four sends gave the same reply in all 8 sets.

I2T at c=16 (Food101, 256 tokens, ignore_eos, 200 requests after 5 warmup waves, prefix caching on): 1863 tok/s, TTFT p50 202 ms, ITL p50 7.6 ms, 0 failed. That's one server on this branch, not an A/B against main. The CPU check above that the schedule and layouts match main is what says nothing moved.

The schedule and the KV layout each grouped the plan into steps on their
own; now both read _thinker_steps, so the layout's opens_step can't drift
from the steps that run. The schedule's skips go: process_prompt already
indexes every part's tensors at intake, after check_attachments and
check_plan, so nothing reaching the conductor has a part to skip.

The layout/schedule test now covers every prompt the parity and prefix
cache tests use.
next_step ends a step at a change of walk, so such a layout would only
show up later as a dropped lookup. Fail it where it's built instead.
@merceod
merceod merged commit a7fa870 into main Nov 3, 2026
6 checks passed
@merceod
merceod deleted the users/atindra/omni-prefill-steps branch November 3, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant