Repository navigation
bench: greedy Qwen3.5 cells and ITL in results.json - #909
Merged
Merged
Conversation
…esults.json qwen3.5 sent no sampling params, so mstar sampled at its 0.6 default. Also pass enable_thinking through chat_template_kwargs so vllm/sglang render the 0.8b/2b prompts the way mstar does. results.json now carries itl, tok/s and token counts (the console table rounds to 1 ms).
Automated review🟡 No findings on Description
What the review checked
Advisory, not a merge gate. Rules: |
Gaurav-Shah05
approved these changes
Oct 27, 2026
| # Greedy everywhere (mstar's sampler defaults to 0.6), and the same prompt | ||
| # render: mstar always opens a <think> block, while the 0.8B/2B templates | ||
| # only do when asked, so vLLM/SGLang get it through chat_template_kwargs. | ||
| return {"temperature": 0.0, "chat_template_kwargs": {"enable_thinking": True}} |
Collaborator
There was a problem hiding this comment.
M*'s Qwen3.5 reads a flat enable_thinking (default True) and ignores chat_template_kwargs, so its think block comes from that default, not from this.
Sending enable_thinking: True flat as well would keep the three renders the same if the default changes.
vasilevklart
approved these changes
Oct 27, 2026
mstar's Qwen3.5 reads a flat enable_thinking and ignores chat_template_kwargs, which only the Cosmos3-Edge adapters map, so flipping the chat_template_kwargs value alone would change vLLM and SGLang but not mstar. vLLM and SGLang ignore the flat key.
dghosef
added a commit
that referenced
this pull request
Oct 28, 2026
Main brought (since 6cfe39c): #892 Parakeet and Nemotron ASR with the transducer as a graph loop (and the shared mrope_position_ids helper), #917 pytest 9.2 flag proxies, #897 one captured all-to-all for Ulysses, #915 NCCL graph registration off by default, #836 Qwen3-Omni Thinker prefix caching (ARNodeSubmodule.cuts_tensor_inputs, marks_dynamic_dims, the conductor's _streamed_partitions), #909 bench cells, #900 Command A+ batched image tiles. Conflict: - mstar/model/submodule_base.py: main's ARNodeSubmodule.cuts_tensor_inputs beside the stack's token_stop and its stop checks. Independent class attributes; both kept. Ported onto the stack's mechanisms: - mstar/model/parakeet/submodules.py: check_stop_batched drops the step_walk parameter (the stack's three-argument signature) and reads its "active" flags through HostRows.first_values instead of its own row-map loop. Its stop is a flag, not a token, so it has no TokenStop. - test/modular/test_transducer_loop.py: the call without step_walk. The conductor's _streamed_partitions (#836) sits beside _pick_replicas / _register_request; it reads only partition_states and _resource_specs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dghosef
added a commit
that referenced
this pull request
Oct 28, 2026
Brings #858's merge of main: #892 Parakeet / Nemotron ASR, #917, #897 Ulysses all-to-all, #915, #836 Qwen3-Omni Thinker prefix caching, #909, #900. Conflicts: - docs/adding_models.rst: main's marks_dynamic_dims sentence and #859's compile_regions paragraph are independent facts; both kept, in one paragraph. - mstar/model/qwen3_omni/submodules.py (3 hunks, plus auto-merged hunks in the same class): AudioEncoderSubmodule. Two mechanisms for one cause (the HF encoder recompiling per clip length): #859 runs it eagerly (disable_torch_compile) and packs a step's clips into calls of at most MAX_BATCH_FRAMES; #836 builds the encoder's chunk tensors in prepare_inputs with mark_dynamic so the per-clip forward compiles once (marks_dynamic_dims, _max_seqlen). One mechanism: #859's class verbatim. Under disable_torch_compile the engine never compiles the encoder, so #836's concern (a recompiled graph rounding differently from the one whose output the prefix cache holds) cannot arise. The auto-merge had taken #836's kwargs-only forward beside #859's forward_batched, which calls forward with the clips positionally; that keep-both is undone. Main's VisionEncoderSubmodule (marks_dynamic_dims, prepared indices) and Thinker changes (cuts_tensor_inputs, split_inputs, mrope_position_ids, image_span) are kept; #859 did not change those. Imports: both cp_embed_accumulate and mrope_position_ids; the audio helpers only #836's prepare_inputs used are dropped. Tests following from it (test/modular/test_qwen3_omni_prefix_cache.py): - the test that #836's prepared chunk tensors embed a clip as the encoder does alone is deleted with that code; - the compile test now asserts the audio encoder stays eager while its output is cached; - the CUDA no-recompile test keeps its vision case only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dghosef
added a commit
to dghosef/mstar
that referenced
this pull request
Oct 28, 2026
…igin/main 73c6c17) into merge/860-main Brings main's mstar-project#892 Parakeet / Nemotron ASR, mstar-project#917, mstar-project#897 Ulysses as one captured all-to-all (prefer_all_gather and the all-gather exchange gone, Cosmos3 captures every tier under TP x SP), mstar-project#915, mstar-project#836 Qwen3-Omni Thinker prefix caching, mstar-project#909 and mstar-project#900, with mstar-project#858's and mstar-project#859's resolutions. No conflicts. Files both sides touched (docs/clients.rst, docs/installation.rst, docs/models.rst, sequence_parallel.py, cosmos3/submodules.py, test_accelerator_graph_capture.py, test_flux2_klein_model.py): each merged change equals mstar-project#859's own delta since 3166401, so nothing kept both sides. mstar-project#860's my_rows / gather_rows (Wan2.2's CFG split) sit beside main's new ulysses_attention and use CommGroup.all_gather, which mstar-project#897 left unchanged; nothing in mstar-project#860 passed prefer_all_gather. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two small runner fixes from the B300 text/VLM comparison.
Qwen3_5_Dense.get_model_kwargssendstemperature: 0andchat_template_kwargs: {enable_thinking: true}. Without it mstar sampled at its 0.6 default while vLLM and SGLang ran at 1.0, and the 0.8B/2B chat templates rendered without the think block that mstar always opens. The three engines were not generating the same thing.results.jsonnow carriesitl_s,text_token_throughput,total_text_tokens,mean_text_tokensandtotal_input_tokensat full precision next tottft_s. Before, ITL and tok/s only existed in the rounded console summary, which is not enough to compare 2-3 ms ITLs across engines.Used for every Qwen3.5 0.8B/2B/4B/9B cell against vLLM 0.32.0 and SGLang 0.5.22 on B300 (closed_loop, 512 in / 256 out, ignore-eos). No engine code touched.