Repository navigation
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
valar-cop
left a comment
valar-cop
left a comment
There was a problem hiding this comment.
some changes needed..
e2e text-only run at 5-16 concurrent requests?
ci green.
The mono target's QSA programs rotated q / k and the index keys at the cache row (seq_lens[b] + s) instead of the position ATOM hands the model. For text the two are equal; for a request with an image, ATOM's decode positions carry the request's MRoPE offset (backends.py: arange(start + delta, stop + delta)), and verify_step only refuses a step that holds image tokens, so the decode steps after an image ran on mono with every new token rotated off by the offset. Measured with an eager diagnostic on synthetic 896-1280 px images: offsets of -736 / -756 / -1160 on the mono-served verify steps, 0 for text. The verify prep (one launch, graph-capturable) now also writes rope_off[b] = positions[b * spec] - (context_lens[b] - spec), 0 for an invalid request; QKNormRope and the indexer rotate at the cache row plus rope_off[b] (the query rows, the span tasks' query vectors and the keys of the blocks a step completes), while pages and blocks are still addressed by cache row. rope_pos clamps at 0 as well. The draft program reads the same argument, left at 0: its rows keep the drafter's cache-row positions as before (an image request's draft only affects MTP acceptance, not the output); matching the drafter's MRoPE positions is a follow-up. Text is unchanged (rope_off 0). The 8 GDN programs build identical ISA; the 12 QSA / MTP programs keep their VGPR / SGPR / LDS, no scratch. On MI355X: 145 GPU tests pass; full 1319 GSM8K with real MTP 1289 / 1319 (acceptance 0.726); fixed load against the parent, before / after, request TPOT C1 64K 1.904 / 1.900 / 1.898 ms, C2 16K 2.428 / 2.446 / 2.428 ms (+0.9%: the extra offset load in the QSA programs). Images at client concurrency 1: OCR 47 / 48 (Native 46 / 48). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yifehuan
changed the title
… path on CPU
- step_meta.draft_attention_fused (CPU-safe) decides whether the fallback
draft's QSA attention takes its fused path; draft_fallback.attention calls
it, and the assert the text-only guard had made dead is gone.
- tests/test_qwen38_mono.py replaces test_qwen38_mono_{plan,prep,ple}.py. The
GPU libraries are imported inside the tests that need them, so the CPU half
runs in CI, now including the attention path at 1 and 6 requests, q1 and
q4, with and without the three-axis position cache. The GPU half (verify
prep, draft metadata, the PLE host table) skips without a GPU or Triton.
The metadata puts test also covers rope_off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
e2e text-only run, as asked in the review: The PR also has a second fix now, 8106f2c (QSA RoPE at ATOM's MRoPE positions for image requests). The description has the evidence. |
|
@valar-cop re-review Head 2460991; CI is green. Since your review on e6a0a63:
|
|
@valar-cop re-review |
|
ci triage for 2460991:
|
Text-only Qwen3.8-Flash-Next deployments leave
rope_position_cacheunset. When MTP leaves the <=4-request draft Mono program, the fused QSA fallback introduced in #2610 asserts that this optional cache exists and terminates the model worker. We reproduced this after a six-request prefill, while the HTTP process continued to respond.Dispatch to ATOM's existing native QSA implementation before fused preparation when the position cache is absent. That implementation reconstructs text positions arithmetically; the fused fallback remains eligible when the three-axis cache is present. The decision is
step_meta.draft_attention_fused, which imports no GPU library, so CI tests it on CPU: q1 / q4 at 1 and 6 requests, with and without the position cache, plus prefill, no QSA metadata and another row count. The Qwen mono tests are now one file,tests/test_qwen38_mono.py(it replacestest_qwen38_mono_{plan,prep,ple}.py). Its CPU half runs in CI; its GPU half skips without a GPU or Triton.Validation:
tests/test_qwen38_mono.py: 35 tests pass on CPU. On MI355X, 141 pass together withtest_mono_runtime.py, none skipped.--hf-overrides {"_multimodal_config": null}), real MTP, Zhang Ling's A_lo server, head 2460991: 300 GSM8K questions at client concurrency 6 scored 294 / 300, and at 16 scored 298 / 300. Acceptance was 0.724 on both, with 0 tracebacks. Mono served the 1-4 request steps, and the fallback draft served the rest.git diff --checkchecks recorded before submission.This follow-up uses the original
yifehuan/qwen38_flash_monobranch name, restored from current main after the merged branch was deleted.Second fix (8106f2c): rotate mono QSA rows at ATOM's MRoPE positions
#2610's mono QSA programs rotated q / k and the index keys at the cache row (
seq_lens[b] + s) instead of the position ATOM hands the model. For text the two are equal. For a request with an image, ATOM's decode positions carry the request's MRoPE offset (backends.py:arange(start + delta, stop + delta)), andverify_steponly refuses a step that holds image tokens, so the decode steps after an image ran on mono with every new token rotated off by that offset. An eager diagnostic on synthetic 896-1280 px images measured offsets of -736 / -756 / -1160 on the mono-served verify steps (0 for text).The verify prep (the same graph-capturable launch) now also writes
rope_off[b] = positions[b * spec] - (context_lens[b] - spec)(0 for an invalid request).QKNormRopeand the indexer rotate at the cache row plusrope_off[b], while pages and blocks are still addressed by cache row. The draft program reads the same argument, left at 0, so it keeps its previous positions; an image request's draft only affects MTP acceptance, and matching the drafter's MRoPE positions is a follow-up.Validation (MI355X, full 48-layer checkpoint, TP1, MTP3, Zhang Ling's A_lo server):
rope_offis 0.Separately, Native itself mixes images across concurrent requests: at client concurrency 4, Native's OCR drops from 46 / 48 to 13-16 / 48, and wrong answers carry other requests' numbers. This is independent of mono and not addressed here.