Skip to content

[Bugfix][SM70] Enable Qwen3.8 AWQ in dual-compile lane - #487

Merged
yangzhuxinyzx merged 2 commits into
1CatAI:mainfrom
Leonccaa:fix/qwen38-awq-dual-compile-admission
Sep 5, 2026
Merged

yangzhuxinyzx merged 2 commits into
1CatAI:mainfrom
Leonccaa:fix/qwen38-awq-dual-compile-admission

Conversation

@Leonccaa

@Leonccaa Leonccaa commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Purpose

Follow up on #477 by admitting the Qwen3.8 AWQ text-only runtime to the
existing SM70 dual-compile lane.

The checkpoint keeps the outer Qwen4ExpForConditionalGeneration
architecture even when it is loaded with --language-model-only, while the
#477 contract admitted only Qwen4ExpForCausalLM. This change also accepts the
conditional-generation wrapper, but only when
multimodal_config.language_model_only is explicitly true. Full multimodal
loading and a missing multimodal config remain rejected.

Real TP4 validation then exposed a second late-binding issue. #477 enables
hybrid PLE from VllmConfig.__post_init__, after ParallelConfig validation
would normally allocate the shared PLE IPC endpoint. As a result, all workers
attempted to connect to an empty address and failed with
zmq.error.ZMQError: Invalid argument (addr='').

This PR makes PLE endpoint initialization idempotent and invokes it when the
model-aware defaults enable PLE late. Explicit PLE configuration retains the
same behavior.

There are no AWQ kernel, quantization-math, weight-format, or scheduler
changes. Operator capability checks and existing fallbacks remain in place.
In particular, this PR does not address mixed-phase latency from chunked
prefill scheduling.

Test Plan

Test Result

  • .venv/bin/pytest -q --confcutdir=tests/compile tests/compile/test_sm70_decode_graph.py:
    7 passed.
  • Changed-file pre-commit suite: all applicable hooks passed, including Ruff,
    formatting, mypy, SPDX, configuration validation, and forbidden-import
    checks.
  • Before the IPC fix, the real model completed weight loading but all four GPU
    workers reproduced the empty-address ZMQ failure.
  • After the fix, both the post-[Core][SM70] Split Qwen3.8 prefill and decode compilation #477 main-only arm and the four-PR integration
    arm completed startup. All four workers used the same non-empty IPC endpoint,
    PLE registered 4/4 workers, and the service built the large prefill graph plus
    decode capture sizes [1, 8].
  • Each arm completed all 13 requests in the three-cell contract. Every request
    returned 320 tokens with finish_reason=length; no traceback, OOM, or
    runtime worker exit was observed.
  • The four-PR integration arm retained its memory benefit: model loading
    21.24 -> 20.24 GiB/rank and KV capacity 386,392 -> 505,574 tokens
    (+30.85%) versus the matched post-[Core][SM70] Split Qwen3.8 prefill and decode compilation #477 main-only arm.
  • The same run still measured C4 x 64K mixed-phase mean ITL at 2.841 s on
    main-only and 2.719 s on the four-PR stack. That scheduler behavior remains
    separate follow-up work.

AI assistance disclosure: OpenAI Codex assisted with implementation, test
execution, runtime evidence collation, and PR preparation. The submitter
reviewed the changes and validation results.


Essential Elements of an Effective PR Description Checklist

Leonccaa and others added 2 commits September 3, 2026 20:07
Allow the text-only Qwen4Exp conditional-generation wrapper to use the existing shared-weight prefill/decode compilation contract while keeping full multimodal models rejected.

Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
@yangzhuxinyzx

Copy link
Copy Markdown
Contributor

已基于包含 #492 的最新 main ba7f6ff 审计和模拟合并。确认此次改动只扩展已有双编译的 LM-only wrapper 准入并补齐 PLE IPC 初始化,不改变 AWQ 运算或采样。7 项定向测试通过,修改文件全部 pre-commit 通过;自动启用及重复初始化共享同一个非空 IPC 地址。结合 PR 的 TP4 启动证据,审计通过。

@yangzhuxinyzx
yangzhuxinyzx merged commit b1a19a9 into 1CatAI:main Sep 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants