[andyluo7]

Current combined AMD evidence - September 30, 2026, 22:53 UTC

The integration at c30e63d04dc9a54c5c4b43235eb2ce535fe68df3 now has two complete AMD passes at the identical commit. Each build has 65/65 jobs finished, every exit 0 and non-soft, with no retries. Both include all five NIGHTLY leaves.

Build	READY	MERGE	Full result
AMD #13061	28/28, 34m01.983s	30/30, 34m26.568s	65/65 passed
AMD #13064	28/28, 33m58.742s	30/30, 35m09.532s	65/65 passed

Phase time runs from the earliest phase job start to the latest finish; queue and image-build time are excluded. Both READY and MERGE are below 40 minutes in both observations.

Repeat Accuracy evaluates all 1,088 Seed-TTS samples, mean WER 0.3088705089, with zero request, PCM or ASR failures. Daily-Omni evaluates all 1,197 samples, accuracy 0.7218045113, with zero HTTP or parse failures. Both result JSON files are downloaded and SHA-256 verified against Buildkite metadata. Full CosyVoice passes all 3 cases in 2,317.40s, including both reference voice-clone cases.

These results exercise the combined c30 tree. They establish combined implementation and interaction coverage; they do not establish a second standalone run at a different source PR SHA. Earlier 024 results remain separate evidence.

CUDA #16446 at c30 still has a Documentation Whisper-small transcript failure against the unchanged 0.8 gate; WAV/JSON retention is qualified, and the exact-model replay remains pending. Intel #9379 failed during a dependency download before runtime testing. Source PR checks and maintainer review remain separate merge gates.

Source head: 4c32be21828b3ddcc73361beafadea8f960e7d44. 1 changed file(s) are identical between this source head and c30. The standalone source checks are classified in the retained validation history.

Retained validation history

Corrected combined-stack measurement

AMD #13051 Stable Audio at integration 6bcc3b8551f20ba14b061308fd063d2f8736dc58: exit 0, non-soft; 1 passed in 1506.61s. The test measured the same worker peaks, 9971.3 MB baseline versus 8030.6 MB with offload. This is another combined-tree observation; the standalone source-head result follows. It does not qualify unrelated full-suite failures.

Fresh current-head AMD result

AMD #13044 Stable Audio on current head 4c32be21828b3ddcc73361beafadea8f960e7d44: exit 0, non-soft, 1 passed in 1655.14s (27m35s). Worker allocator peak: 9971.3 MB without offload versus 8030.6 MB with layerwise offload, saving 1940.7 MB. This reproduces the intended worker-local measurement after rebase. The aggregate has independent AuK/Breeze failures plus an unresolved Engine startup failure; no full-build qualification is claimed. A second same-head observation and human approval remain required.

Summary

• reset and read peak allocator memory inside each diffusion worker through the existing test worker extension
• compare process-local allocator peaks instead of device-global used memory
• add coverage for nested collective-RPC memory probe responses

Root cause

The Stable Audio layerwise-offload test sampled total - free for the entire GPU from the parent test process. On shared ROCm workers that includes allocations from unrelated sibling workloads, so two otherwise isolated inference subprocesses can report contradictory totals. Build 12825 measured 11,070 MB incremental peak without offload and 12,964 MB with offload, producing an impossible -1,894 MB saving even though the offloader state probe confirmed the feature was enabled.

Failure: https://buildkite.com/vllm/vllm-omni-amd-ci/builds/12825/table?jid=01a0dc12-13d9-4a57-9160-28138a52c931&tab=output

The replacement resets and reads PyTorch allocator peaks in the actual diffusion worker after model construction. Those statistics are process-local, so unrelated GPU users no longer contaminate the comparison.

Validation

• python3 -m py_compile tests/diffusion/offloader/test_diffusion_layerwise_offload.py
• git diff --check origin/main...HEAD
• exact-head AMD merge-suite runtime validation requested after opening

Signed-off-by: andyluo7 andy.luo@amd.com

NPU check classification - September 30

Exact source head 4c32be21828b3ddcc73361beafadea8f960e7d44: NPU #8137 has 20 jobs finished with exit 0, non-soft. The sole failing A5 layer job never started: started_at, agent assignment and agent finish are null. Buildkite reports that its Kubernetes pod remained Pending for three hours, with PodScheduled=False, insufficient NPU/CPU capacity, unreachable-node taints and unschedulable nodes. This explains the aggregate red check as a runner scheduling failure. A5 runtime remains unqualified; no passing test is inferred for that job.

[andyluo7]

Self-review: the probe runs inside the diffusion worker, resets the accelerator allocator peak after model construction, synchronizes before both reset and snapshot, and filters nested collective-RPC responses by probe and operation. The existing offload-state and audio-equivalence checks remain intact. This removes only device-global cross-process contamination. py_compile and git diff --check pass; I am requesting the AMD merge suite because that is where the quarantined Stable Audio case runs.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/diffusion/index.md, docs/design/module/diffusion/offloader.md.

Module owners: @david6666666 @Isotr0py @princepride

Routing: @david6666666 via module named in the PR description, semantic router; @Isotr0py via module named in the PR description; @princepride via module named in the PR description

@andyluo7, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on 22432f114b67 refers to an earlier head; the pull request now points at 444ea5982e67.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on 444ea5982e67 refers to an earlier head; the pull request now points at 4c32be21828b.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Direct under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

The layerwise-offload test no longer samples device-global GPU used memory from the parent process. After each isolated OmniRunner finishes construction, it collective-RPCs into the diffusion worker, resets that process's PyTorch allocator peak, runs generate, and reads max_memory_allocated. Sibling jobs on a shared ROCm GPU therefore cannot inflate or invert the savings comparison. Offload-state and Stable Audio equivalence checks stay in place; a CPU test covers nested collective-RPC filtering.

Change flow

flowchart TD
  A["[EXISTING] Isolated OmniRunner generate"]:::existing
  B["[NEW] Worker allocator reset and snapshot probes"]:::new
  C["[NEW] Nested collective-RPC memory filter"]:::new
  D["[CHANGED] run_inference peak collection"]:::changed
  E["[CHANGED] Savings assertion on peak_allocated_mb"]:::changed
  A --> B
  B --> C
  C --> D
  D --> E
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at 444ea5982e67 (2026-09-27T20:39:54.567048+00:00): required check(s) blocking: buildkite/vllm-omni (failed), and -5 more. Observed Buildkite: buildkite/vllm-omni-amd-ci (failed), buildkite/vllm-omni-npu-ci (passed), buildkite/vllm-omni (failed).

No actionable findings.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

This PR changes only the Stable Audio / layerwise-offload memory test so peak usage is measured inside each diffusion worker instead of from the parent process. After model construction, a test-only worker extension resets and snapshots process-local PyTorch allocator peaks over collective RPC, then the savings assert compares those peaks. That removes false negatives on shared ROCm GPUs where sibling jobs inflate device-global mem_get_info totals, while keeping the existing offload-state and audio-equivalence checks.

Change flow

flowchart TD
  A["[EXISTING] Shared GPU sibling workloads"]:::existing --> B["[REMOVED] Parent DeviceMemoryMonitor / mem_get_info peak"]:::removed
  C["[CHANGED] run_inference after OmniRunner init"]:::changed --> D["[NEW] OffloadStateProbe reset/snapshot RPC"]:::new
  D --> E["[CHANGED] Assert savings via peak_allocated_mb"]:::changed
  B -.-> E
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at 4c32be21828b (2026-09-30T22:40:55.084729+00:00): all required and operator-watched checks reported green. Observed Buildkite: buildkite/vllm-omni-npu-ci (failed), buildkite/vllm-omni-intel-ci (passed), buildkite/vllm-omni-amd-ci (failed), and 1 more.

No actionable findings.

---

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

[hsliuustc0106]

No code blockers found for the current single-GPU configurations at 4c32be21828b. All 8 CPU helper tests passed, and the reset/snapshot RPCs execute in the diffusion worker on its initialized accelerator. The full model memory A/B was not rerun locally, so the reported savings remain CI evidence; standalone AMD/NPU aggregates are still red. The generic multiprocess RPC returns rank 0's payload, so this metric would need additional work before claiming an all-rank maximum.