Skip to content

Allow callers to pass pre-allocated moe_buf to avoid output copy#2663

Closed
tpopp wants to merge 1 commit intoROCm:mainfrom
tpopp:moe-buf-passthrough
Closed

Allow callers to pass pre-allocated moe_buf to avoid output copy#2663
tpopp wants to merge 1 commit intoROCm:mainfrom
tpopp:moe-buf-passthrough

Conversation

@tpopp
Copy link
Copy Markdown
Contributor

@tpopp tpopp commented Apr 9, 2026

Summary

  • Add an optional moe_buf parameter to _moe_sorting_impl, moe_sorting, fused_moe, fused_moe_fake, and fused_moe_ functions
  • When moe_buf is provided, the sorting kernel writes directly into the caller's buffer instead of allocating a new tensor, eliminating a redundant device-to-device copy on the output path

Test plan

  • Verify existing MoE tests pass (no moe_buf → existing allocation behavior unchanged)
  • Verify with moe_buf provided that the output is written in-place
  • Benchmark to confirm reduced memory allocation overhead

Made with Cursor

Add an optional `moe_buf` parameter through the moe_sorting and
fused_moe call chain. When provided, the sorting kernel writes
directly into the caller's buffer instead of allocating a new one,
eliminating a redundant copy on the output path.

Made-with: Cursor
@tpopp tpopp requested a review from a team April 9, 2026 07:15
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 9, 2026

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-355 Run Triton tests on MI355 in addition to MI325
ci:sglang SGLang integration tests
ci:atom ATOM benchmark (DeepSeek-R1 + GPT-OSS)
ci:vllm vLLM benchmark
ci:all All of the above

Add labels via the sidebar or gh pr edit 2663 --add-label <label>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant