[Doc] Add FusedMoE Modular Kernel Documentation

[varun-sundar-rabindranath]

Purpose

Documentation for FusedMoEModularKernel

Test Plan

None

Test Result

None

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request adds documentation for the FusedMoE Modular Kernel. The documentation is comprehensive and provides a good overview of the architecture. I've identified a few high-impact issues to improve clarity and correctness: there are some recurring typos in key class names, and a couple of tables are not formatted correctly in Markdown, which hinders readability. The suggested changes should address these points.

[varun-sundar-rabindranath]

@bnellnm PTAL! Thanks 🙌

[bnellnm]

I think we should elaborate on this section for a bit.

[bnellnm]

I think there should be a brief description of the contiguous vs. batched formats at the beginning. I don't think it's obvious what the batched format is unless you've been working on MoE stuff already.

[varun-sundar-rabindranath]

I think there should be a brief description of the contiguous vs. batched formats at the beginning. I don't think it's obvious what the batched format is unless you've been working on MoE stuff already.

Done 👍 Please take a look 🙌

[bnellnm]

I think it's important to note that layer.py sets self.fused_experts to the fused moe object and that the subclass needs to use this in it's own apply method.

[bnellnm]

LGTM, thanks! Just one final comment about init_prepare_finalize.

[hmellor]

Here's the rendered page preview https://vllm--21623.org.readthedocs.build/en/21623/design/fused_moe_modular_kernel.html

If you use a level 1 heading in the MD I believe the title will be capitalised properly

[varun-sundar-rabindranath]

Here's the rendered page preview https://vllm--21623.org.readthedocs.build/en/21623/design/fused_moe_modular_kernel.html

If you use a level 1 heading in the MD I believe the title will be capitalised properly

@hmellor done 👍 Please let me know if you see more issues. Thanks.

[hmellor]

Sorry, I meant if you title it

# Fused MoE Modular Kernels

[hmellor]

Here is the rendered preview: https://vllm--21623.org.readthedocs.build/en/21623/design/fused_moe_modular_kernel.html. Lots of things aren't rendering correctly

[varun-sundar-rabindranath]

Thanks @hmellor - Sorry you had to go through the entire documents. I believe things should be fixed now.

[hmellor]

Thanks for making those fixes, I've replied in the thread about the !!! note syntax