Support strided Liger MoE input weights

[supercharleszhu]

Summary

Support strided input weights for MOE.

Problems

Qwen-style MoE checkpoints store both input projections in one packed tensor:

w13 shape: [experts, 2 * intermediate, hidden]

gate = w13[:, :intermediate, :]
up   = w13[:, intermediate:, :]

Each gate/up matrix is contiguous internally, but adjacent experts are
2 * intermediate * hidden elements apart. The current binding rejects these
views because the overall tensor is not contiguous. Serving integrations must
therefore call .contiguous() on both views. For Qwen3-235B, those retained
copies consume approximately 0.375 GiB per layer, or 35.25 GiB per GPU across
94 MoE layers, reducing KV-cache capacity and causing OOM at 90% memory
utilization.

Relaxing only the binding check would be incorrect. The old kernel
defines num_n_tiles_1 as intermediate_dim / TileN
and then
uses expert * dims.num_n_tiles_1 as the expert's starting weight tile.
Together, these lines assume adjacent experts are separated by exactly
intermediate / TileN weight tiles:

I=1536, TileN=128
useful tiles per expert = 1536 / 128 = 12

packed physical stride = 2 * 1536 / 128 = 24 tiles

For packed gate weights, physical tiles are:

tiles  0-11: expert 0 gate
tiles 12-23: expert 0 up
tiles 24-35: expert 1 gate

Using the old expert * 12 offset makes expert 1 gate computation read expert
0 up weights. The same issue causes the up descriptor to read the next expert's
gate weights. Outputs would therefore be silently corrupted.

This PR passes the actual expert stride from the tensor metadata and uses it
only to locate each expert's first tile. Each expert still computes exactly
intermediate / TileN useful tiles. For ordinary contiguous weights, the
physical stride equals the useful tile count, so existing behavior is unchanged.

Notes

This PR supports expert-strided gate/up weights for forward no-grad inference.
The public moe_fused no-grad path calls the native forward directly and pops
its symmetric buffers immediately.

The grad-enabled path enters LigerExpertParallelFusedMoEFunction.forward,
which rejects non-contiguous gate/up tensors before resolving the NVSHMEM team
or launching native forward:

if not all_B.is_contiguous() or not all_C.is_contiguous():
    raise ValueError(...)

Testing Done

• Hardware Type: 8x NVIDIA H200
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence
• Local code review completed

Targeted validation:

• Built
  liger_cute_kernels-0.1.0+cu130.torch2.11.0-cp312-cp312-linux_x86_64.whl
  (sha256: 22a12d6f8825b377f94f8b4a6c48a0350c2c488e2baa33bf569c8b2ce4ce3993).

• Ran the complete fused-MoE CUDA graph test file on eight GPUs: 6 passed,
  including packed-view parity, forward/backward graphs, unaligned token tails,
  and single-token automatic dispatch.

• Verified the final review fixes:

  • mismatched gate hidden dimensions fail before TMA descriptor construction;
  • packed w13 views match contiguous weights on eight GPUs; and
  • distributed autograd accepts strided no-grad inference, rejects strided
    grad mode before native allocation, and preserves the contiguous grad path.

• Integrated DP8+EP8 Qwen3-235B vLLM replay at 90% GPU memory completed 2,048
  requests with zero errors in 36.20 seconds, allocating 47.98 GiB of KV cache
  and 267,584 KV tokens per rank.

Native layout microbenchmark using identical weights, routes, tuned
configurations, and alternating execution order:

Tokens per rank	Contiguous weights	Strided packed views	Difference
512	1.615 ms	1.623 ms	+0.5%
4,096	4.000 ms	4.005 ms	+0.1%
16,384	12.301 ms	12.210 ms	-0.7%

The differences are within measurement noise; expert-strided TMA addressing
has no meaningful kernel latency penalty. Raw measurements:
/shared/public/sharing/seeker_summary_rl/v07_test/liger_strided_weight_microbench_20260812_080042.

🤖 Generated with GitHub Copilot CLI

[kolehma8]

would this break the existing use cases?

[supercharleszhu]

It will not (verified in unit test as well), weight_expert_stride_rows = (a.weight_expert_stride / hidden_dim), where weight_expert_stride is all_stride(0) by default. In original setup it is I * D. So expert_n_stride_1 will become I / TileN which is same as num_n_tiles_1