feat(models): add native Qwen3.8 Flash Next

[S1ro1]

Summary

• Add a fully Prime-owned Qwen3.8 Flash Next trainer implementation covering expanded residual streams, Gated DeltaNet layers, indexed sparse attention, position-learning enhancement, and sigmoid-output-gated MoE blocks.
• Keep the implementation independent of Transformers modeling classes. Prime owns the text configuration, model assembly, initialization, packed-sequence boundaries, context-parallel contracts, and checkpoint conversion.
• Vendor the indexed sparse-attention forward/backward kernel in prime-kernels with GQA, sentinel padding, context parallelism, compile, and selective activation checkpointing support.
• Implement sparse-index scoring and exact radix selection as standalone Python-only TileLang kernels over Prime's contiguous packed training keys. No compiled selector extension or Triton runtime dependency is required.
• Store the PLE N-gram table as one flat checkpoint parameter and shard its rows over dp_shard_cp at runtime. Lookups exchange token IDs and embeddings over a dedicated NCCL group; weights are never communicated during forward or backward.
• Support PLE CPU offload with asynchronous prefetch while preserving DCP/DTensor loading of the ordinary global checkpoint tensor.
• Extend NIXL's generic lazy weight graph to trace concatenated checkpoint sources, and route vLLM's PLE shard writes through the standard parameter loader required by the RL weight-update lifecycle.
• Pin the Qwen-capable vLLM revision and use its official precompiled CUDA 13 extensions so CPU CI does not attempt a local CUDA build.
• Register the model with the Qwen3.8 renderer from feat(renderers): register Qwen3.8 Flash Next renderers#145.

This PR is stacked on #3429. It depends on PrimeIntellect-ai/prime-kernels#8 and pins the implementation from vllm-project/vllm#53896.

Runtime contract

• model.impl = "custom" selects the Prime model for qwen4_exp and qwen4_exp_text configurations.
• Decoder layers use the checkpoint-declared schedule: Gated DeltaNet for linear-attention layers and indexed gated attention for full-attention layers.
• Sparse attention uses the frozen QSA indexer and a 2048-token causal selection budget. Only the attention values are trainable; index selection matches the inference implementation.
• MoE blocks use the shared Prime split gate_proj, up_proj, and down_proj runtime layout with a sigmoid shared-expert output gate.
• Packed sequence and context-parallel boundaries are explicit for Gated DeltaNet, indexed attention, PLE hashing, and the dilated PLE convolution.
• The Prime checkpoint contains one global ple_embedding.ngram_embedding.weight. DCP loads the correct local DTensor shard; there is no sharding-aware checkpoint format or loader.
• NIXL policy reload does not require routed-expert metadata. Trainer, orchestrator, and inference run with routed-expert returns disabled.
• Vision and MTP weights are outside this text-training implementation. Router auxiliary-loss configuration is ignored.

Validation

• Ruff, Ruff formatting, uv lock --check, and git diff --check passed.
• Focused checkpointing and MoE conversion suite: 6 passed.
• Renderer suite: 10 passed.
• GitHub CPU CI passed after building and importing the pinned vLLM source through the configured precompiled CUDA 13 extensions; a separate local reproduction also passed with CUDA_HOME removed.
• Indexed-attention BF16 forward/backward matched an FP32 PyTorch reference on H200; production-width selection, fullgraph compile, selective checkpointing, and two-rank context parallelism completed with finite matching outputs and gradients.
• The TileLang scorer matched the inference reference bit-for-bit and the radix selector matched selected sets exactly on two 16K sequences, ragged 32K packing, smaller packed batches, and sub-block sequences. At the production 2x16K shape, the complete TileLang selector took 5.598 ms, including 2.852 ms for score computation. The 4-layer EP8 forward-only profile completed in 331 ms with compile, selective activation checkpointing, and no LM head. Perfetto trace.
• A 16-H200 production-shape PLE probe passed exact head ownership, forward lookup, and backward gradient aggregation with unequal token counts per rank.
• The real flat .prime-v1 checkpoint loaded through DCP into the head-sharded DTensor parameter without checkpoint-specific loading logic.
• SLURM job 3013 completed 20 RL steps on 24 H200s: 16 trainer GPUs and 8 vLLM GPUs. The run used sequence length 2048, i3_math, SignSGD, compile, selective activation checkpointing, eight-way expert parallelism, NIXL updates, and the real checkpoint.
• All 20 policy updates were applied by every vLLM rank. Generated samples were coherent, the final update completed, and trainer/orchestrator/inference exited successfully. Peak trainer memory was 130.3 GiB.

Step	Mean mismatch KL
1	0.001930760
2	0.001745779
3	0.001625292
4	0.001853466
5	0.002050616
6	0.001903126
7	0.001503883
8	0.002082327
9	0.001638428
10	0.001815931
11	0.001630328
12	0.001574650
13	0.001749059
14	0.001496617
15	0.001717085
16	0.001813328
17	0.002259263
18	0.001881316
19	0.001684328
20	0.001593116

Mean: 0.001777435. Minimum: 0.001496617. Maximum: 0.002259263. Final: 0.001593116; there is no rising trend.

_{Stack maintained with GitHub Stacks CLI • Give Feedback 💬}

---

Note

High Risk
Large new model stack plus vLLM git pin, distributed embedding offload, and NIXL/inference weight-loading changes affect training–inference parity and multi-node RL updates.

Overview
Adds a Prime-owned Qwen3.8 Flash Next (qwen4_exp) trainer: hyper-connection residual streams, Gated DeltaNet / indexed sparse attention (prime_kernels.indexed_attention), position-learning enhancement with hashed N-gram tables, and shared sigmoid-output-gated MoE (also refactors Qwen3.5 MoE to the same building blocks). Checkpoint conversion merges sharded PLE N-gram weights into one runtime table.

Distributed training for huge PLE tables: new HeadShardedEmbedding shards rows on dp_shard_cp, stays outside FSDP sharding, and optional ngram_embedding_cpu_offload keeps weights in pinned CPU with prefetch; optimizer CPU offload, grad clipping, and Gloo setup are extended for mixed GPU/CPU-sharded parameters.

Inference / RL weight path: pins vLLM on x86_64 to git c855760 with precompiled cu130 wheels; updates API server hooks to vllm.entrypoints.launchers.*; replaces the custom fp32 lm_head monkey-patch with native hf_overrides["head_dtype"]="float32"; adds a vLLM patch for Qwen4 PLE checkpoint shard loading. NIXL lazy weights now trace torch.cat via ConcatenatedLazyWeight for concatenated checkpoint sources.

Docs/skills note indexed_attention, Python-only kernels, and UV_NO_SYNC for concurrent SLURM uv run from one worktree.

^{Reviewed by Cursor Bugbot for commit 410151b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit adbd3c3. Configure here.}