docs: recipe pages for Kimi-K3, Nemotron-3-Ultra and Gemma-4

docs/docs.json

@@ -129,6 +129,7 @@
         "group": "Kimi",
         "root": "models/kimi/index",
         "pages": [
+         "models/kimi/kimi-k3",
          "models/kimi/kimi-k2.5",
          "models/kimi/kimi-k2",
          "models/kimi/moonlight"
@@ -147,7 +148,16 @@
           ],
           "expanded": false
          },
-         "models/nemotron/nemotron-3-super"
+         "models/nemotron/nemotron-3-super",
+         "models/nemotron/nemotron-3-ultra"
+        ],
+        "expanded": false
+       },
+       {
+        "group": "Gemma",
+        "root": "models/gemma/index",
+        "pages": [
+         "models/gemma/gemma-4"
         ],
         "expanded": false
        },

docs/index.md

@@ -79,9 +79,9 @@ of the box, including older generations of the families below.
 | **Thinking Machines** | [Inkling](/models/thinkingmachines/inkling)<br/>[Inkling-Small](/models/thinkingmachines/inkling-small) |
 | **Qwen** | [Qwen3.6 MoE](/models/qwen/qwen3-6-moe)<br/>[Qwen3.6](/models/qwen/qwen3-6)<br/>[Qwen3.5-35B-A3B](/models/qwen/qwen3-5-moe)<br/>[Qwen3.5-4B / 9B / 27B](/models/qwen/qwen3-5) |
 | **GLM** | [GLM-5.2](/models/glm/glm5-2)<br/>[GLM-5.1](/models/glm/glm5)<br/>[GLM-5](/models/glm/glm5)<br/>[GLM-4.7-Flash](/models/glm/glm4-7-flash) |
-| **Kimi** | [Kimi-K3](https://github.com/radixark/miles/pull/1825)<br/>[Kimi-K2.6](/models/kimi/kimi-k2.5)<br/>[Kimi-K2.5](/models/kimi/kimi-k2.5) |
-| **Nemotron** | [Nemotron-3-Ultra-550B-A55B](https://github.com/radixark/miles/blob/main/scripts/run_nemotron_3_ultra_550b_a55b.py)<br/>[Nemotron-3-Super-120B-A12B-FP8](/models/nemotron/nemotron-3-super)<br/>[Nemotron-3-Nano MoE](/models/nemotron/nemotron-3-nano-moe)<br/>[Nemotron-3-Nano](/models/nemotron/nemotron-3-nano) |
-| **Gemma** | [Gemma-4 26B-A4B](https://github.com/radixark/miles/blob/main/scripts/run_gemma_4_26b_a4b.py)<br/>[Gemma-4 31B](https://github.com/radixark/miles/blob/main/scripts/run_gemma_4_31b.py) |
+| **Kimi** | [Kimi-K3](/models/kimi/kimi-k3)<br/>[Kimi-K2.6](/models/kimi/kimi-k2.5)<br/>[Kimi-K2.5](/models/kimi/kimi-k2.5) |
+| **Nemotron** | [Nemotron-3-Ultra-550B-A55B](/models/nemotron/nemotron-3-ultra)<br/>[Nemotron-3-Super-120B-A12B-FP8](/models/nemotron/nemotron-3-super)<br/>[Nemotron-3-Nano MoE](/models/nemotron/nemotron-3-nano-moe)<br/>[Nemotron-3-Nano](/models/nemotron/nemotron-3-nano) |
+| **Gemma** | [Gemma-4 26B-A4B](/models/gemma/gemma-4)<br/>[Gemma-4 31B](/models/gemma/gemma-4) |
 | **JoyAI** | [JoyAI-LLM-Flash](https://github.com/radixark/miles/blob/main/scripts/run_joy_ai_llm_flash.py) |
 
 See [Models](/models/index) for exact conversion commands, launch scripts, and

docs/models/gemma/gemma-4.md

@@ -0,0 +1,151 @@
+---
+title: Gemma-4
+description: Launch recipes for Gemma-4 26B-A4B-it (MoE) and 31B-it (dense) via the HF to Megatron bridge.
+---
+## 1. Model Introduction
+
+[Gemma-4](https://huggingface.co/google) is Google's multimodal model line.
+miles trains both released instruction-tuned sizes as language models, on the
+base VLM checkpoint directly.
+
+Both go through the HF to Megatron bridge (`--megatron-to-hf-mode bridge`), and
+on the rollout side sglang runs `Gemma4ForConditionalGeneration`, which loads
+Gemma-4's hybrid `head_dim` weights correctly. There is no offline `torch_dist`
+conversion and no LLM-view rewrite of the checkpoint.
+
+**Key highlights:**
+
+- **Two shapes, one recipe family**: 26B-A4B is MoE (128 experts, top-8), 31B is
+  dense. They differ mainly in expert parallelism and the token budget.
+- **Bridge-mode load** straight from the VLM checkpoint.
+- **Tied embeddings**: neither config passes
+  `--untie-embeddings-and-output-weights`.
+- **Single node**: both recipes target 8 × H200.
+
+## 2. Supported Variants
+
+| Model | Class | Active / Total | Layers | Hidden | HF ID |
+|---|---|---|---|---|---|
+| Gemma-4 26B-A4B-it | MoE, 128 experts top-8 | 4 B / 26 B | 30 | 2816 | [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) |
+| Gemma-4 31B-it | Dense | 31 B | 60 | 5376 | [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) |
+
+Both use GQA with `kv_channels=256`, RoPE base 1e6, and a 262144-token vocab.
+
+The 31B recipe requires the `zhichen/gemma4-dense` branch of `radixark/Megatron-Bridge`.
+
+## 3. Environment Setup
+
+### 3.1 Download model + datasets
+
+```bash
+hf download --repo-type dataset zhuzilin/dapo-math-17k --local-dir /root/datasets/dapo-math-17k
+hf download google/gemma-4-26B-A4B-it --local-dir /root/models/gemma-4-26B-A4B-it
+```
+
+`--model-dir` and `--data-dir` default to `/root/models` and `/root/datasets`.
+`prepare` performs these downloads for you.
+
+### 3.2 No `torch_dist` conversion
+
+The bridge reads the HF checkpoint directly, so `--hf-checkpoint` and
+`--ref-load` both point at the download:
+
+```bash
+--hf-checkpoint <model-dir>/<model-name>
+--ref-load      <model-dir>/<model-name>
+--megatron-to-hf-mode bridge
+```
+
+## 4. Launch
+
+```bash
+cd /root/miles
+
+# MoE, single node
+python scripts/run_gemma_4_26b_a4b.py full-train --num-nodes 1
+
+# dense, single node
+python scripts/run_gemma_4_31b.py full-train --num-nodes 1
+```
+
+Passing `--num-nodes 1` puts the recipe into `debug_minimal` mode, which shortens
+`--rollout-max-response-len` to 256 for a quick smoke test. Multi-node runs use
+the full 8192.
+
+## 5. Recipe Configuration
+
+### 5.1 Parallelism
+
+| Model | TP | PP | CP | EP | ETP | `max_tokens_per_gpu` | GPUs |
+|---|---|---|---|---|---|---|---|
+| 26B-A4B (MoE) | 4 | 1 | 1 | 8 | 1 | 1024 | 8 (1 × 8) |
+| 31B (dense) | 4 | 1 | 1 | — | — | 512 | 8 (1 × 8) |
+
+Activation checkpointing is on for the MoE recipe
+(`--recompute-granularity full --recompute-method uniform
+--recompute-num-layers 1`). The dense 31B runs a smaller token budget because
+its 60 dense layers at hidden 5376 cost more activation memory per token than
+the MoE's 30 layers at 2816.
+
+### 5.2 Algorithm
+
+GRPO. The MoE recipe adds low-variance KL; the dense one runs without it:
+
+```bash
+--advantage-estimator grpo
+--entropy-coef 0.00
+--eps-clip 0.2
+--eps-clip-high 0.28
+--rm-type gemma_math
+--balance-data
+
+# 26B-A4B only
+--use-kl-loss
+--kl-loss-coef 0.00
+--kl-loss-type low_var_kl
+```
+
+Rollout batch 32 at 8 samples per prompt, global batch 256, `--lr 1e-6`. AIME
+evaluation every 20 steps is available behind `--enable-eval` and is off by
+default.
+
+### 5.3 Rollout & SGLang
+
+```bash
+--rollout-num-gpus-per-engine 4
+--sglang-mem-fraction-static 0.55   # 0.5 for the dense 31B
+```
+
+The MoE recipe pins sglang to conservative kernels:
+
+```bash
+--sglang-attention-backend triton
+--sglang-moe-runner-backend triton
+--sglang-disable-custom-all-reduce
+--sglang-disable-cuda-graph
+--sglang-disable-overlap-schedule
+--sglang-disable-radix-cache
+--use-rollout-routing-replay
+```
+
+`--use-rollout-routing-replay` replays the rollout's expert routing during the
+training forward pass, so train log-probs match rollout log-probs. Every
+sigmoid- or softmax-routed MoE recipe in miles needs this; the dense 31B does
+not.
+
+### 5.4 Notable quirks
+
+- **Trained on the VLM checkpoint.** miles does not strip the vision tower; the
+  bridge and sglang both handle the multimodal config, and the RL recipe simply
+  trains the language stack.
+- `--attention-backend unfused` on the training side for the MoE recipe.
+- Routing is softmax with `seq_aux_loss` balancing and the bias update rate at 0
+  (`--moe-router-bias-update-rate 0 --moe-aux-loss-coeff 0`), plus
+  `--moe-grouped-gemm` and `--moe-router-dtype fp32`.
+- The 31B recipe needs the `gemma4-dense` branch of `radixark/Megatron-Bridge`,
+  because the dense config is driven straight through `Gemma4VLBridge`.
+
+## 6. Pairs Well With
+
+- [Backends Beyond Megatron](/advanced/architecture-support)
+- [P2P Weight Transfer](/advanced/p2p-weight-transfer)

docs/models/gemma/index.md

@@ -0,0 +1,39 @@
+---
+title: Gemma
+description: Miles recipes for Google's Gemma-4 line, trained on the base VLM checkpoint through the HF to Megatron bridge.
+---
+Miles supports Google's Gemma-4 in both released instruction-tuned sizes. Both train as
+language models on the base VLM checkpoint, through the HF to Megatron bridge
+(`--megatron-to-hf-mode bridge`), so there is no offline `torch_dist` conversion.
+
+## Variants
+
+| Model | Class | Active / Total | HF ID | Recipe |
+|---|---|---|---|---|
+| Gemma-4 26B-A4B-it | MoE, 128 experts top-8 | 4 B / 26 B | `google/gemma-4-26B-A4B-it` | [gemma-4](/models/gemma/gemma-4) |
+| Gemma-4 31B-it | Dense | 31 B | `google/gemma-4-31B-it` | [gemma-4](/models/gemma/gemma-4) |
+
+## Fastest path to train
+
+Both recipes run on a single 8-GPU node:
+
+```bash
+cd /root/miles
+python scripts/run_gemma_4_26b_a4b.py full-train --num-nodes 1
+```
+
+`--num-nodes 1` shortens the response length for a smoke test. See
+[Gemma-4](/models/gemma/gemma-4) for the full walkthrough.
+
+## Which variant do I pick?
+
+- **Cheaper to train, sparse** → 26B-A4B. Four billion active parameters, and expert
+  parallelism carries the width.
+- **Dense, no routing to reason about** → 31B. It needs the `gemma4-dense` branch of
+  `radixark/Megatron-Bridge`, and runs a smaller per-GPU token budget because its 60 dense
+  layers cost more activation memory per token.
+
+## Pairs well with
+
+- [Backends Beyond Megatron](/advanced/architecture-support), the bridge path Gemma rides on.
+- [P2P Weight Transfer](/advanced/p2p-weight-transfer)

docs/models/index.md

@@ -13,11 +13,13 @@ Each model name links to its recipe page.
 | Family | Models |
 |---|---|
 | **DeepSeek** | [DeepSeek-V4 Pro](/models/deepseek/deepseek-v4-pro)<br/>[DeepSeek-V4 Flash](/models/deepseek/deepseek-v4-flash)<br/>[DeepSeek-V3.2](/models/deepseek/deepseek-v3-2) |
-| **Thinking Machines** | [Inkling](/models/thinkingmachines/inkling) |
+| **Thinking Machines** | [Inkling](/models/thinkingmachines/inkling)<br/>[Inkling-Small](/models/thinkingmachines/inkling-small) |
 | **Qwen** | [Qwen3.6 MoE](/models/qwen/qwen3-6-moe)<br/>[Qwen3.6](/models/qwen/qwen3-6)<br/>[Qwen3.5-35B-A3B](/models/qwen/qwen3-5-moe)<br/>[Qwen3.5-4B / 9B / 27B](/models/qwen/qwen3-5)<br/>[Qwen3-Next-80B-A3B-Thinking](/models/qwen/qwen3-next)<br/>[Qwen3-30B-A3B / 235B-A22B](/models/qwen/qwen3-moe)<br/>[Qwen3-0.6B / 1.7B / 4B / 8B / 14B / 32B](/models/qwen/qwen3) |
 | **GLM** | [GLM-5.2](/models/glm/glm5-2)<br/>[GLM-5.1](/models/glm/glm5)<br/>[GLM-5](/models/glm/glm5)<br/>[GLM-4.7-Flash](/models/glm/glm4-7-flash)<br/>[GLM-4.5](/models/glm/glm4-5)<br/>[GLM-Z1-9B-0414](/models/glm/glm4) |
-| **Kimi** | [Kimi-K2.6](/models/kimi/kimi-k2.5)<br/>[Kimi-K2.5](/models/kimi/kimi-k2.5)<br/>[Kimi-K2-Instruct / Thinking](/models/kimi/kimi-k2)<br/>[Moonlight-16B-A3B](/models/kimi/moonlight) |
-| **Nemotron** | [Nemotron-3-Super-120B-A12B-FP8](/models/nemotron/nemotron-3-super)<br/>[Nemotron-3-Nano MoE](/models/nemotron/nemotron-3-nano-moe)<br/>[Nemotron-3-Nano](/models/nemotron/nemotron-3-nano) |
+| **Kimi** | [Kimi-K3](/models/kimi/kimi-k3)<br/>[Kimi-K2.6](/models/kimi/kimi-k2.5)<br/>[Kimi-K2.5](/models/kimi/kimi-k2.5)<br/>[Kimi-K2-Instruct / Thinking](/models/kimi/kimi-k2)<br/>[Moonlight-16B-A3B](/models/kimi/moonlight) |
+| **Nemotron** | [Nemotron-3-Ultra-550B-A55B](/models/nemotron/nemotron-3-ultra)<br/>[Nemotron-3-Super-120B-A12B-FP8](/models/nemotron/nemotron-3-super)<br/>[Nemotron-3-Nano MoE](/models/nemotron/nemotron-3-nano-moe)<br/>[Nemotron-3-Nano](/models/nemotron/nemotron-3-nano) |
+| **Gemma** | [Gemma-4 26B-A4B](/models/gemma/gemma-4)<br/>[Gemma-4 31B](/models/gemma/gemma-4) |
+| **JoyAI** | [JoyAI-LLM-Flash](https://github.com/radixark/miles/blob/main/scripts/run_joy_ai_llm_flash.py) |
 | **MiMo** | [MiMo-7B-RL](/models/mimo/mimo) |
 | **GPT-OSS** | [gpt-oss-20b](/models/gpt-oss/gpt-oss) |