Add on-policy distillation example (Qwen3-8B + Qwen3-32B vLLM teacher)

[CalvinXKY]

Summary

• Add examples/on_policy_distillation/ aligned with the slime example layout:
  • run-qwen3-8B-opd.sh: external vLLM teacher (Qwen3-8B student + Qwen3-32B teacher)
  • run-qwen3-8B-opd-megatron.sh: Megatron-loaded teacher (self-distillation demo)
• Student rollout always uses vLLM (--opd-type vllm / megatron; no sglang).
• Hyperparams in run-qwen3-8B-opd.sh match a validated 8×A800 run.

Related to #327.

Validated results (run-qwen3-8B-opd.sh)

Model	GSM8K Accuracy
Qwen3-8B (pre-OPD)	79.7% (n=300)
Qwen3-8B (post-OPD, ~220 rollouts)	88.2% (n=1319, +8.5 pp)
Qwen3-32B teacher	87.0% (n=300)

• rollout/opd_reverse_kl: 0.145 → ~0.10 (−38%)
• Train data: dapo-math-17k; offline GSM8K greedy eval

Test plan

• Pre-commit passed locally
• Rebased onto latest main
• End-to-end validated on 8×A800 (8B student + 32B vLLM teacher)
• Review README / script paths and GPU layout
• (Optional) Smoke-test Megatron teacher script with reduced --num-rollout

[read-the-docs-community]

Documentation build overview

📚 vime | 🛠️ Build #33645529 | 📁 Comparing 058a41c against latest (2702eef)

  🔍 Preview build  

45 files changed · + 3 added · ± 42 modified

+ Added

• examples/gemma4.html
• platform_support/ascend_tutorial.html
• _examples_synced/on_policy_distillation/README.html

± Modified

• genindex.html
• index.html
• search.html
• advanced/arch-support-beyond-megatron.html
• advanced/delta-weight-sync.html
• advanced/external-rollout-engines.html
• advanced/fault-tolerance.html
• advanced/low-precision.html
• advanced/megatron-config.html
• advanced/observability.html
• and 32 more...

[gemini-code-assist]

Code Review

This pull request introduces an on-policy distillation (OPD) example, adding a comprehensive README and two shell scripts (run-qwen3-4b-32b-opd.sh and run-qwen3-8b-opd-megatron.sh) to demonstrate both vLLM and Megatron teacher modes. The feedback highlights several shell script improvements: replacing overly aggressive pkill -9 python commands with targeted process termination to avoid disrupting unrelated workloads, adding set -o pipefail to prevent masked pipeline failures when piping to tee, and ensuring previous vLLM processes are cleaned up at startup to avoid port and GPU conflicts.

[gemini-code-assist]

Using pkill -9 python is extremely aggressive and dangerous as it will kill all Python processes running on the system, which can disrupt other users' workloads or even kill the parent process/CI runner if it is Python-based. It is much safer to use a more targeted command like pkill -9 -f "train.py".

Suggested change

	pkill -9 ray 2>/dev/null || true
	pkill -9 python 2>/dev/null || true
	pkill -9 ray 2>/dev/null || true
	pkill -9 -f "train.py" 2>/dev/null || true

[gemini-code-assist]

Similarly, avoid using pkill -9 python at the end of the script to prevent killing unrelated Python processes on the system. Use pkill -9 -f "train.py" instead.

Suggested change

	pkill -9 ray 2>/dev/null || true
	pkill -9 python 2>/dev/null || true
	pkill -9 ray 2>/dev/null || true
	pkill -9 -f "train.py" 2>/dev/null || true

[gemini-code-assist]

The script uses set -ex but does not set set -o pipefail. Since the training command on line 189 pipes its output to tee (2>&1 | tee ...), any failure in ray job submit or train.py will be masked because tee will exit with status 0, causing the script to incorrectly report success. It is highly recommended to add set -o pipefail to ensure pipeline failures are properly propagated.

Suggested change

	set -ex
	set -ex
	set -o pipefail

[gemini-code-assist]

If a previous run of this script failed or was interrupted, the teacher vLLM server might still be running on port 13141 and holding GPU memory. It is recommended to clean up any existing vLLM processes at the start of the script to prevent port/GPU conflicts.

Suggested change

	echo "=== Step 1: Clean up previous Ray / training processes ==="
	ray stop --force 2>/dev/null || true
	echo "=== Step 1: Clean up previous Ray / training processes ==="
	pkill -9 -f "vllm" 2>/dev/null || true
	ray stop --force 2>/dev/null || true