[OPD] Add Qwen3.5-35B-A3B single-node self-distillation example

[maocheng23]

What

A reproducible two-phase on-policy-distillation example for Qwen3.5-35B-A3B on a single 8×H200 node, using the in-process Megatron teacher (--opd-type megatron).

• Phase 1 (phase1_rlvr_teacher.sh): RLVR-train the base 35B into a teacher that is measurably better and more concise — eval/dapo_heldout 0.828 → 0.887, response length ~14k → ~6k (lr 1e-5).
• Phase 2 (phase2_opd_selfdistill.sh): distill that teacher into the base student.
  • pure mode: training reward = 0, so the only signal is the teacher's reverse-KL — clean attribution. Result: eval length −57% (14k→6.1k) with accuracy preserved/slightly up (0.840→0.852), opd_reverse_kl 0.045→0.013 (student converging onto the teacher).
  • grounded mode: keep a correctness reward (so rollout/raw_reward is meaningful and climbs) plus the teacher reverse-KL.

Why it's distinct from the existing examples

• run-qwen3-8B-opd-megatron.sh uses teacher == base — a mechanism demo where opd_reverse_kl ≈ 0. This example trains a genuinely diverged teacher first, which is the prerequisite for OPD to actually move the student.
• Real MoE at scale on one node: the 2-node/16-GPU recipe is re-tiled to 8 GPUs (TP2/PP1/CP2/EP8/ETP1, world=8), --colocate + --optimizer-cpu-offload, ≈124/143 GB per GPU.
• Self-distillation is required: Qwen3.5's tokenizer (vocab 248320) is not token-compatible with the smaller Qwen3 models (vocab 151936).

Contents

file	purpose
README.md	full pipeline, single-node parallelism derivation, results, gotchas
phase1_rlvr_teacher.sh	Phase-1 RLVR teacher training
phase2_opd_selfdistill.sh	Phase-2 OPD (pure / grounded via MODE)
rm.py	format-agnostic correctness reward + pure-OPD reward
make_split.py	seeded, disjoint train/eval split of dapo-math-17k
eval_dapo_heldout.yaml	held-out eval config

Gotchas documented (each cost a wasted run)

• --rm-type deepscaler needs a </think> tag → scores Qwen3.5 (inline reasoning) as 0; math/dapo each read only one answer format. The example uses a format-agnostic reward and always sets --label-key label.
• The 35B's CoT is ~14–17k tokens; an 8k response cap truncates ~95% of rollouts → reward 0. Use ≥24k (CP2 makes it fit).
• --opd-teacher-load must point at the checkpoint parent dir (with latest_checkpointed_iteration.txt), not an iter_* subdir — otherwise it silently falls back to base and opd_reverse_kl ≈ 0.
• The teacher must actually diverge: lr 1e-6 / few steps ≈ base (inert); lr 1e-5 diverges fast.
• with_ref = (--use-kl-loss or --kl-coef≠0) — dropping --use-kl-loss keeps only student+teacher (2×35B) in memory and avoids a 3rd-model OOM.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a reproducible two-phase self-distillation pipeline for the Qwen3.5-35B-A3B MoE model on a single node, including training scripts, evaluation configurations, dataset splitting utilities, and custom format-agnostic reward functions. The feedback suggests improving the robustness of the prompt parsing logic in make_split.py to handle missing keys or malformed structures, and double-quoting array expansions in the Bash scripts (phase1_rlvr_teacher.sh and phase2_opd_selfdistill.sh) to prevent unexpected word splitting.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To improve robustness and adhere to defensive programming practices, handle cases where d might not contain the "prompt" key, or where elements in the prompt list are None or not dictionaries (which would cause AttributeError or KeyError).

Suggested change

	def prompt_text(d):
	p = d["prompt"]
	return "\n".join(m.get("content", "") for m in p) if isinstance(p, list) else str(p)
	def prompt_text(d):
	p = d.get("prompt", "")
	if isinstance(p, list):
	return "\\n".join(m.get("content", "") if isinstance(m, dict) else str(m) for m in p if m is not None)
	return str(p)

[gemini-code-assist]

In Bash, expanding arrays without double quotes (e.g., ${MODEL_ARGS[@]}) can lead to unexpected word splitting and glob expansion if any of the arguments contain spaces or special characters. Always use double quotes around array expansions: "${ARRAY[@]}".

Suggested change

	${MODEL_ARGS[@]} ${CKPT_ARGS[@]} ${ROLLOUT_ARGS[@]} ${OPTIMIZER_ARGS[@]} ${GRPO_ARGS[@]} \
	${WANDB_ARGS[@]} ${PERF_ARGS[@]} ${EVAL_ARGS[@]} ${SGLANG_ARGS[@]} ${MISC_ARGS[@]} ${RM_ARGS[@]}
	"${MODEL_ARGS[@]}" "${CKPT_ARGS[@]}" "${ROLLOUT_ARGS[@]}" "${OPTIMIZER_ARGS[@]}" "${GRPO_ARGS[@]}" \\
	"${WANDB_ARGS[@]}" "${PERF_ARGS[@]}" "${EVAL_ARGS[@]}" "${SGLANG_ARGS[@]}" "${MISC_ARGS[@]}" "${RM_ARGS[@]}"

[gemini-code-assist]

In Bash, expanding arrays without double quotes (e.g., ${MODEL_ARGS[@]}) can lead to unexpected word splitting and glob expansion if any of the arguments contain spaces or special characters. Always use double quotes around array expansions: "${ARRAY[@]}".

Suggested change

	${MODEL_ARGS[@]} ${CKPT_ARGS[@]} ${OPD_ARGS[@]} ${ROLLOUT_ARGS[@]} ${OPTIMIZER_ARGS[@]} ${GRPO_ARGS[@]} \
	${WANDB_ARGS[@]} ${PERF_ARGS[@]} ${EVAL_ARGS[@]} ${SGLANG_ARGS[@]} ${MISC_ARGS[@]} ${RM_ARGS[@]}
	"${MODEL_ARGS[@]}" "${CKPT_ARGS[@]}" "${OPD_ARGS[@]}" "${ROLLOUT_ARGS[@]}" "${OPTIMIZER_ARGS[@]}" "${GRPO_ARGS[@]}" \\
	"${WANDB_ARGS[@]}" "${PERF_ARGS[@]}" "${EVAL_ARGS[@]}" "${SGLANG_ARGS[@]}" "${MISC_ARGS[@]}" "${RM_ARGS[@]}"

[Shi-Dong]

LGTM!