From 6a0238186eb2bdfb819218c3de04a9709dcecc22 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=B8=A9=E5=B7=AE?= Date: Thu, 5 Feb 2026 15:20:14 +0800 Subject: [PATCH 1/2] docs: polish Important Metrics to Monitor section Add comprehensive metrics monitoring guide with three subsections: - Reward Metrics: eval-rollout/reward, rollout/reward, ppo/actor/task_reward with troubleshooting for high variance - Importance Weight Metrics: Explain decoupled PPO loss formulation with importance_weight and behave_imp_weight, including troubleshooting tips - Sequence Length Metrics: no_eos_ratio and seq_len for detecting truncation Co-Authored-By: Claude Opus 4.5 --- docs/best_practices/algo_perf.md | 73 ++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) diff --git a/docs/best_practices/algo_perf.md b/docs/best_practices/algo_perf.md index e60d20220e..1b887c799c 100644 --- a/docs/best_practices/algo_perf.md +++ b/docs/best_practices/algo_perf.md @@ -43,3 +43,76 @@ diagnostic steps: DAPO) by passing a `should_accept_fn` parameter to `prepare_batch` to ensure task difficulty remains appropriate during runtime. See our [detailed code walk-through](../tutorial/gsm8k_grpo.md) for more information. + + +## Important Metrics to Monitor + +Monitoring these metrics helps ensure stable training and diagnose issues early. + +### Reward Metrics + +| Metric | Description | +|--------|-------------| +| `eval-rollout/reward` | Reward on the **test set**. Primary indicator of model generalization. | +| `rollout/reward` | Reward on the **training set**. Tracks learning progress during training. | +| `ppo/actor/task_reward` | Reward of trajectories **actually used for training**. Differs from `rollout/reward` when dynamic filtering is enabled—filtered trajectories are excluded here but still count in `rollout/reward`. | + +**Troubleshooting high variance in `task_reward`:** + +If `task_reward` fluctuates significantly, your training has high variance. Consider +increasing batch size if resources allow—this is often an effective remedy. + +### Importance Weight Metrics + +We recommend **asynchronous training with decoupled PPO loss** (`use_decoupled_loss=true`) +for optimal throughput. The two importance weight metrics below are critical for monitoring +training stability—their average values should remain very close to 1.0. + +With `use_decoupled_loss=true`, the loss function separates three policies: + +- **π_behave**: Behavior policy that generated the samples during rollout +- **π_proximal**: Proximal policy, one training step behind the current policy +- **π_θ**: Current policy being optimized + +The decoupled PPO loss combines two importance weights: + +$$L = -\mathbb{E}\left[ \underbrace{\frac{\pi_{\text{proximal}}}{\pi_{\text{behave}}}}_{\text{behave\_imp\_weight}} \cdot \min\left( \underbrace{\frac{\pi_\theta}{\pi_{\text{proximal}}}}_{\text{importance\_weight}} A, \text{clip}\left(\frac{\pi_\theta}{\pi_{\text{proximal}}}, 1-\epsilon, 1+\epsilon\right) A \right) \right]$$ + +| Metric | Formula | Description | +|--------|---------|-------------| +| `ppo_actor/update/importance_weight` | π_θ / π_proximal | Ratio for PPO clipping between current and proximal policies | +| `ppo_actor/update/behave_imp_weight` | π_proximal / π_behave | Off-policy correction for distribution mismatch in async training | + +**Troubleshooting `importance_weight` deviations:** + +- If `importance_weight/avg` deviates significantly from 1, reduce `ppo_n_minibatches`. +- With `ppo_n_minibatches == 1`, theoretically `importance_weight` should equal 1 exactly. +- If deviation persists at `ppo_n_minibatches == 1` (common in MoE training), add + `actor.megatron.use_deterministic_algorithms=1` to your config. + +**Troubleshooting `behave_imp_weight` deviations:** + +- Ensure `behav_imp_weight_cap` is set (recommended value: 5). +- If deviation persists, reduce `max_head_offpolicyness` to decrease sample staleness. + +### Sequence Length Metrics + +When training with long trajectories, monitor these metrics to detect truncation issues: + +| Metric | Description | +|--------|-------------| +| `ppo_actor/no_eos_ratio` | Fraction of trajectories truncated before generating EOS token | +| `ppo_actor/seq_len` | Average sequence length during training | + +**Troubleshooting high `no_eos_ratio`:** + +If `no_eos_ratio` exceeds 0.05 (5% of trajectories truncated): + +- Increase `max_new_tokens` to allow longer generations +- Use dynamic filtering to exclude overly long trajectories + +**Monitoring sequence length growth:** + +If `seq_len` increases steadily during training, watch `no_eos_ratio` closely—growing +sequence lengths often lead to more truncations. + From 77d8f049a3917ffdfae1e17a72e488f1c1d0897a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=8D=9A=E6=83=9F?= Date: Thu, 5 Feb 2026 16:02:31 +0800 Subject: [PATCH 2/2] fix doc format --- .pre-commit-config.yaml | 2 +- docs/best_practices/algo_perf.md | 32 ++++++++++++++++---------------- docs/cli_reference.md | 20 ++++++++++---------- 3 files changed, 27 insertions(+), 27 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 34e9e1c92b..3c37f845d5 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -34,7 +34,7 @@ repos: - mdformat-gfm - mdformat-tables - mdformat-frontmatter - exclude: ^docs/algorithms/.*\.md$ + exclude: ^(docs/algorithms/.*\.md|docs/best_practices/algo_perf.md)$ - repo: https://github.com/astral-sh/ruff-pre-commit # Ruff version. diff --git a/docs/best_practices/algo_perf.md b/docs/best_practices/algo_perf.md index 1b887c799c..3eec4c70ef 100644 --- a/docs/best_practices/algo_perf.md +++ b/docs/best_practices/algo_perf.md @@ -44,17 +44,16 @@ diagnostic steps: difficulty remains appropriate during runtime. See our [detailed code walk-through](../tutorial/gsm8k_grpo.md) for more information. - ## Important Metrics to Monitor Monitoring these metrics helps ensure stable training and diagnose issues early. ### Reward Metrics -| Metric | Description | -|--------|-------------| -| `eval-rollout/reward` | Reward on the **test set**. Primary indicator of model generalization. | -| `rollout/reward` | Reward on the **training set**. Tracks learning progress during training. | +| Metric | Description | +| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `eval-rollout/reward` | Reward on the **test set**. Primary indicator of model generalization. | +| `rollout/reward` | Reward on the **training set**. Tracks learning progress during training. | | `ppo/actor/task_reward` | Reward of trajectories **actually used for training**. Differs from `rollout/reward` when dynamic filtering is enabled—filtered trajectories are excluded here but still count in `rollout/reward`. | **Troubleshooting high variance in `task_reward`:** @@ -64,9 +63,10 @@ increasing batch size if resources allow—this is often an effective remedy. ### Importance Weight Metrics -We recommend **asynchronous training with decoupled PPO loss** (`use_decoupled_loss=true`) -for optimal throughput. The two importance weight metrics below are critical for monitoring -training stability—their average values should remain very close to 1.0. +We recommend **asynchronous training with decoupled PPO loss** +(`use_decoupled_loss=true`) for optimal throughput. The two importance weight metrics +below are critical for monitoring training stability—their average values should remain +very close to 1.0. With `use_decoupled_loss=true`, the loss function separates three policies: @@ -78,15 +78,16 @@ The decoupled PPO loss combines two importance weights: $$L = -\mathbb{E}\left[ \underbrace{\frac{\pi_{\text{proximal}}}{\pi_{\text{behave}}}}_{\text{behave\_imp\_weight}} \cdot \min\left( \underbrace{\frac{\pi_\theta}{\pi_{\text{proximal}}}}_{\text{importance\_weight}} A, \text{clip}\left(\frac{\pi_\theta}{\pi_{\text{proximal}}}, 1-\epsilon, 1+\epsilon\right) A \right) \right]$$ -| Metric | Formula | Description | -|--------|---------|-------------| -| `ppo_actor/update/importance_weight` | π_θ / π_proximal | Ratio for PPO clipping between current and proximal policies | +| Metric | Formula | Description | +| ------------------------------------ | --------------------- | ----------------------------------------------------------------- | +| `ppo_actor/update/importance_weight` | π_θ / π_proximal | Ratio for PPO clipping between current and proximal policies | | `ppo_actor/update/behave_imp_weight` | π_proximal / π_behave | Off-policy correction for distribution mismatch in async training | **Troubleshooting `importance_weight` deviations:** - If `importance_weight/avg` deviates significantly from 1, reduce `ppo_n_minibatches`. -- With `ppo_n_minibatches == 1`, theoretically `importance_weight` should equal 1 exactly. +- With `ppo_n_minibatches == 1`, theoretically `importance_weight` should equal 1 + exactly. - If deviation persists at `ppo_n_minibatches == 1` (common in MoE training), add `actor.megatron.use_deterministic_algorithms=1` to your config. @@ -99,10 +100,10 @@ $$L = -\mathbb{E}\left[ \underbrace{\frac{\pi_{\text{proximal}}}{\pi_{\text{beha When training with long trajectories, monitor these metrics to detect truncation issues: -| Metric | Description | -|--------|-------------| +| Metric | Description | +| ------------------------ | -------------------------------------------------------------- | | `ppo_actor/no_eos_ratio` | Fraction of trajectories truncated before generating EOS token | -| `ppo_actor/seq_len` | Average sequence length during training | +| `ppo_actor/seq_len` | Average sequence length during training | **Troubleshooting high `no_eos_ratio`:** @@ -115,4 +116,3 @@ If `no_eos_ratio` exceeds 0.05 (5% of trajectories truncated): If `seq_len` increases steadily during training, watch `no_eos_ratio` closely—growing sequence lengths often lead to more truncations. - diff --git a/docs/cli_reference.md b/docs/cli_reference.md index a4d27e3974..9cc7a1d88e 100644 --- a/docs/cli_reference.md +++ b/docs/cli_reference.md @@ -884,17 +884,17 @@ Refer to Megatron-LM documentation for implementation details. Configuration for OpenAI proxy when using agent workflows. -| Parameter | Type | Default | Description | -| ------------------------- | --------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `mode` | string | `"inline"` | OpenAI proxy mode: 'inline' (in-process) or 'subproc' (subprocess). `inline` mode runs the provided agent workflow directly in the same process. It can use the provided `base_url` and `http_client` to reduce overhead. `subproc` mode launches a separate process to run the agent with `OPENAI_BASE_URL` environment variable, which offers more flexible deployment options at the cost of larger overhead. **Choices:** `inline`, `subproc` | -| `tool_call_parser` | string | `"qwen3"` | Parser for tool calls in model output. | -| `reasoning_parser` | string | `"qwen3"` | Parser for reasoning content ( tags). | -| `chat_template_type` | string | `"hf"` | Chat template type: 'hf' (standard) or 'concat' (multi-turn concatenation). **Choices:** `hf`, `concat` | -| `engine_max_tokens` | integer \| None | `None` | Maximum total tokens for the engine (prompt + completion). | -| `turn_discount` | float | `1.0` | Discount factor for multi-turn reward propagation. | +| Parameter | Type | Default | Description | +| ------------------------- | --------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `mode` | string | `"inline"` | OpenAI proxy mode: 'inline' (in-process) or 'subproc' (subprocess). `inline` mode runs the provided agent workflow directly in the same process. It can use the provided `base_url` and `http_client` to reduce overhead. `subproc` mode launches a separate process to run the agent with `OPENAI_BASE_URL` environment variable, which offers more flexible deployment options at the cost of larger overhead. **Choices:** `inline`, `subproc` | +| `tool_call_parser` | string | `"qwen3"` | Parser for tool calls in model output. | +| `reasoning_parser` | string | `"qwen3"` | Parser for reasoning content ( tags). | +| `chat_template_type` | string | `"hf"` | Chat template type: 'hf' (standard) or 'concat' (multi-turn concatenation). **Choices:** `hf`, `concat` | +| `engine_max_tokens` | integer \| None | `None` | Maximum total tokens for the engine (prompt + completion). | +| `turn_discount` | float | `1.0` | Discount factor for multi-turn reward propagation. | | `export_style` | string | `"individual"` | Export style: 'individual' (all interactions) or 'concat' (leaf nodes only). The 'individual' style exports each interaction (input-output-reward) step separately, and treats them as independent samples to train the model. The 'concat' style exports only the final concatenated trajectory from the root. It is only suitable for linear conversation histories without token mismatching (whether valid depends on the tokenizer). **Choices:** `individual`, `concat` | -| `subproc_max_workers` | integer | `4` | Maximum number of worker processes for subprocess mode execution pool. | -| `session_timeout_seconds` | integer | `3600` | Session timeout in seconds. Sessions inactive longer than this will be garbage collected. | +| `subproc_max_workers` | integer | `4` | Maximum number of worker processes for subprocess mode execution pool. | +| `session_timeout_seconds` | integer | `3600` | Session timeout in seconds. Sessions inactive longer than this will be garbage collected. | (section-perf-tracer)=