Skip to content

docs: add Important Metrics to Monitor section #894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
garrett4wade merged 2 commits into from
Feb 5, 2026
Merged

docs: add Important Metrics to Monitor section #894

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6a02381
docs: polish Important Metrics to Monitor section
xssstory Feb 5, 2026
77d8f04
fix doc format
garrett4wade Feb 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ repos:
- mdformat-gfm
- mdformat-tables
- mdformat-frontmatter
exclude: ^docs/algorithms/.*\.md$
exclude: ^(docs/algorithms/.*\.md|docs/best_practices/algo_perf.md)$

- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
Expand Down
73 changes: 73 additions & 0 deletions docs/best_practices/algo_perf.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,76 @@ diagnostic steps:
DAPO) by passing a `should_accept_fn` parameter to `prepare_batch` to ensure task
difficulty remains appropriate during runtime. See our
[detailed code walk-through](../tutorial/gsm8k_grpo.md) for more information.

## Important Metrics to Monitor

Monitoring these metrics helps ensure stable training and diagnose issues early.

### Reward Metrics

| Metric | Description |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `eval-rollout/reward` | Reward on the **test set**. Primary indicator of model generalization. |
| `rollout/reward` | Reward on the **training set**. Tracks learning progress during training. |
| `ppo/actor/task_reward` | Reward of trajectories **actually used for training**. Differs from `rollout/reward` when dynamic filtering is enabled—filtered trajectories are excluded here but still count in `rollout/reward`. |

**Troubleshooting high variance in `task_reward`:**

If `task_reward` fluctuates significantly, your training has high variance. Consider
increasing batch size if resources allow—this is often an effective remedy.

### Importance Weight Metrics

We recommend **asynchronous training with decoupled PPO loss**
(`use_decoupled_loss=true`) for optimal throughput. The two importance weight metrics
below are critical for monitoring training stability—their average values should remain
very close to 1.0.

With `use_decoupled_loss=true`, the loss function separates three policies:

- **π_behave**: Behavior policy that generated the samples during rollout
- **π_proximal**: Proximal policy, one training step behind the current policy
- **π_θ**: Current policy being optimized

The decoupled PPO loss combines two importance weights:

$$L = -\mathbb{E}\left[ \underbrace{\frac{\pi_{\text{proximal}}}{\pi_{\text{behave}}}}_{\text{behave\_imp\_weight}} \cdot \min\left( \underbrace{\frac{\pi_\theta}{\pi_{\text{proximal}}}}_{\text{importance\_weight}} A, \text{clip}\left(\frac{\pi_\theta}{\pi_{\text{proximal}}}, 1-\epsilon, 1+\epsilon\right) A \right) \right]$$

| Metric | Formula | Description |
| ------------------------------------ | --------------------- | ----------------------------------------------------------------- |
| `ppo_actor/update/importance_weight` | π_θ / π_proximal | Ratio for PPO clipping between current and proximal policies |
| `ppo_actor/update/behave_imp_weight` | π_proximal / π_behave | Off-policy correction for distribution mismatch in async training |

**Troubleshooting `importance_weight` deviations:**

- If `importance_weight/avg` deviates significantly from 1, reduce `ppo_n_minibatches`.
- With `ppo_n_minibatches == 1`, theoretically `importance_weight` should equal 1
exactly.
- If deviation persists at `ppo_n_minibatches == 1` (common in MoE training), add
`actor.megatron.use_deterministic_algorithms=1` to your config.

**Troubleshooting `behave_imp_weight` deviations:**

@gemini-code-assist gemini-code-assist Bot Feb 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There's a slight naming inconsistency that could be confusing for users. This section and the metric itself are named behave_imp_weight, but the corresponding configuration parameter is behav_imp_weight_cap. While this reflects the codebase, adding a small note to clarify this difference would enhance the documentation's clarity.


- Ensure `behav_imp_weight_cap` is set (recommended value: 5).
- If deviation persists, reduce `max_head_offpolicyness` to decrease sample staleness.

### Sequence Length Metrics

When training with long trajectories, monitor these metrics to detect truncation issues:

| Metric | Description |
| ------------------------ | -------------------------------------------------------------- |
| `ppo_actor/no_eos_ratio` | Fraction of trajectories truncated before generating EOS token |
| `ppo_actor/seq_len` | Average sequence length during training |

**Troubleshooting high `no_eos_ratio`:**

If `no_eos_ratio` exceeds 0.05 (5% of trajectories truncated):

- Increase `max_new_tokens` to allow longer generations
- Use dynamic filtering to exclude overly long trajectories

**Monitoring sequence length growth:**

If `seq_len` increases steadily during training, watch `no_eos_ratio` closely—growing
sequence lengths often lead to more truncations.
20 changes: 10 additions & 10 deletions docs/cli_reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -884,17 +884,17 @@ Refer to Megatron-LM documentation for implementation details.

Configuration for OpenAI proxy when using agent workflows.

| Parameter | Type | Default | Description |
| ------------------------- | --------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mode` | string | `"inline"` | OpenAI proxy mode: 'inline' (in-process) or 'subproc' (subprocess). `inline` mode runs the provided agent workflow directly in the same process. It can use the provided `base_url` and `http_client` to reduce overhead. `subproc` mode launches a separate process to run the agent with `OPENAI_BASE_URL` environment variable, which offers more flexible deployment options at the cost of larger overhead. **Choices:** `inline`, `subproc` |
| `tool_call_parser` | string | `"qwen3"` | Parser for tool calls in model output. |
| `reasoning_parser` | string | `"qwen3"` | Parser for reasoning content (<think> tags). |
| `chat_template_type` | string | `"hf"` | Chat template type: 'hf' (standard) or 'concat' (multi-turn concatenation). **Choices:** `hf`, `concat` |
| `engine_max_tokens` | integer \| None | `None` | Maximum total tokens for the engine (prompt + completion). |
| `turn_discount` | float | `1.0` | Discount factor for multi-turn reward propagation. |
| Parameter | Type | Default | Description |
| ------------------------- | --------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mode` | string | `"inline"` | OpenAI proxy mode: 'inline' (in-process) or 'subproc' (subprocess). `inline` mode runs the provided agent workflow directly in the same process. It can use the provided `base_url` and `http_client` to reduce overhead. `subproc` mode launches a separate process to run the agent with `OPENAI_BASE_URL` environment variable, which offers more flexible deployment options at the cost of larger overhead. **Choices:** `inline`, `subproc` |
| `tool_call_parser` | string | `"qwen3"` | Parser for tool calls in model output. |
| `reasoning_parser` | string | `"qwen3"` | Parser for reasoning content (<think> tags). |
| `chat_template_type` | string | `"hf"` | Chat template type: 'hf' (standard) or 'concat' (multi-turn concatenation). **Choices:** `hf`, `concat` |
| `engine_max_tokens` | integer \| None | `None` | Maximum total tokens for the engine (prompt + completion). |
| `turn_discount` | float | `1.0` | Discount factor for multi-turn reward propagation. |
| `export_style` | string | `"individual"` | Export style: 'individual' (all interactions) or 'concat' (leaf nodes only). The 'individual' style exports each interaction (input-output-reward) step separately, and treats them as independent samples to train the model. The 'concat' style exports only the final concatenated trajectory from the root. It is only suitable for linear conversation histories without token mismatching (whether valid depends on the tokenizer). **Choices:** `individual`, `concat` |
| `subproc_max_workers` | integer | `4` | Maximum number of worker processes for subprocess mode execution pool. |
| `session_timeout_seconds` | integer | `3600` | Session timeout in seconds. Sessions inactive longer than this will be garbage collected. |
| `subproc_max_workers` | integer | `4` | Maximum number of worker processes for subprocess mode execution pool. |
| `session_timeout_seconds` | integer | `3600` | Session timeout in seconds. Sessions inactive longer than this will be garbage collected. |

(section-perf-tracer)=

Expand Down