Fix evaluation script and update tutorials#2
Merged
Conversation
antoinegg1
pushed a commit
that referenced
this pull request
Jul 7, 2025
Fix evaluation script and update tutorials
dhh1995
pushed a commit
that referenced
this pull request
Oct 16, 2025
Fixed issue with tau2 domain cli.
zhshgmail
added a commit
to HsiaoTsan/AReaL
that referenced
this pull request
Nov 4, 2025
- Replace Python for loop with scatter_add_ for GPU performance - Simplify advantage aggregation description in documentation - All 18 GSPO tests pass with identical results Addresses GitHub review comments areal-project#1 and areal-project#2.
garrett4wade
pushed a commit
that referenced
this pull request
Nov 4, 2025
* feat: implement GSPO (Group-level Sequential Policy Optimization)
Add GSPO support with sequence-level advantage computation instead of token-level.
This implementation includes:
- Add importance_sampling_level configuration ("token" for PPO, "sequence" for GSPO)
- Modify ppo_actor_loss_fn to compute advantages at sequence level for GSPO
- Add comprehensive tests for sequence-level importance sampling
- Add GSPO example with GSM8K dataset configuration
- Update weight_update_meta initialization and validation workflow
- Merge DAPO configs for better flexibility
- Update README with GSPO usage and configuration details
The key difference from standard PPO is that GSPO sums token-level advantages
within each sequence before computing the policy gradient, enabling more stable
sequence-level optimization.
* Address PR review comments
Changes:
- Add more assertions in unit tests for GSPO
- Fix GSPO packed sequences to use cu_seqlens for per-sequence ratios
- Fix GSPO gradient scaling bug by averaging instead of summing advantages
- Add GSPO algorithm description and implementation notes to documentation
- Add PPOActor configuration logging for reproducibility
- Simplify documentation by removing detailed implementation notes
- Update examples to reuse existing gsm8k_grpo.py with GSPO config
- Remove dedicated gsm8k_gspo.py script (not necessary)
- Remove unnecessary overlong penalty parameters from config
GSPO can be enabled with any PPO-like training by setting
importance_sampling_level: sequence in the configuration.
* Ajust code format to pass pipeline black formatter check.
* perf: vectorize GSPO 1D packed tensor computation and simplify docs
- Replace Python for loop with scatter_add_ for GPU performance
- Simplify advantage aggregation description in documentation
- All 18 GSPO tests pass with identical results
Addresses GitHub review comments #1 and #2.
* Ajust code format to pass pipeline black formatter check.
---------
Co-authored-by: bruceli_ed24 <bruce.li2@h-partners.com>
edbeeching
pushed a commit
to edbeeching/AReaL
that referenced
this pull request
Mar 20, 2026
* docs: explain how to enable kernels in training Co-authored-by: OpenAI Codex <codex@openai.com> * docs: link Hugging Face kernels docs Co-authored-by: OpenAI Codex <codex@openai.com> * Apply suggestion from @lewtun * refactor: share attn_impl validation Co-authored-by: Codex <codex@openai.com> * refactor: reuse transformers kernel validator Co-authored-by: OpenAI Codex <codex@openai.com> --------- Co-authored-by: OpenAI Codex <codex@openai.com>
leandermaben
pushed a commit
to leandermaben/AReaL
that referenced
this pull request
Mar 24, 2026
…eal-project#501) * feat: implement GSPO (Group-level Sequential Policy Optimization) Add GSPO support with sequence-level advantage computation instead of token-level. This implementation includes: - Add importance_sampling_level configuration ("token" for PPO, "sequence" for GSPO) - Modify ppo_actor_loss_fn to compute advantages at sequence level for GSPO - Add comprehensive tests for sequence-level importance sampling - Add GSPO example with GSM8K dataset configuration - Update weight_update_meta initialization and validation workflow - Merge DAPO configs for better flexibility - Update README with GSPO usage and configuration details The key difference from standard PPO is that GSPO sums token-level advantages within each sequence before computing the policy gradient, enabling more stable sequence-level optimization. * Address PR review comments Changes: - Add more assertions in unit tests for GSPO - Fix GSPO packed sequences to use cu_seqlens for per-sequence ratios - Fix GSPO gradient scaling bug by averaging instead of summing advantages - Add GSPO algorithm description and implementation notes to documentation - Add PPOActor configuration logging for reproducibility - Simplify documentation by removing detailed implementation notes - Update examples to reuse existing gsm8k_grpo.py with GSPO config - Remove dedicated gsm8k_gspo.py script (not necessary) - Remove unnecessary overlong penalty parameters from config GSPO can be enabled with any PPO-like training by setting importance_sampling_level: sequence in the configuration. * Ajust code format to pass pipeline black formatter check. * perf: vectorize GSPO 1D packed tensor computation and simplify docs - Replace Python for loop with scatter_add_ for GPU performance - Simplify advantage aggregation description in documentation - All 18 GSPO tests pass with identical results Addresses GitHub review comments areal-project#1 and areal-project#2. * Ajust code format to pass pipeline black formatter check. --------- Co-authored-by: bruceli_ed24 <bruce.li2@h-partners.com>
SathyaGnanakumar
pushed a commit
to danielkiely/AReaL
that referenced
this pull request
Apr 29, 2026
…eal-project#501) * feat: implement GSPO (Group-level Sequential Policy Optimization) Add GSPO support with sequence-level advantage computation instead of token-level. This implementation includes: - Add importance_sampling_level configuration ("token" for PPO, "sequence" for GSPO) - Modify ppo_actor_loss_fn to compute advantages at sequence level for GSPO - Add comprehensive tests for sequence-level importance sampling - Add GSPO example with GSM8K dataset configuration - Update weight_update_meta initialization and validation workflow - Merge DAPO configs for better flexibility - Update README with GSPO usage and configuration details The key difference from standard PPO is that GSPO sums token-level advantages within each sequence before computing the policy gradient, enabling more stable sequence-level optimization. * Address PR review comments Changes: - Add more assertions in unit tests for GSPO - Fix GSPO packed sequences to use cu_seqlens for per-sequence ratios - Fix GSPO gradient scaling bug by averaging instead of summing advantages - Add GSPO algorithm description and implementation notes to documentation - Add PPOActor configuration logging for reproducibility - Simplify documentation by removing detailed implementation notes - Update examples to reuse existing gsm8k_grpo.py with GSPO config - Remove dedicated gsm8k_gspo.py script (not necessary) - Remove unnecessary overlong penalty parameters from config GSPO can be enabled with any PPO-like training by setting importance_sampling_level: sequence in the configuration. * Ajust code format to pass pipeline black formatter check. * perf: vectorize GSPO 1D packed tensor computation and simplify docs - Replace Python for loop with scatter_add_ for GPU performance - Simplify advantage aggregation description in documentation - All 18 GSPO tests pass with identical results Addresses GitHub review comments #1 and areal-project#2. * Ajust code format to pass pipeline black formatter check. --------- Co-authored-by: bruceli_ed24 <bruce.li2@h-partners.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.