Skip to content

afmoe ring attention support. - #1848

Merged
samsja merged 5 commits into
PrimeIntellect-ai:mainfrom
sapiosaturn:afmoe-ring-attn
Feb 24, 2026
Merged

samsja merged 5 commits into
PrimeIntellect-ai:mainfrom
sapiosaturn:afmoe-ring-attn

Conversation

@sapiosaturn

@sapiosaturn sapiosaturn commented Feb 22, 2026

Copy link
Copy Markdown
Contributor

tested on an RL run, reward goes up, eval goes up, mismatch looks good. however, not a huge fan of how it's currently a special case for the afmoe model, but unsure how this could be streamlined (with gated attn and all).


Note

Medium Risk
Touches core attention execution and distributed CP ring-attention patching, so regressions could affect training correctness/perf across models (especially sliding-window/FA3 paths).

Overview
Enables context-parallel ring attention for AFMoE by refactoring FlashAttention/AfmoeFlashAttention to route kernel invocation through a _compute_attention hook and patching that hook to a ring-attention implementation via the new substitute_ring_attn.

Extends the FA3 ring-attention wrapper (ring_fa3_varlen_func) to accept and forward window_size, so sliding-window/local-attention layers and full-attention layers work correctly under ring attention. RL and SFT training now call substitute_ring_attn when cp is enabled, and the changelog documents the new AFMoE+CP support.

Written by Cursor Bugbot for commit 7e0edf1. This will update automatically on new commits. Configure here.

sapiosaturn and others added 4 commits February 21, 2026 16:06
Signed-off-by: Matej Sirovatka <54212263+S1ro1@users.noreply.github.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

Comment thread src/prime_rl/trainer/models/layers/attn.py
Signed-off-by: samsja <55492238+samsja@users.noreply.github.com>
@samsja
samsja merged commit db50929 into PrimeIntellect-ai:main Feb 24, 2026
8 of 9 checks passed
S1ro1 added a commit that referenced this pull request Jun 23, 2026
Temporary pin to the fix/ensure-uv-robust-old-images branch (verifiers PR #1848)
so the bash harness can bootstrap uv in the scaleswe sandboxes. Re-pin to a main
commit once #1848 merges.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
S1ro1 added a commit that referenced this pull request Jun 23, 2026
Picks up verifiers 49228fb4 (PR #1848): probe `uv python find --script` so images
whose uv only supports `uv sync --script` also get a fresh-uv install. Applies to
future launches; does not affect the running job.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
S1ro1 added a commit that referenced this pull request Jun 24, 2026
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mikasenghaas added a commit that referenced this pull request Jun 24, 2026
7c347f61 (#1848).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants