afmoe ring attention support. - #1848
Merged
Merged
Conversation
Signed-off-by: Matej Sirovatka <54212263+S1ro1@users.noreply.github.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
Signed-off-by: samsja <55492238+samsja@users.noreply.github.com>
S1ro1
added a commit
that referenced
this pull request
Jun 23, 2026
Picks up verifiers 49228fb4 (PR #1848): probe `uv python find --script` so images whose uv only supports `uv sync --script` also get a fresh-uv install. Applies to future launches; does not affect the running job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
S1ro1
added a commit
that referenced
this pull request
Jun 24, 2026
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mikasenghaas
added a commit
that referenced
this pull request
Jun 24, 2026
7c347f61 (#1848). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tested on an RL run, reward goes up, eval goes up, mismatch looks good. however, not a huge fan of how it's currently a special case for the afmoe model, but unsure how this could be streamlined (with gated attn and all).
Note
Medium Risk
Touches core attention execution and distributed CP ring-attention patching, so regressions could affect training correctness/perf across models (especially sliding-window/FA3 paths).
Overview
Enables context-parallel ring attention for AFMoE by refactoring FlashAttention/AfmoeFlashAttention to route kernel invocation through a
_compute_attentionhook and patching that hook to a ring-attention implementation via the newsubstitute_ring_attn.Extends the FA3 ring-attention wrapper (
ring_fa3_varlen_func) to accept and forwardwindow_size, so sliding-window/local-attention layers and full-attention layers work correctly under ring attention. RL and SFT training now callsubstitute_ring_attnwhencpis enabled, and the changelog documents the new AFMoE+CP support.Written by Cursor Bugbot for commit 7e0edf1. This will update automatically on new commits. Configure here.