Skip to content

docker: use FA3 3.0.0 for transformer_engine 2.17 - #1799

Merged
Zhichenzzz merged 1 commit into
mainfrom
fa3-3.0.0-te-2.17
Jul 27, 2026
Merged

docker: use FA3 3.0.0 for transformer_engine 2.17#1799
Zhichenzzz merged 1 commit into
mainfrom
fa3-3.0.0-te-2.17

Conversation

@yueming-yuan

Copy link
Copy Markdown
Collaborator

Problem

#1781 bumped transformer_engine to 2.17.0. TE 2.17 moved use_flash_attn_3 out
of the window_size-tuple branch in cp_p2p_fwd_flash_attn /
cp_p2p_bwd_flash_attn:

# TE 2.12
if use_flash_attn_3 or (v2_3_plus and not v2_7_0_plus):
    fa_forward_kwargs["window_size"] = (-1, -1)        # FA3 took the tuple
elif v2_7_0_plus:
    fa_forward_kwargs["window_size_left"] = -1

# TE 2.17
if v2_3_plus and not v2_7_0_plus:
    fa_forward_kwargs["window_size"] = (-1, -1)
elif use_flash_attn_3 or v2_7_0_plus:                  # FA3 moved here
    fa_forward_kwargs["window_size_left"] = -1

The flash_attn_3 wheel we ship was 3.0.0b1, whose _flash_attn_forward only
accepts a single window_size tuple, so every context-parallel path raises
TypeError: got an unexpected keyword argument 'window_size_left'. There are
eight such call sites in context_parallel.py (five forward, three backward).

TE has no FA3-only kill switch — NVTE_FLASH_ATTN disables FA2 and FA3
together. FA2 in the same image (2.7.4.post1) already takes the new kwargs, so
only the FA3 path is affected.

Why this never broke before

TE's version gate is a silent capability gate that also guards the import. Under
TE 2.12 the tuple was correct, so nothing failed. The 2.17 bump is the first
time this pairing is exercised — and main's scheduled docker builds have been
failing since #1781 landed, so :dev still carries TE 2.12 and main looks green.

Change

yueming-yuan/miles-wheels#10 rebuilt the wheel at flash-attention 00756db
(version 3.0.0) and replaced the cu130-x86_64 release asset, so the existing
pip install /tmp/wheels/flash_attn_3-*.whl already resolves to it.

This PR only drops the separate flash_attn_interface.py fetch. That fetch was a
second revision pin that can drift from the one the extension module was built
at. It is now both unnecessary and harmful:

  • the wheel ships flash_attn_interface top-level, which is what TE 2.17 imports;
  • it also ships flash_attn_3.flash_attn_interface as a re-export shim;
  • overwriting that shim with a copy of the full module re-runs
    @torch.library.custom_op("flash_attn_3::_flash_attn_forward") and
    double-registers the ops.

Nothing in miles or sglang imports flash_attn_3.flash_attn_interface.

Verification

Built and installed on an h200-sci devbox running the miles image itself
(torch 2.11.0+cu130, CUDA 13.0), so the wheel matches the toolchain it ships with:

[INFO     | DotProductAttention]: Running with FlashAttention backend (version 3.0.0)
[PASS] all 5 symbols TE imports exist
[PASS] _flash_attn_forward schema takes window_size_left/right  ['SymInt window_size_left=-1', 'SymInt window_size_right=-1']
[PASS] _flash_attn_backward schema takes window_size_left/right  ['SymInt window_size_left=-1', 'SymInt window_size_right=-1']
[PASS] import transformer_engine.pytorch
[PASS] TE sees flash-attn-3 == 3.0.0  fa3_version=3.0.0 v3_0_0_beta=False
[PASS] fwd accepts window_size_left/right kwargs  out=(2, 512, 8, 128) lse=(2, 8, 512)
[PASS] returns (out, softmax_lse, ...) in the order TE indexes [0]/[1]
[PASS] TE DotProductAttention fwd+bwd on FlashAttention backend  grad_finite=True
[PASS] output matches SDPA reference  max_abs_err=0.001953
RESULT: ALL PASS

Checked so it is not rediscovered later:

  • Arch coverage is unchanged: sm_80 + sm_90a, confirmed with cuobjdump
    against both the old and new _C.abi3.so. hopper/setup.py derives arch from
    source-file suffixes and ignores TORCH_CUDA_ARCH_LIST, so FA3 has never had
    Blackwell kernels here.
  • 3.0.0b1 -> 3.0.0 is benign. It flips FlashAttentionUtils.v3_0_0_beta
    to False; its only consumer appends an "update your flash-attn v3 beta" hint
    to an already-raised TypeError. No functional branch depends on it.
  • Return shape is compatible. 3.0.0 returns
    (out, softmax_lse, out_accum, softmax_lse_accum); TE reads [0] and [1]
    under use_flash_attn_3.

Full CP coverage runs in #1795, which cherry-picks this change.

TE 2.17 moved use_flash_attn_3 out of the window_size-tuple branch in
cp_p2p_fwd/bwd_flash_attn, so it passes window_size_left/right to FA3. The
flash_attn_3 wheel in miles-wheels was 3.0.0b1, whose _flash_attn_forward
only accepts a window_size tuple, so every context-parallel path raises
TypeError. TE has no FA3-only kill switch -- NVTE_FLASH_ATTN disables FA2 too.

miles-wheels#10 rebuilt the wheel at flash-attention 00756db (3.0.0) and
replaced the release asset, so the pip install here already picks it up. That
wheel ships flash_attn_interface top-level and a flash_attn_3 re-export shim,
making the separate interface fetch unnecessary -- and harmful, since it
pinned a second revision that can drift from the one the extension module was
built at, and overwriting the shim with the full module double-registers the
flash_attn_3:: custom ops.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@Zhichenzzz
Zhichenzzz merged commit 4722e0d into main Jul 27, 2026
40 checks passed
@Zhichenzzz
Zhichenzzz deleted the fa3-3.0.0-te-2.17 branch July 27, 2026 01:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants