ci(build): bump FA4, quack and ffpa-attn

[akoumpa]

What

Bumps the pinned flash-attention ref used to build the FA3 (Hopper) and FA4 (Blackwell CuTe) wheels:

ARG FLASH_ATTN_REF=002cce0a1   ->   ce088ab9ce0f

and the FA4 CUTLASS DSL install nvidia-cutlass-dsl[cu13]==4.6.0.dev0 -> ==4.6.2.

It also enables FA4 by default (INSTALL_FA4 false -> true) and moves the Python dependency
set onto that same CUTLASS DSL, so everything in the image agrees on one cutlass:

Package	Before	After	Where
nvidia-cutlass-dsl	4.6.0.dev0 / 4.6.0	4.6.2	docker/Dockerfile, pyproject.toml (ffpa extra)
quack-kernels	0.6.1	0.6.4	pyproject.toml (core dependency)
ffpa-attn	>=0.2.2	>=0.2.3	pyproject.toml (ffpa extra)

uv.lock and docker/common/uv-pytorch.lock were regenerated with uv 0.8.22 (the version CI pins).
The lock diff is confined to those three packages plus the four nvidia-cutlass-dsl-libs-* splits
they pull -- seven in total, no marker churn.

Why

The old pin 002cce0a1 dates from 2026-07-03 and predates the CuTe compile-key fixes. On that ref, any call with a tensor max_seqlen — i.e. every varlen / packed-sequence step — rebuilds the compile key, so the kernel recompiles on every call instead of hitting the compile cache.

This was reported externally against a Qwen3.5-VL packed-sequence full finetune on B300 (examples/vlm_finetune/qwen3_5/qwen3_5_4b_neat_packing.yaml, pack_size 16384), where it shows up as per-step recompilation stalls.

Commits gained over the old pin:

PR	Title	Why it matters
#2507	[CuTe, Bwd] Fix backward compile key churn due to pickling, max_seqlen is a tensor	Backward half of the same bug — needed for training, not just inference
#2762	[CuTe, Fwd] Stabilize tensor max_seqlen compile key	The originally reported fix
#2745	[CuTe] Fix forward dynamic-shape correctness	Correctness fix landing right after #2762; stopping at #2762 would leave it out
#2819	[CuTe] Speed up scalar SM100 mask compilation	Further cuts Blackwell compile time

29 upstream commits / 67 files total in the range.

CUTLASS DSL pin

#2798 raised flash_attn/cute's requirement from nvidia-cutlass-dsl==4.6.0.dev0 to >=4.6.2, so the Dockerfile's DSL install has to move with the ref or FA4 runs against a DSL older than it declares. Pinned exactly at 4.6.2 (the declared floor, and [cu13] is published for it) rather than left floating, to keep the image reproducible.

The Dockerfile alone was not enough, because quack-kernels and ffpa-attn import the same
cutlass package and pinned it independently. quack-kernels 0.6.1 pinned 4.6.0; ffpa-attn
0.2.2 hard-pinned 4.6.0 and quack-kernels==0.6.1. Both had to move for the resolution to
admit 4.6.2. ffpa-attn 0.2.3 is the first release that pins nvidia-cutlass-dsl==4.6.2 and
quack-kernels==0.6.4 -- exactly what FA4 needs -- so the conflict resolves on its own and no
override-dependencies entries are required. Do not relax the >=0.2.3 floor.

Note that ffpa-attn has no runtime surface in CI: the image builds with AUTOMODEL_INSTALL=all,
and the all extra does not include ffpa. That bump only affects resolution.

Blast radius

FLASH_ATTN_REF feeds both the FA3 and FA4 builds, and INSTALL_FA3=true is the default on x86 — so this changes the default x86 image, not only INSTALL_FA4=true builds. That is the main thing CI needs to confirm. The range touches Hopper (#2746 removes SM100 functions from hopper, #2756 block-sparse bwd wait on SM90), which is why this wants a full pipeline rather than a smoke test.

This PR does change which image gets FA4: INSTALL_FA4 flips false -> true. The ARG is
declared in two stages and Docker does not inherit ARGs across stages, so both were flipped --
wheel_builder (gates building the flash_attn/cute wheel) and automodel_final (gates the
nvidia-cutlass-dsl[cu13] install and the flash_attn/cute symlink). Flipping only the first
would leave FA4 half-installed: the wheel present, the DSL and symlink missing.

That means the default image now ships FA4 on both arches. arm64 skips FA3 (TARGETARCH
guard) but not FA4, so the arm build now runs the nvidia-cutlass-dsl[cu13]==4.6.2 install --
manylinux_2_28_aarch64 wheels are published for it, for both libs-cu13 and libs-base.

Both FA4 wheels install with --no-deps (pip wheel --no-deps, then pip install --no-deps), so
flash-attn-4's declared apache-tvm-ffi>=0.1.12,<0.2 is never resolved and the <=0.1.11 cap that
keeps the tilelang kernels working still holds, so FA4 and tilelang can share one image.

FA4 does import tvm_ffi (flash_attn/cute/cache_utils.py, eagerly via flash_attn.cute.__init__),
so it runs against an ffi older than it declares. The two symbols it references,
tvm_ffi.Function and tvm_ffi.__version__, both exist in 0.1.11. This is unexercised here --
nothing selects flash_attention_4 yet -- and worth knowing because the Dockerfile sets
FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1, which enables the AOT cache path in that same file.

Fallback

If the range proves too wide, the conservative stopping point is 69e1bcbe7 (2026-08-07) — keeps #2507 + #2762 + #2745 and needs no CUTLASS DSL change, giving up only #2819.

Testing

All runs are against cd642d497 (FA4 enabled). Each pipeline builds a fresh container from this
branch, so the regenerated uv.lock is exercised end to end. Each eos scope was run twice on the
same commit, so a recipe that fails in one and passes in the other is noise rather than a regression.

Cluster	Scope	Result	Pipelines
eos (H100)	performance, folder benchmark	5/5 pass, twice	65241834, 65243991
eos (H100)	nightly, all folders	57 pass / 7 fail; 55 pass / 9 fail	65241836, 65243992
oci_hsg (GB200, arm)	performance, folder benchmark	qwen2_5_7b_quack + _baseline pass on SM100	65271508
eos (H100)	main control on the real failures	see below	65263128

GitHub Actions "CICD NeMo": success on cd642d497.

Container builds are green on both arches: uv sync --locked installs the regenerated lock, the
FA3 source build against the new FLASH_ATTN_REF compiles (x86), and the FA4 wheel plus
nvidia-cutlass-dsl[cu13]==4.6.2 install on x86 and arm.

Why Blackwell was required

FA4's CuTe kernels are SM100-targeted and are never reached on H100, and cutlass-dsl / quack dispatch
to different codegen per arch (gemm_sm90 vs gemm_sm100). An H100-only run would leave the point of
enabling FA4 untested. The arm image is the sharper test: arm64 skips FA3 entirely, so FA4 is the only
flash backend present there.

Failure accounting

No failure in any FA4-on run is attributable to this PR. Every one resolves to:

• Pre-existing on main — nemotron_nano_9b_squad, nemotron_nano_9b_squad_peft,
  nemotron_super_v3_hellaswag_peft all fail on main too (control 65263128), with
  CHECKPOINT_ROBUSTNESS_PHASE_FAILURE.
• Flakes — gpt_oss_20b and nemotron_flash_1b_squad each fail once and pass on main plus the
  duplicate run.
• Infrastructure — pyxis container-import failures (curl: (5) Could not resolve proxy, ~65 s
  walltime) and one runner-workspace fault (getcwd: cannot access parent directories). Both also
  occur on the FA4-off runs.
• Downstream plumbing — *_vllm_deploy jobs fail on a checkpoint their upstream test never wrote.

The three GB200 failures are environmental, not kernel: qwen3_moe_30b_quack_deepep hits
non_pp_size=4 must be a multiple of ep_size=8 (the recipe hardcodes ep_size: 8; GB200 nodes have
4 GPUs), and both llama3_1_8b_quack_rope variants fail identically on a cold HF cache
(Couldn't instantiate the backend tokenizer) — the quack variant and its baseline failing the same
way rules out anything quack-specific.

automodel_{llm,vlm}_benchmark_tests reporting failed on the nightly-scope runs is the known
empty-child case (the benchmark folders ship no nightly_recipes.yml); it reproduces on main.

Known gap

No test on this branch executes an FA4 kernel. Nothing selects attn_implementation="flash_attention_4"
until the FA4 backend lands in #3734. The INSTALL_FA4=true flip is therefore validated as builds,
installs and breaks nothing — on both arches — rather than as FA4 produces correct results. The
latter is covered by #3734's tests/functional_tests/attention/test_fa4_packed_parity.py.

nemo-ci fixes required to reach GB200

Two nemo-ci bugs blocked Blackwell entirely and are fixed in separate MRs:

• nemo-ci!2863 — oci_hsg never set GRES, so sbatch rejected every submission
  (Cannot find GPU specification).
• nemo-ci!2864 — NPROC_PER_NODE came from a hardcoded cluster list missing oci_hsg and hecate,
  so torchrun spawned 8 ranks onto 4-GPU nodes (CUDA error: invalid device ordinal).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[akoumpa]

/ok to test 325d1ed

[akoumpa]

Runtime validation update:

• GitHub install/container checks are green at the exact head 325d1edf2af861b91aaefc103dc7843005e00432.
• A quick GB200 bootstrap smoke installed the ce088ab9ce0f... FA4 wheel into the existing image, but that image's venv still resolves the old CUTLASS DSL 4.6.0 stack. As expected for that mismatched injection, the first FA4 vision call rejects tensor max_seqlen_q at the CUTLASS JIT boundary.
• That smoke is not evidence against upstream cp: fix(mistral3): preserve medium VLM checkpoint layout (2758) into r0.5.0 #2762 or the new pin. It confirms that the source ref and CUTLASS DSL must move as one dependency unit, which this PR's INSTALL_FA4=true image does (nvidia-cutlass-dsl==4.6.2). The proper image-build pipelines are still waiting for build resources.

The separate native text-path/config work is stacked in #3734.

[akoumpa]

/ok to test 9fe6cfd

[akoumpa]

all tests passed in https://github.com/NVIDIA-NeMo/Automodel/actions/runs/33269762967/job/99153796598?pr=3733 ; waiting for nemoci

[akoumpa]

/ok to test cd642d4

[akoumpa]

(i'll fm since the last commit is just a comment change).