Repository navigation
Conversation
This was referenced Oct 19, 2026
merceod
force-pushed
the
users/atindra/pr-10a-docs
branch
8 times, most recently
from
October 20, 2026 06:21
9697447 to
c1b855c
Compare
merceod
force-pushed
the
users/atindra/pr-10a-docs
branch
from
October 21, 2026 02:30
c1b855c to
b8114c8
Compare
merceod
force-pushed
the
users/atindra/pr-10a-docs
branch
3 times, most recently
from
October 21, 2026 10:32
24c6859 to
375bd7b
Compare
…fer's Both strategies take the CommGroup now and vote through its agree; the module helper is a thin wrapper over it, kept as the seam the tests fake.
Resource.capture_refusal(spec) is asked by both graph runners. A refused bucket is warmed up when captures are sized, so its eager peak counts toward the floor, but it is not captured to size it, so the capture plan never takes it. Every rank refuses the same buckets and skips them at capture, scope and all, as it skips an unplanned one. The runner logs one warning with the reason, and the bucket stays in dropped_buckets (or dropped_shapes), so MSTAR_REQUIRE_ACCELERATOR_GRAPHS still fails startup. CGSlotSpec.max_row_tokens is the longest row a bucket can hold.
LinearAttnBackend gains AUTO (the default) and AITER. AUTO resolves once at build: AITER on an AMD GPU, FlashInfer elsewhere. FlashInfer's calls move unchanged into FlashInferGDNKernels, and its prefill state-dtype table now asks the platform for an NVIDIA capability. On AITER every kernel works the K-last pool in place by slot: eager prefill on chunk_gated_delta_rule_opt_vk with its schedule built once per step, captured prefill buckets on the sequential kernel up to max_captured_prefill_tokens (larger ones are refused and run eagerly), and decode on FlyDSL, or the sequential kernel where FlyDSL's fixed scale or tiling does not fit. A split walk's mixed step (GDNMixedWrapper) takes both halves from the same GDNKernels, through without_qk_l2norm since the caller normalises q and k for both. On NVIDIA that makes the calls it made before; on AITER the decode rows run on FlyDSL beside the prefill kernel. A kernel that can normalise reports it from its kernels' qk_l2norm. test_gdn_mixed_split.py's FlashInfer parity runs on NVIDIA only, and test/rocm/test_gdn_aiter.py checks a mixed step on AITER.
…checkpoints AITER's GDN kernels update the pool in place and keep no state checkpoints, so a snapshot slot would hold whatever it held before. The GDN manager tells the pool, whose prefix cache then stays shut, and the engine keeps the node's caches shut with it, as it does above one rank.
The engine's process-wide 'high' matmul precision runs the oracle's fp32 einsums as XF32 on gfx942, about 1e-2 off fp64, which failed the kernel check there; the Triton kernel itself is within 4e-6 of fp64.
The pool sizes its snapshot slots before the GDN manager can say its kernels write none, so on AMD GPUs the slots sat allocated and idle: about 2.3 GiB on Qwen3.5 0.8B and 2B and 6 GiB on 4B and 9B with the fp32 state and 128 slots. refuse_snapshots now reallocates the blocks without them and hands the memory back before anything is captured. The docs say prefix caching is off there.
dynamo traces through lru_cache, so each compiled MoE call took 5 graph breaks and re-ran cpp_extension.load eagerly. This changes the compiled graph on NVIDIA too. It's a fix but not measured there yet, so it needs an NVIDIA worker_phases A/B before it goes upstream.
AUTO is Triton everywhere and uses a tuned config when one ships for the GPU. Qwen3-Omni reads moe_backend from model_kwargs. Qwen3-Omni writes the choice into its thinker and talker configs, so the trust_remote_code test's stand-in config carries both sections.
Weights get shuffled once after loading. The call is a custom op with a fake, so compile and capture see one node.
Main's expert-parallel block and padding masks dispatch with skip_invalid, and the Thinker now builds its MoE through instantiate_moe_block. Triton hands skip_invalid to fused_experts, AITER refuses it, the EP block builds its backend like the other blocks, and instantiate_moe_block passes moe_backend on.
Main's get_config tables share configs/, named without topk and with a down entry, so the check failed on its GB200 table.
Same entries. The names drop topk and give N as w1's rows (2 x the intermediate size), and the triton_version key goes (all four were tuned with Triton 3.7.1).
…ader One loader, cache and nearest-M rule, in kernels.py. The backend loads its table once when it's built, since its calls run inside compiled forwards, and hands fused_experts the (gate/up, down) pair get_config returns, so a table's down overrides apply there too. config.py and the second tuner go, and MoEProblem moves to backend.py.
On ROCm the tiles run at two stages with one to eight warps, and the best three tiles of each GEMM and BLOCK_SIZE_M are timed again across waves_per_eu, matrix_instr_nonkdim and kpack. --batch-sizes picks the token counts. The NVIDIA search is unchanged.
merceod
force-pushed
the
users/atindra/pr-10a-docs
branch
from
November 1, 2026 07:49
375bd7b to
f4863b5
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first of two docs PRs for the ROCm work. It describes the norm (#683), attention (#685), all-reduce (#684), GDN (#686) and MoE (#711) PRs, and stacks on all five. Until they merge the diff here includes them. This PR's own change is the last 7 commits.
installation.rstgets a ROCm section. The stack we tested (torch 2.12.1+rocm7.2, the AITER 0.1.25 release wheel with FlyDSL, amd-flashinfer 0.6.18+amd.1 built from source), the uv constraints that keep CUDA-only packages out (torchcodec, flash_attn, flashinfer-python) and what replaces them, AITER's JIT cache variables, cache placement, GPU selection and a short check.architecture.rstgets "Hardware platforms", which says which kernel each resource runs on NVIDIA and on ROCm, and how capture differs on ROCm.adding_models.rstcovers capture refusal,environment_variables.rstthe platform defaults.models.rstgets a status table of the models that have run on AMD GPUs, built only from runs we have results for. Also a markup fix there.A few small code edits ride along.
_fa3_unavailable_reasonbecomes_varlen_unavailable_reason, since it now covers AITER's varlen kernel too. An unused_on_cudagoes. Comments that said dense attention is FA3-only or GDN is CUDA-only are fixed (no code change in those files).NVIDIA: docs, one rename and comments.
Tests
ruff check .test_dense_attention.py,test_attn_backend_auto.py,test_aiter_attention_cpu.py,test_cosmos3_edge_model.py,test_model_registration.py,test_startup_graph_validation.pytest/modular,test/distributedandtest/sampling_teston an MI300X node with GPUs hidden, on the integration branch that carries this whole stack, rebased on mainsphinx-build -b html docswithautosummary_generate=0andapi.rstexcludedmodels.rstwarning, an unclosed**in the LTX-2.5 section, goes with the markup fix heretest/modular,test/distributedand the model test dirs, plus the TP lockstep sim; torch 2.9.1 CPU, merged with main, with the five PRs this stacks on)test/modularonly fails the twotest_dump_stacksenvironment cases main fails too (7174 passed before the last three test fixes, targeted rerun on the tip 398 passed), other CPU suites 248 passed, TP lockstep sim 7 passed,test/rust368 passed. One MI300X:test/rocm675 passed, fused-MoE and sampler GPU tests 228 passed, served smoke rows for Qwen3.5 0.8B, Kokoro, Whisper large-v3 and Cosmos3 DROID match the earlier runsNot in this PR
MSTAR_*knobs are undocumented, andadding_models.rstdocuments aCudaGraphConfig.requiredthat doesn't exist.Checks on this rebase
The stack now sits on main f7cb5ca (2026-10-29), every branch cut from one integration tree.
test/ci, the TP lockstep sim,test/rust, CPU Coretest/rust436 passed, CPU Core fails nothing main doesn't (main's own 34:test_dump_stacksx2,loop_output_ownershipx32)test/rocmon one MI325X (a VM slice, torch 2.12.1+rocm7.2)test_*moe*failures are main's GLM-5.2 kernel tests, the same set on the previous headNothing fails on this tree that the previous head doesn't: the failing-set diff against it is empty for
test/rocm, MoE, the sampler and the rows.