Repository navigation
Conversation
…y aliases
The deterministic MoE grouped GEMM kept its names from before the
deterministic-family rename as aliases. They are removed with no
replacement:
- the package flashinfer.experimental.cake_moe_grouped_gemm and its
cake_autograd, cake_backend, cake_build and cake_jit submodules;
- the _RENAMED maps and their __getattr__ branches in the det package
(CakeGroupedMm, GroupedGemmLaunch, cake_grouped_mm, grouped_gemm_*,
prepare_grouped_gemm_*), its cake_autograd (CakeGroupedMm,
cake_grouped_mm) and its cake_backend (GroupedGemmLaunch,
grouped_mm_bf16_cake, prepare_grouped_gemm_*);
- flashinfer.grouped_mm.{grouped_gemm_fwd, grouped_gemm_dgrad,
grouped_gemm_wgrad, cake_grouped_mm} and the core module's map;
- the test-helper key cake_moe_grouped_gemm.
The det-free spellings now name only the non-deterministic package
flashinfer.experimental.cake.cake_moe_grouped_gemm and its facade.
…e to cake_det The thin core module holds only the deterministic entry points (grouped_det_gemm_fwd / _dgrad / _wgrad and cake_grouped_det_mm), so its name now carries det like the other deterministic thin modules. flashinfer.grouped_mm imports from .cake_det; the old module name is removed (no alias module). flashinfer.grouped_mm.grouped_mm_bf16 and the exported names of flashinfer.grouped_mm are unchanged.
… of the det entries cake_grouped_det_mm (det package and flashinfer.grouped_mm) took a deterministic argument and silently ran the deterministic programs for deterministic=False. The deterministic entries now take no mode argument: - cake_grouped_det_mm(x, w, offs): passing deterministic raises TypeError (unexpected keyword); - CakeGroupedDetMm.apply(x, w, offs): forward accepts only the internal deferred-launch marker in the fourth position, so a stale positional apply(x, w, offs, True) or apply(x, w, offs, False) raises TypeError instead of binding the mode to the launch slot. torch.use_deterministic_algorithms(True) changes nothing for these entries. The facade's deterministic route and the det benchmark's direct apply arm call the new signatures.
…deterministic only The selectable facade flashinfer.experimental.cake.grouped_mm ran the deterministic package by default, rerouted deterministic=False to it under torch.use_deterministic_algorithms(True) with a one-time warning, composed the deterministic accumulate itself and raised NotImplementedError for the deterministic fuse_wgrad_accumulation. It now hands every call to the non-deterministic package with the same arguments; the deterministic routing, the fallback and its warning, the facade compose and the NotImplementedError route are removed (the deterministic accumulate belongs to the deterministic package). Mode rules of the package and the facade: - deterministic is keyword-only on every entry, cake_grouped_mm included; every value but the bool False (the default True, None, 0, 1, "False") raises ValueError naming the deterministic entry (grouped_det_gemm_*, prepare_grouped_det_gemm_*, cake_grouped_det_mm) and saying to pass deterministic=False for the non-deterministic path; - under torch.use_deterministic_algorithms(True), PyTorch's non-determinism alert (RuntimeError; warn_only=True: UserWarning, then the launch runs) is raised immediately before an order-varying launch only: GroupedGemmLaunch.order_varying, the weight gradient whose 512-wide k tile splits its tail (overwrite or accumulate), at launch(), in grouped_gemm_wgrad and in CakeGroupedMm.backward. The preparation, the forward, the data gradient, the 256-wide k tile and an unsplit tail run in a fixed order and never alert. _check_accumulate_call, used only by the removed facade compose, is removed from the package.
…rministic mode contract - Delete test_cake_moe_grouped_det_gemm_alias.py (it asserted the removed aliases). - Deterministic API suite: absence tests for the removed package path, its submodules and flashinfer.grouped_mm.cake (ModuleNotFoundError), every removed det-free name on the det package, its cake_autograd and cake_backend, flashinfer.grouped_mm.cake_det and flashinfer.grouped_mm (AttributeError / ImportError) and the test-helper key (KeyError); no det entry has a deterministic parameter; cake_grouped_det_mm and the det functions refuse deterministic= (TypeError); a stale positional CakeGroupedDetMm.apply(x, w, offs, True|False) raises TypeError; torch deterministic mode (strict and warn_only) changes nothing for the det entries. - Non-deterministic API suite: the facade hands every call to the package (deterministic objects are tripwires), refuses every value but the bool False with ValueError naming the det entry, has keyword-only deterministic on every entry; the alert is raised immediately before an order-varying launch only (recorded launches on CPU: strict RuntimeError with nothing launched, warn_only UserWarning naming the caller then the launch; fixed-order launches never alert; the grouped_gemm_wgrad name through the package and the facade); the det-free names are only non-deterministic objects; on a GPU, a split and an ordered weight gradient and the autograd backward. - Accumulate suite: order_varying follows the plan (meta operands at every SM count), the accumulate alert contract, the facade accumulate and fuse rows hand off to the package and refuse the default mode; the facade's deterministic compose test is removed; the unsplit autograd case runs under torch deterministic mode without an alert, and a split-tail fuse alerts before touching main_grad. - Det backend suite: call sites without deterministic; the direct apply arm refuses a stale mode argument. - Reproducibility suite: the facade fwd / dgrad (deterministic=False) give the deterministic bits.
…e the det benchmark arms - docs/api/cake_moe_grouped_det_gemm.rst and the det package README: a "Mode and dispatch rules" section (deterministic only, no deterministic argument: TypeError, also for a fourth argument of CakeGroupedDetMm.apply; torch deterministic mode changes nothing; grouped_mm_bf16(backend="cake") is the deterministic forward) and the removal of the det-free names with no alias; the deprecated-alias sections and the alias test are gone. - docs/api/cake_moe_grouped_gemm.rst and the ND package README: the non-deterministic-only rules (keyword-only deterministic, ValueError naming the det entry for every value but False, the alert only before an order-varying launch, no deterministic route or fallback), the facade's accumulate rows, examples with deterministic=False and the det / ND name table without aliases. - benchmarks/bench_cake_moe_grouped_det_gemm.py: the det arms are cake_det and cake_det_inforward (the JSON comparison key is cake_det_vs_inforward); the ND benchmark's docstring no longer names a facade deterministic route.
prepare_grouped_det_gemm_wgrad and grouped_det_gemm_wgrad (also flashinfer.grouped_mm.grouped_det_gemm_wgrad) take a keyword-only accumulate=False. accumulate=True adds the deterministic weight gradient into the caller's fp32 or bf16 out: out[e] += G[e].T @ X[e]. The launch runs the unchanged overwrite plan into a contiguous scratch of out.dtype owned by the prepared launch, then out.add_ on the same stream outside autograd (inference mode for an inference out). out then holds AccumulateGrad's bits of the deterministic weight gradient in its dtype, bitwise reproducible run to run. plan["accumulate"] is "composed" or "off"; every other plan key is the overwrite's. The checks of out (given, out_dtype None or out.dtype, bf16 or fp32, no self overlap, no byte shared with g, x or offs) run before anything is allocated or bound. sum_m == 0 launches nothing and leaves out unchanged. Every launch and every CUDA graph replay adds again.
…n into main_grad cake_grouped_det_mm (also flashinfer.grouped_mm.cake_grouped_det_mm) takes a keyword-only fuse_wgrad_accumulation=False with the Megatron / Transformer Engine convention of the non-deterministic wrapper: - w.main_grad (fp32 or bf16, w's shape and device, unit stride along K) is checked at a forward call that autograd records and read again at the backward; - the backward adds the deterministic dW into it with the package's accumulate mode (AccumulateGrad's bits of that dW, run to run bitwise) and returns a fresh placeholder when w has grad_added_to_main_grad (zeros under zero_out_wgrad), else None; - every backward adds: retain_graph twice adds twice, and so does an autograd.grad whose inputs exclude w; - create_graph, torch.jit.trace and torch.compile raise NotImplementedError; a reentrant checkpoint raises ValueError. CakeGroupedDetMm.forward takes fuse_wgrad_accumulation as its fifth argument and refuses a non-bool value (ValueError). The fourth argument stays the internal deferred-launch slot: anything but None or the internal sentinel still raises TypeError. Torch's deterministic mode changes nothing. The docstrings state the scratch and NaN-fill cost.
…nt accumulation New tests/experimental/test_cake_moe_grouped_det_gemm_acc.py. CPU (no program loaded): keyword-only switches and signatures; the accumulate plans equal the overwrite plans on the owner rows at every architecture and part SM count, and every bound program writes a contiguous launch-owned scratch; forced-form shapes with and without a split tail exist on every architecture; the out checks refuse before anything is allocated or bound; sum_m == 0; the launch adds once after the programs, outside autograd, also into an inference tensor, with no alert under torch's deterministic mode; the fuse flag's refusals, trace and compile; and the Megatron contract of the autograd wrapper with CPU stand-ins for the preparations (placeholder, zero_out_wgrad, retain_graph, autograd.grad without w, frozen w, re-pointed main_grad, create_graph, direct apply, reentrant / non-reentrant / nested checkpoints). GPU (oracle: AccumulateGrad of the deterministic dW, out0.add_(dW) in out's dtype, bitwise): every weight-gradient form with and without a split tail, fp32 and bf16, run to run over 10 launches and against the fp64 reference; owner rows; empty groups; sum_m == 0; row padding, one group, m_indptr, inference tensor; two streams; CUDA graph replays; allocations; version counter; torch's deterministic mode; and the autograd flag with the generated programs, including a bf16 main_grad against autograd's own AccumulateGrad. The det API suite also runs the accumulate and fuse entries under torch's deterministic mode.
…ation and its cost The package README and the API page of the deterministic package describe accumulate=True and fuse_wgrad_accumulation=True: the composed launch (the overwrite plan into a launch-owned scratch, then out.add_), the numerical contract (AccumulateGrad's bits of the deterministic dW in out's dtype, bitwise reproducible run to run, empty groups, sum_m == 0), the checks, prepared launches and CUDA graphs, the Megatron convention of the autograd flag, and the cost: a scratch of 4 * E * N * K bytes for an fp32 out (192 MiB to 3 GiB on the MoE projection rows), 8 * E * N * K bytes of extra traffic, and torch's NaN fill of the scratch and the placeholder under its deterministic mode. The mode-rule tables name the fifth argument of CakeGroupedDetMm.apply, and the non-deterministic API page points to the deterministic accumulation.
…ccumulation The accumulation rows of benchmarks/bench_cake_moe_grouped_gemm.py gain the arm det_mgX: cake_grouped_det_mm(..., fuse_wgrad_accumulation=True) into the fp32 main_grad with the grad_added_to_main_grad placeholder in the checkpointed step and the backward, and prepare_grouped_det_gemm_wgrad(..., accumulate=True) in the weight-gradient row. acc_ratios adds det_mgX/det_acc, det_mgX/det_mgB and mgX/det_mgX.
…en the autograd checks Both autograd functions bound the CUDA context of the backward's thread with a stream query, which a CUDA graph capture of the backward forbids. The backward now makes the capture-status query first and queries the stream only outside a capture. CakeGroupedMm.apply refuses anything but None in its internal fifth slot (TypeError), so apply(x, w, offs, False, True) no longer drops the fuse flag, and its omitted mode is True and refused like every other value but False. The non-determinism alert names the deterministic entry. The deterministic main_grad check at the forward also refuses overlapping rows or experts. Tests: the capture of the fused backward (GPU, both packages), the capture-status logic, the internal slot, the alert's entry names, the facade under warn_only, the forward main_grad layout check, bf16 forced form cases, fresh-scratch repeats, every form under torch's deterministic mode, and a per-rounding bf16 budget.
…rules The accumulate launch's add bumps out._version at launch() and at capture, not at a graph replay; its scratch lives on the preparation stream, so a launch run on another stream must stay alive until that launch completes; the backward's peak rises by 2 * S bytes only for an fp32 main_grad. The forward checks main_grad's layout, the backward also its memory sharing. The warn_only warning names the caller's line for a direct call, a frame of PyTorch's autograd for a backward; the alert names the deterministic entry. CakeGroupedMm.apply's omitted mode and internal slot are stated. The deterministic API page title names its mode, and the README tells how to drop a stale cache of the removed package.
…odules to the flashinfer-ai#8928 paths Move the MoE grouped GEMM training code to the layout of flashinfer-ai#8928: one top-level module per mode, with its backend package under flashinfer/experimental/cake/<module>/. - flashinfer/experimental/cake_moe_grouped_det_gemm/ -> flashinfer/experimental/cake/cake_moe_grouped_gemm_det_train/ - flashinfer/experimental/cake/cake_moe_grouped_gemm/ -> flashinfer/experimental/cake/cake_moe_grouped_gemm_nondet_train/ - flashinfer/grouped_mm/cake_det.py -> flashinfer/cake_moe_grouped_gemm_det_train.py - flashinfer/experimental/cake/grouped_mm.py -> flashinfer/cake_moe_grouped_gemm_nondet_train.py - the tests, benchmarks and API pages are named after the modules flashinfer.grouped_mm no longer re-exports the Cake entry points. grouped_mm_bf16(backend="cake") keeps its value and runs the moved deterministic backend. The generated sources, cake_build.py and the program tables move unchanged; only the "package" field of each build.json and the package path in the cake_jit.py messages change. The test-helper keys are the module names. Symbol names are unchanged here.
…ries by the module Rename every public object of the deterministic MoE grouped GEMM after its module (flashinfer-ai#8928 rules 2-6): - grouped_det_gemm_fwd / _dgrad / _wgrad -> cake_moe_grouped_gemm_det_train_forward / _dgrad / _wgrad - prepare_grouped_det_gemm_fwd / _dgrad / _wgrad -> cake_moe_grouped_gemm_det_train_prepare_forward / _prepare_dgrad / _prepare_wgrad - GroupedDetGemmLaunch -> CakeMoEGroupedGemmDetLaunch - cake_grouped_det_mm -> cake_moe_grouped_gemm_det - CakeGroupedDetMm -> CakeMoEGroupedGemmDetFunction flashinfer.cake_moe_grouped_gemm_det_train now carries all nine: the seven functions with the experimental-API marking (feature "cake_moe_grouped_gemm_det_train.<name>") and the two types, resolved from the backend package on first access. The weight-gradient accumulation (accumulate= / fuse_wgrad_accumulation=) is kept under the new names. The non-deterministic package's messages name the deterministic module. Backend helpers, constants, program names and the stable-API handoff keep their names.
… module and remove the deterministic keyword Rename every public object of the non-deterministic MoE grouped GEMM after its module (flashinfer-ai#8928 rules 2-6): - grouped_gemm_fwd / _dgrad / _wgrad -> cake_moe_grouped_gemm_nondet_train_forward / _dgrad / _wgrad - prepare_grouped_gemm_fwd / _dgrad / _wgrad -> cake_moe_grouped_gemm_nondet_train_prepare_forward / _prepare_dgrad / _prepare_wgrad - GroupedGemmLaunch -> CakeMoEGroupedGemmNondetLaunch - cake_grouped_mm -> cake_moe_grouped_gemm_nondet - CakeGroupedMm -> CakeMoEGroupedGemmNondetFunction The module is the mode (flashinfer-ai#8928 rule 1): no entry takes a deterministic argument any more. The keyword-only deterministic parameter, its "must be False" check and the ValueError it raised are removed; passing deterministic= is now a TypeError. CakeMoEGroupedGemmNondetFunction.apply loses its mode slot, so apply(x, w, offs, None, fuse) replaces apply(x, w, offs, False, None, fuse), and a stale fourth argument raises TypeError. flashinfer.cake_moe_grouped_gemm_nondet_train now carries all nine names: the seven functions with the experimental-API marking (feature "cake_moe_grouped_gemm_nondet_train.<name>") and the two types, resolved from the backend package on first access. The torch deterministic-mode alert is unchanged: it fires only immediately before an order-varying weight-gradient launch, warn_only warns and then runs the non-deterministic launch, and nothing falls back to the deterministic module. The alert now names the counterpart in flashinfer.cake_moe_grouped_gemm_det_train. Kernel code, program names and backend helpers are unchanged.
Add the support query of flashinfer-ai#8928 rule 5 to both MoE grouped GEMM modules: - flashinfer.cake_moe_grouped_gemm_det_train.is_cake_moe_grouped_gemm_det_train_supported - flashinfer.cake_moe_grouped_gemm_nondet_train.is_cake_moe_grouped_gemm_nondet_train_supported Each takes device=None (the current CUDA device) and is True when the device is a visible CUDA device whose architecture has a registered program for every operation of the module (forward, data gradient and weight gradient). It is False without CUDA and for a device that is not CUDA, and it imports the backend package only on a CUDA device. The query uses the backend's generated_program_available, so it loads no program and launches nothing. It is a plain function, not an experimental entry point.
Add tests/experimental/test_cake_moe_grouped_gemm_det_train_names.py and tests/experimental/test_cake_moe_grouped_gemm_nondet_train_names.py (cpu_only, nothing launches): - every former import path fails with ModuleNotFoundError (an empty namespace left by untracked files in a development checkout holds no module); - every former name raises AttributeError on the module, the backend package and its submodules, flashinfer.grouped_mm and flashinfer, "from ... import" raises ImportError, and no __all__ or dir() lists it; - the former test-helper keys raise KeyError; - the module lists the nine objects and the support query, the package the nine objects, and every name but a constant carries det / nondet; - no public callable (the methods of the public classes included) takes a deterministic argument or a catch-all **kwargs; - each object is named after itself and defined where the package resolves it from. The removed-path and removed-name tests of the deterministic API suite move into its names test, and the signature-only duplicates leave both API suites.
Per the cross-family flashinfer-ai#8928 rulings, every top-level training module is a stable FlashInfer API. The seven entry points of flashinfer.cake_moe_grouped_gemm_det_train and of flashinfer.cake_moe_grouped_gemm_nondet_train now use @flashinfer_api instead of @flashinfer_experimental_api: no experimental marking, no docstring banner and no ExperimentalWarning. Each entry keeps the signature and the docstring of its backend function and still imports the backend package only inside its body. The API suites check the stable status and that no call warns. The names tests check the mode by token (nondet is one token, not det) and that signature.bind_partial(deterministic=False) raises TypeError for every public callable, the autograd Function.forward included. grouped_mm_bf16(backend="cake") is unchanged.
…rministic arms by the mode The arms of the non-deterministic benchmark carry the mode, as the deterministic ones (cake_det, cake_det_inforward) already do: - cake -> cake_nondet, cake_inforward -> cake_nondet_inforward (--arms values, the ARMS default and the per-row JSON "arms" keys) - cake_arm / cake_inforward_arm -> cake_nondet_arm / cake_nondet_inforward_arm - the JSON key cake_vs_inforward -> cake_nondet_vs_inforward Measurements are unchanged.
…h the flashinfer-ai#8928 names Rewrite the API pages and the package READMEs of the MoE grouped GEMM for the module-per-mode layout: - docs/api/cake_moe_grouped_gemm_{det,nondet}_train.rst: the module is the mode and every name carries det / nondet; no entry takes a deterministic argument; the alert rule of the non-deterministic module; autosummaries of the seven entries and the support query of each module (the non-deterministic page gets its currentmodule); the experimental banners are gone; a removed-name table on each page. - both READMEs: the same rules, the usage examples import from the modules, a removed-name table, the test lists include the names tests, and the "git clean -fdX" note covers every former package directory.
…hinfer Add the MoE grouped GEMM rows to flashinfer._LAZY_EXPORTS, so that import flashinfer still loads no backend: - the ten names of flashinfer.cake_moe_grouped_gemm_det_train (its __all__) bind on first access to that module's objects; - flashinfer.cake_moe_grouped_gemm_nondet_train binds as a module row; its names are reached through the module only. The names tests of both modules check the rows; the two API pages say how the names are reached.
…nistic package by its new path The benchmark still imported the removed package flashinfer.experimental.cake.cake_moe_grouped_gemm, so it failed at import. Import flashinfer.experimental.cake.cake_moe_grouped_gemm_nondet_train instead; every name the benchmark reads from it already has its new name.
…infer_api is_cake_moe_grouped_gemm_det_train_supported and is_cake_moe_grouped_gemm_nondet_train_supported are stable APIs of their modules, like every other function there, so they now carry @flashinfer_api too. Their behavior is unchanged.
…strict removed paths - The deterministic names test also checks that grouped_mm_bf16 and the backend function that its backend="cake" calls, cake_backend.grouped_det_mm_bf16_cake, take no deterministic argument. - Both names tests require ModuleNotFoundError for a former import path unless a development leftover directory exists at that path; only then is an empty namespace package accepted.
…he mode it ran
The migration tables mapped flashinfer.experimental.cake.grouped_mm and
the names without det only to the non-deterministic module. Those calls
ran the deterministic package unless they passed deterministic=False:
- flashinfer.experimental.cake.grouped_mm defaulted to
deterministic=True;
- flashinfer.grouped_mm.{grouped_gemm_*, cake_grouped_mm},
flashinfer.grouped_mm.cake and the packages
flashinfer.experimental.cake_moe_grouped_det_gemm and
flashinfer.experimental.cake_moe_grouped_gemm resolved the names
without det to the deterministic objects.
Each row now names the former module and mode, and maps the call to
flashinfer.cake_moe_grouped_gemm_det_train or
flashinfer.cake_moe_grouped_gemm_nondet_train accordingly. The
non-deterministic page and README also state the one behavior change:
under torch.use_deterministic_algorithms(True), deterministic=False ran
the deterministic package with a warning, and the non-deterministic
module raises the alert instead (warn_only: warns and runs).
Also say how to import the deterministic module itself, and drop
"experimental" from the flashinfer.grouped_mm docstring: both modules
are stable APIs.
flashinfer-ai#8924 is on main: the deterministic names carry det and have no aliases, and every non-deterministic entry requires deterministic=False. Map those names to the two modules, add flashinfer.grouped_mm.cake_det and flashinfer.experimental.cake.grouped_mm to the removed modules, and state that the behavior under torch.use_deterministic_algorithms(True) does not change.
…modules Call assert_cake_mode_module of tests/test_helpers/cake_mode_names.py from the names test of each module, with flashinfer, the backend package and its cake_autograd and cake_backend modules as namespaces, every facade name with its backend object as a forward, and the former module paths and names. The family's own checks stay.
…e mode they ran The removed-name tables listed the former names as they were on main only. Add the names of the v0.7.3 release (the deterministic package flashinfer.experimental.cake_moe_grouped_gemm, flashinfer.grouped_mm.cake and the grouped_gemm_* / cake_grouped_mm names) and of the v0.7.4rc1 prerelease, where flashinfer.experimental.cake.grouped_mm ran the deterministic package by default and the non-deterministic package only with deterministic=False. Those default calls now move to cake_moe_grouped_gemm_det_train and the deterministic=False calls to cake_moe_grouped_gemm_nondet_train. State the behavior change under torch.use_deterministic_algorithms(True): in v0.7.4rc1 deterministic=False fell back to the deterministic package with a warning; the non-deterministic module alerts right before an order-varying weight-gradient launch, runs it with warn_only=True and never falls back.
…radient from the package The accumulate tests composed their reference from a name of the base test module that the rename removed; take it from the package.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Collaborator
Author
|
@flashinfer-bot run tests/experimental/test_cake_moe_grouped_gemm_det_train.py tests/experimental/test_cake_moe_grouped_gemm_det_train_acc.py tests/experimental/test_cake_moe_grouped_gemm_det_train_api.py tests/experimental/test_cake_moe_grouped_gemm_det_train_determinism.py tests/experimental/test_cake_moe_grouped_gemm_det_train_names.py tests/experimental/test_cake_moe_grouped_gemm_det_train_standalone.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_acc.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_api.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_names.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_reproducibility.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_standalone.py tests/grouped_mm/test_grouped_mm_bf16.py tests/experimental/test_containment.py tests/cpu/test_flashinfer_lazy_exports.py tests/cpu/test_cake_packages_standalone.py |
origin/main 1f6053b brings flashinfer-ai#8957, which covers the mode-word backend csrc trees in the shared pre-commit exclude. One conflict in .pre-commit-config.yaml, resolved to main's file: its generic cake_moe_grouped_gemm_(det|nondet)_train/csrc/ entry covers both moved packages, so this branch no longer edits that file.
Collaborator
Author
|
@flashinfer-bot run tests/experimental/test_cake_moe_grouped_gemm_det_train.py tests/experimental/test_cake_moe_grouped_gemm_det_train_acc.py tests/experimental/test_cake_moe_grouped_gemm_det_train_api.py tests/experimental/test_cake_moe_grouped_gemm_det_train_determinism.py tests/experimental/test_cake_moe_grouped_gemm_det_train_names.py tests/experimental/test_cake_moe_grouped_gemm_det_train_standalone.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_acc.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_api.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_names.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_reproducibility.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_standalone.py tests/grouped_mm/test_grouped_mm_bf16.py tests/experimental/test_containment.py tests/cpu/test_flashinfer_lazy_exports.py tests/cpu/test_cake_packages_standalone.py |
yyihuang
pushed a commit
to yyihuang/flashinfer
that referenced
this pull request
Nov 6, 2026
…se note to flashinfer-ai#8963 Every MoE module, name and backend directory now names flashinfer-ai#8963, the PR that renames the family. The MoE table lists all old public names of 0.7.3 and 0.7.4rc1, including the forward/dgrad/wgrad entries, the prepare functions, the launch classes and grouped_mm_bf16_cake. The deterministic= table gets the MoE exception: the old det entries ran the det programs for every deterministic value.
Collaborator
Author
|
@flashinfer-bot run tests/experimental/test_cake_moe_grouped_gemm_det_train.py tests/experimental/test_cake_moe_grouped_gemm_det_train_acc.py tests/experimental/test_cake_moe_grouped_gemm_det_train_api.py tests/experimental/test_cake_moe_grouped_gemm_det_train_determinism.py tests/experimental/test_cake_moe_grouped_gemm_det_train_names.py tests/experimental/test_cake_moe_grouped_gemm_det_train_standalone.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_acc.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_api.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_names.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_reproducibility.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_standalone.py tests/grouped_mm/test_grouped_mm_bf16.py tests/experimental/test_containment.py tests/cpu/test_flashinfer_lazy_exports.py tests/cpu/test_cake_packages_standalone.py tests/cpu/test_cake_mode_names.py |
Collaborator
|
No CI pipeline was started: not a valid test scope -- too many targets (17); maximum is 16 (scripts/test_sharding/runner.py enforces the same cap, but only after the lanes have started). Specify test paths with |
Collaborator
Author
|
@flashinfer-bot run tests/experimental/test_cake_moe_grouped_gemm_det_train.py tests/experimental/test_cake_moe_grouped_gemm_det_train_acc.py tests/experimental/test_cake_moe_grouped_gemm_det_train_api.py tests/experimental/test_cake_moe_grouped_gemm_det_train_determinism.py tests/experimental/test_cake_moe_grouped_gemm_det_train_names.py tests/experimental/test_cake_moe_grouped_gemm_det_train_standalone.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_acc.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_api.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_names.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_reproducibility.py tests/experimental/test_cake_moe_grouped_gemm_nondet_train_standalone.py tests/grouped_mm/test_grouped_mm_bf16.py tests/experimental/test_containment.py tests/cpu/test_flashinfer_lazy_exports.py tests/cpu/test_cake_packages_standalone.py |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migration
This PR names the Cake MoE grouped GEMM by its mode, as #8928 requires. It follows #8924. The module selects the mode:
flashinfer.cake_moe_grouped_gemm_det_trainorflashinfer.cake_moe_grouped_gemm_nondet_train. No entry takesdeterministic=. The former names were removed with no alias; importing one raisesModuleNotFoundError,ImportErrororAttributeError.Each former call maps to the module of the mode that it ran. The parentheses give the versions that had each former name: the v0.7.3 release, the v0.7.4rc1 prerelease and
mainbefore this PR. "A former det package" isflashinfer.experimental.cake_moe_grouped_gemm(v0.7.3; in v0.7.4rc1 an alias of the next one) orflashinfer.experimental.cake_moe_grouped_det_gemm(v0.7.4rc1,main). "The former nondet package" isflashinfer.experimental.cake.cake_moe_grouped_gemm(v0.7.4rc1,main), one level lower; every entry of it ran only withdeterministic=False. The facadeflashinfer.experimental.cake.grouped_mmselected the mode: in v0.7.4rc1 its defaultdeterministic=Trueran the det package; onmainonlydeterministic=Falseran.flashinfer.grouped_mm.cake(v0.7.3, v0.7.4rc1) orflashinfer.grouped_mm.cake_det(main)flashinfer.cake_moe_grouped_gemm_det_train; backendflashinfer.experimental.cake.cake_moe_grouped_gemm_det_train(same submodule names)grouped_gemm_fwd/_dgrad/_wgrad(v0.7.3, v0.7.4rc1) andgrouped_det_gemm_fwd/_dgrad/_wgrad(v0.7.4rc1,main), fromflashinfer.grouped_mm, its module above or a former det packagecake_moe_grouped_gemm_det_train_forward/_dgrad/_wgradflashinfer.experimental.cake.grouped_mm.grouped_gemm_fwd/_dgrad/_wgradwith the default ordeterministic=True(v0.7.4rc1)cake_moe_grouped_gemm_det_train_forward/_dgrad/_wgradprepare_grouped_gemm_fwd/_dgrad/_wgrad(v0.7.3, v0.7.4rc1) andprepare_grouped_det_gemm_fwd/_dgrad/_wgrad(v0.7.4rc1,main), from a former det packagecake_moe_grouped_gemm_det_train_prepare_forward/_prepare_dgrad/_prepare_wgradGroupedGemmLaunch(v0.7.3, v0.7.4rc1) andGroupedDetGemmLaunch(v0.7.4rc1,main), from a former det packageCakeMoEGroupedGemmDetLaunchcake_grouped_mm(v0.7.3, v0.7.4rc1) andcake_grouped_det_mm(v0.7.4rc1,main), fromflashinfer.grouped_mm, its module above or a former det packagecake_moe_grouped_gemm_detflashinfer.experimental.cake.grouped_mm.cake_grouped_mmwith the default ordeterministic=True(v0.7.4rc1)cake_moe_grouped_gemm_detCakeGroupedMm(v0.7.3, v0.7.4rc1) andCakeGroupedDetMm(v0.7.4rc1,main), from a former det packageCakeMoEGroupedGemmDetFunctioncake_backend.grouped_mm_bf16_cakeof a former det package (v0.7.3, v0.7.4rc1)flashinfer.grouped_mm.grouped_mm_bf16(..., backend="cake")flashinfer.cake_moe_grouped_gemm_nondet_train; backendflashinfer.experimental.cake.cake_moe_grouped_gemm_nondet_train(same submodule names)grouped_gemm_fwd/_dgrad/_wgradwithdeterministic=False, fromflashinfer.experimental.cake.grouped_mmor the former nondet package (v0.7.4rc1,main)cake_moe_grouped_gemm_nondet_train_forward/_dgrad/_wgradprepare_grouped_gemm_fwd/_dgrad/_wgradwithdeterministic=False, from the former nondet packagecake_moe_grouped_gemm_nondet_train_prepare_forward/_prepare_dgrad/_prepare_wgradGroupedGemmLaunch, from the former nondet packageCakeMoEGroupedGemmNondetLaunchcake_grouped_mmwithdeterministic=False, fromflashinfer.experimental.cake.grouped_mmor the former nondet package (v0.7.4rc1,main)cake_moe_grouped_gemm_nondetCakeGroupedMm, from the former nondet packageCakeMoEGroupedGemmNondetFunctiondeterministicargument: the mode offlashinfer.experimental.cake.grouped_mm(v0.7.4rc1);Falseonly for the nondet entries; any value forcake_grouped_mm/cake_grouped_det_mmand the fourth argument ofCakeGroupedMm.apply/CakeGroupedDetMm.applyof a former det package (v0.7.3, v0.7.4rc1)TypeErroris_cake_moe_grouped_gemm_det_train_supported,is_cake_moe_grouped_gemm_nondet_train_supportedflashinfer.grouped_mm.grouped_mm_bf16(..., backend="cake")Behavior under
torch.use_deterministic_algorithms(True):flashinfer.cake_moe_grouped_gemm_nondet_trainraises PyTorch's non-determinism alert (RuntimeError) right before an order-varying weight-gradient launch; withwarn_only=Trueit warns and runs the non-deterministic launch; it never falls back to the deterministic module.flashinfer.cake_moe_grouped_gemm_det_trainnever alerts. This changes the v0.7.4rc1 behavior: there,flashinfer.experimental.cake.grouped_mmwithdeterministic=Falseran the det package with a one-timeUserWarning, and the former nondet package alerted at every weight-gradient call, its preparation included.mainbefore this PR already behaved as the new module does.Top-level names:
flashinfer.<name>binds the 10 names offlashinfer.cake_moe_grouped_gemm_det_trainon first access, andflashinfer.cake_moe_grouped_gemm_nondet_trainbinds the non-deterministic module. To use the deterministic module object itself,import flashinfer.cake_moe_grouped_gemm_det_train.Backend directories (vendoring builds; each moved whole with
build.json,cake_build.pyandcsrc/):flashinfer/experimental/cake_moe_grouped_gemm/(det; v0.7.3 sources, an alias with no sources in v0.7.4rc1)flashinfer/experimental/cake/cake_moe_grouped_gemm_det_train/flashinfer/experimental/cake_moe_grouped_det_gemm/(det; v0.7.4rc1,main)flashinfer/experimental/cake/cake_moe_grouped_gemm_det_train/flashinfer/experimental/cake/cake_moe_grouped_gemm/(nondet; v0.7.4rc1,main)flashinfer/experimental/cake/cake_moe_grouped_gemm_nondet_train/The former name
cake_moe_grouped_gemmmeant det one level belowexperimental/, but nondet underexperimental/cake/.READY grep allowlist. A whole-word and dotted-path grep of every former name above, over
flashinfer/ tests/ benchmarks/ docs/ scripts/ .github/after mergingmain, finds no live use. It finds only these:csrc/**andbuild.jsonof both packages, and the route, program and record literals of theircake_jit.py: kept build identities (cake_moe_grouped_det_gemm_*,cake_moe_grouped_gemm_*).git clean -fdXnotes indocs/api/cake_moe_grouped_gemm_{det,nondet}_train.rstand in both package READMEs.tests/experimental/test_cake_moe_grouped_gemm_{det,nondet}_train_names.py.grouped_mm_bf16_cakeintests/experimental/test_cake_moe_grouped_gemm_nondet_train_{api,standalone}.py, and the route and program literals intests/experimental/test_cake_moe_grouped_gemm_nondet_train_reproducibility.py._DET_ENTRIESintests/norm/test_cake_rmsnorm_train_api.py,flashinfer/cake_gdn_train_kcp.py,flashinfer/experimental/cake/cake_gdn_train/cake_chain.pyandflashinfer/experimental/cake_gdn_train_kcp/cake_kcp.py: names of other families.docs/training_backend_names.md.Performance
The kernels are unchanged (moved byte-identical). The two-baseline table per architecture (B200 sm_100a, GB300 sm_103a, R200 sm_107a) follows when the final-head GPU runs finish.
Validation
Pending: final-head GPU tests and memcheck on B200, GB300 and R200.