Enable TE custom quantization recipe by negvet · Pull Request #2005 · NVIDIA/Megatron-LM

negvet · 2025-10-28T17:14:19Z

What does this PR do ?

This PR enables a custom quantization recipe via TE.
A custom recipe can be anything: fp8, fp4, mixture, etc.
Added custom to both fp8_utils and fp4_utils.
Custom recipe enabled in TE: NVIDIA/TransformerEngine#2039 [Merged]
Custom NVFP4 recipe in TE: NVIDIA/TransformerEngine#2259 [Merged]
Quantization (and custom quantization) refactor in TE: NVIDIA/TransformerEngine#2276 [Merged]

Contribution process

flowchart LR
    A[Pre-checks] --> B[PR Tests]
    subgraph Code Review/Approval
        C1[Expert Review] --> C2[Final Review]
    end
    B --> C1
    C2 --> D[Merge]

Pre-checks

I want this PR in a versioned release and have added the appropriate Milestone (e.g., Core 0.8)
I have added relevant unit tests
I have added relevant functional tests
I have added proper typing to my code Typing guidelines
I have added relevant documentation
I have run the autoformatter.sh on my PR

Code review

The following process is enforced via the CODEOWNERS file for changes into megatron/core. For changes outside of megatron/core, it is up to the PR author whether or not to tag the Final Reviewer team.

For MRs into `main` branch

(Step 1): Add PR label `Expert Review`

(Step 2): Collect the expert reviewers reviews

Attach the Expert Review label when your PR is ready for review.
GitHub auto-assigns expert reviewers based on your changes. They will get notified and pick up your PR soon.

⚠️ Only proceed to the next step once all reviewers have approved, merge-conflict are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

(Step 3): Final Review

Add Final Review label
GitHub auto-assigns final reviewers based on your changes. They will get notified and pick up your PR soon.

(Optional Step 4): Cherry-pick into release branch

If this PR also needs to be merged into core_r* release branches, after this PR has been merged, select Cherry-pick to open a new PR into the release branch.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

Merging your PR

Any member of core-adlr and core-nemo will be able to merge your PR.

Signed-off-by: Evgeny <etsykunov@nvidia.com>

copy-pr-bot · 2025-10-28T17:14:23Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

ChenhanYu · 2025-10-29T21:35:08Z

+    return fn
+
+
+def _get_custom_recipe(quantizer_factory_python_path: str) -> Union[Fp8Recipe, Fp4Recipe]:


All these seem more reasonable to be put in TE. Not sure why the recipe and import checking need to be in mcore.

mcore owns cli/config, flag parsing, UX errors etc.
_get_custom_recipe is an orchestration kind of thing.
That is why I put it at the mcore side.
Import path resolution is similar, because import management fits mcore well.
TE just owns abstractions (CustomRecipe etc.), being at a lower level of the stack.

With that being said, I do not mind having a minor narrow helper utility function transformer_engine.utils.resolve_path(path) in TE that can do some basic path resolution for the quantizer factory.
But from my perspective, this is not strictly necessary (or can be added later), although I am neutral.

ko3n1g · 2025-10-30T19:11:17Z

/ok to test 36dbfbf

copy-pr-bot · 2025-10-30T19:11:20Z

/ok to test 36dbfbf

@ko3n1g, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

ko3n1g · 2025-10-30T19:11:35Z

/ok to test bd21b45

Signed-off-by: Evgeny <etsykunov@nvidia.com>

ananthsub · 2025-11-06T17:52:06Z

+                if not config.fp8_quantizer_factory:
+                    raise ValueError(
+                        "Python import path, e.g. package.module.quantizer_factory, "
+                        "must be provided via --fp8-quantizer-factory when "
+                        "--fp8-recipe custom is selected."
+                    )


why not add this validation in the transformer config __post_init__ ?

Sure, fixed in 2245c22

Signed-off-by: root <Evgeny>

ananthsub · 2025-11-10T10:28:44Z

/ok to test 43e01a3

* ci: Move test optimizer into its own bucket (NVIDIA#1909) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Use matrix for approval-bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update function name Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Adjust approval-bot for copy-pr-bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Parametrize workflow Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Parametrize workflow Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Remove attribute Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update container image tag to use GitHub SHA * chore: Remove file * ci: Fix approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Configure cherrypick bot (NVIDIA#1925) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ci approve dev (NVIDIA#1933) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update nightly schedule (NVIDIA#1934) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Bump pre-flight for runs on main/dev (NVIDIA#1935) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Allow skipping on main (NVIDIA#1936) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/pr template community bot (NVIDIA#1937) * ci: More granular unit tests buckets (NVIDIA#1932) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add sequence packing to RL (NVIDIA#1911) Add sequence packing to RL * chore: Update template (NVIDIA#1939) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Add description about who can merge (NVIDIA#1940) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/fix main on eos (NVIDIA#1938) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/internal mrs (NVIDIA#1942) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix branch of approval bot (NVIDIA#1944) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approvalbot for other branches (NVIDIA#1947) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): Approval bot (NVIDIA#1949) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): Approval gate Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval gate rule Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update golden values nightly Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval gate Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Sync branches Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Smaller image Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Better output Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: sync branches Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix sync bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Finalize Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Finalize Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/sync branches (NVIDIA#1956) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Increase time limit for main tests Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/add milestone (NVIDIA#1951) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove M-FSDP testing under LTS environment (NVIDIA#1959) * ci: Run on push to release branch (NVIDIA#1960) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Add golden values for inference Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix typo in rl section of CODEOWNERS (NVIDIA#1968) * ci: Update copyright checker (NVIDIA#1973) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/auto reminder GitHub (NVIDIA#1955) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update secret Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): `Run tests` label (NVIDIA#1970) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Disable tests again Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Add merge-group to copyright check Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Copyright check on merge-queue Signed-off-by: oliver könig <okoenig@nvidia.com> * zarr soft deprecation (NVIDIA#2004) Signed-off-by: dimapihtar <dpihtar@gmail.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Make `get_asyncio_loop` safe to use repeatedly (NVIDIA#1990) Co-authored-by: oliver könig <okoenig@nvidia.com> * Update symmetric registration interface to sync-up with upstream pytorch change (NVIDIA#1924) Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * chore: Update codeowners (NVIDIA#2012) Signed-off-by: oliver könig <okoenig@nvidia.com> * Deduplicate dynamic engine + coordinator. (NVIDIA#1981) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Safely access state dict args in load ckpt (NVIDIA#1957) Signed-off-by: Maanu Grover <maanug@nvidia.com> * Allow mixed-batch sampling in dynamic inference (NVIDIA#1927) * Stop Nemo_CICD_Test from failing in forks (NVIDIA#2024) * Clean up dynamic inference step (NVIDIA#1992) Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * ci: Auto-update copy-pr-bot vetters (NVIDIA#1850) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> * Have datasets account for tokenizers which incorrectly define PAD (NVIDIA#2017) * ci: Enable integration tests (NVIDIA#2023) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix build-push-wheel workflow (NVIDIA#2022) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update tooling for interactive jobs (NVIDIA#2032) Signed-off-by: oliver könig <okoenig@nvidia.com> * revert(hotfix): ci: trustees_override (NVIDIA#2041) Signed-off-by: oliver könig <okoenig@nvidia.com> * add missing warnings import in model parallel config (NVIDIA#2039) Signed-off-by: ykarnati <ykarnati@nvidia.com> * Reduce-scatter implementation with FP32 accumulation (NVIDIA#1967) Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> * ci(fix): Workflows on `main` (NVIDIA#2045) Signed-off-by: oliver könig <okoenig@nvidia.com> * build: Bump modelopt (NVIDIA#2046) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove TestCaptureFreezeGC unit test. (NVIDIA#1978) * ci: Add multi-approval action (NVIDIA#2051) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Repair codeowners file * ci(hotfix): Set docs allowed to fail Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/test iteration time (NVIDIA#2067) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove performance for ckpt-resume Signed-off-by: oliver könig <okoenig@nvidia.com> * Allow inference test throughput to vary by 10% (NVIDIA#2070) * ci(hotfix): Inference test pipeline Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Fix autoformatter (NVIDIA#2073) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove iteration-time from t5 Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): disable inference test Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Disable inference test Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Bypass approvalbot in merge-queue (NVIDIA#2082) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Enable merge-group for approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update local tooling (NVIDIA#2066) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add extra RL files (NVIDIA#2077) Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Prevent summary jobs from running in forks (NVIDIA#2083) Co-authored-by: oliver könig <okoenig@nvidia.com> * ci: Fix test scope (NVIDIA#2091) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove publish workflows Signed-off-by: oliver könig <okoenig@nvidia.com> * Refactor the attention metadata into separate classes (NVIDIA#2001) Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Guard against incorrectly using MoE prefill graphs (NVIDIA#2030) Co-authored-by: oliver könig <okoenig@nvidia.com> * Revert "Refactor the attention metadata into separate classes (NVIDIA#2001)" This reverts commit a652e2c. * Run mr-slim tests in lightweight-mode (NVIDIA#2106) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Inference | Lazy compile UVM allocator. (NVIDIA#1977) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * chore: Reenable trustees (NVIDIA#2108) Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "Inference | Lazy compile UVM allocator. (NVIDIA#1977)" This reverts commit 7487c53. * ci(fix): Changeset of copyright checker (NVIDIA#2110) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/chore/update release settings (NVIDIA#2097) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove unnecessary check on rotary_pos_cos (NVIDIA#2003) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * (Reverted) Inference | Lazy compile UVM allocator. (NVIDIA#2125) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Refactor Attention Metadata to Separate Classes (NVIDIA#2112) Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Refactor model_provider to model_builder format for ModelOpt examples (NVIDIA#2107) * wandb Inference stats logging (NVIDIA#2026) Co-authored-by: root <root@gpu-h100-0058.cm.cluster> Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster> Co-authored-by: root <root@gpu-h100-0220.cm.cluster> * Make `PipelineParallelLayout` always return str from ` __repr__` (NVIDIA#2055) Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Add flash_attn_3 as first option for FA3 import (NVIDIA#2010) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> * Add debugging hint for case when cudagraphs are created but no matching runner is found (NVIDIA#2129) * ci: LTS container (NVIDIA#2133) Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "ci: LTS container (NVIDIA#2133)" This reverts commit eb48e81. * Fix param init (NVIDIA#2033) Signed-off-by: Chen Cui <chcui@nvidia.com> * Hotfix to unit tests on hopper FA3 (NVIDIA#2143) * Add BytesIO to safe_globals (NVIDIA#2074) * add deprecation warning for legacy tokenizer system (NVIDIA#2145) Signed-off-by: dimapihtar <dpihtar@gmail.com> * replay: ci: Bump LTS container (NVIDIA#2157) Signed-off-by: oliver könig <okoenig@nvidia.com> * Hotfix to unit tests on hopper FA3 (bis) (NVIDIA#2179) * Fix has_modelopt_state() for native Torch checkpoint format (NVIDIA#2160) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * chore: Remove codeowners (NVIDIA#2175) Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix FP8 inference with sequence parallelism (NVIDIA#2009) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> * Replace ModelOpt generation server (NVIDIA#2147) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * Add hybrid model support for dynamic inference engine (NVIDIA#1907) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> * Async task and event loop safety in Megatron Core (NVIDIA#2025) Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> * Rename skip_prompt_log_probs (NVIDIA#2181) * Dynamic inference context | UVM only. (NVIDIA#1983) Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> * Update copy-pr-bot.yaml [skip ci] Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "Dynamic inference context | UVM only. (NVIDIA#1983)" This reverts commit d6979d6. * ci: Run `auto-update-copy-pr-bot` only on forks (NVIDIA#2191) Signed-off-by: oliver könig <okoenig@nvidia.com> * Inference throughput tests: refactor goldens to be in list format (NVIDIA#2072) * Enable TE custom quantization recipe (NVIDIA#2005) Signed-off-by: Evgeny <etsykunov@nvidia.com> Signed-off-by: root <Evgeny> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: root <Evgeny> * Remove redundant logits calculations in gpt_model --------- Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: ykarnati <ykarnati@nvidia.com> Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> Signed-off-by: Evgeny <etsykunov@nvidia.com> Signed-off-by: root <Evgeny> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com> Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com> Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com> Co-authored-by: Deepak Narayanan <2724038+deepakn94@users.noreply.github.com> Co-authored-by: helen ngo <helenn@nvidia.com> Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: kanz-nv <kanz@nvidia.com> Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com> Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com> Co-authored-by: root <root@gpu-h100-0058.cm.cluster> Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster> Co-authored-by: root <root@gpu-h100-0220.cm.cluster> Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> Co-authored-by: Evgeny Tsykunov <e.tsykunov@gmail.com>

* ci: Move test optimizer into its own bucket (NVIDIA#1909) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Use matrix for approval-bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update function name Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Adjust approval-bot for copy-pr-bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Parametrize workflow Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Parametrize workflow Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Remove attribute Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update container image tag to use GitHub SHA * chore: Remove file * ci: Fix approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Configure cherrypick bot (NVIDIA#1925) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ci approve dev (NVIDIA#1933) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update nightly schedule (NVIDIA#1934) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Bump pre-flight for runs on main/dev (NVIDIA#1935) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Allow skipping on main (NVIDIA#1936) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/pr template community bot (NVIDIA#1937) * ci: More granular unit tests buckets (NVIDIA#1932) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add sequence packing to RL (NVIDIA#1911) Add sequence packing to RL * chore: Update template (NVIDIA#1939) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Add description about who can merge (NVIDIA#1940) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/fix main on eos (NVIDIA#1938) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/internal mrs (NVIDIA#1942) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix branch of approval bot (NVIDIA#1944) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approvalbot for other branches (NVIDIA#1947) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): Approval bot (NVIDIA#1949) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): Approval gate Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval gate rule Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update golden values nightly Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval gate Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Sync branches Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Smaller image Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Better output Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: sync branches Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix sync bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Finalize Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Finalize Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/sync branches (NVIDIA#1956) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Increase time limit for main tests Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/add milestone (NVIDIA#1951) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove M-FSDP testing under LTS environment (NVIDIA#1959) * ci: Run on push to release branch (NVIDIA#1960) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Add golden values for inference Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix typo in rl section of CODEOWNERS (NVIDIA#1968) * ci: Update copyright checker (NVIDIA#1973) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/auto reminder GitHub (NVIDIA#1955) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update secret Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): `Run tests` label (NVIDIA#1970) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Disable tests again Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Add merge-group to copyright check Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Copyright check on merge-queue Signed-off-by: oliver könig <okoenig@nvidia.com> * zarr soft deprecation (NVIDIA#2004) Signed-off-by: dimapihtar <dpihtar@gmail.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Make `get_asyncio_loop` safe to use repeatedly (NVIDIA#1990) Co-authored-by: oliver könig <okoenig@nvidia.com> * Update symmetric registration interface to sync-up with upstream pytorch change (NVIDIA#1924) Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * chore: Update codeowners (NVIDIA#2012) Signed-off-by: oliver könig <okoenig@nvidia.com> * Deduplicate dynamic engine + coordinator. (NVIDIA#1981) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Safely access state dict args in load ckpt (NVIDIA#1957) Signed-off-by: Maanu Grover <maanug@nvidia.com> * Allow mixed-batch sampling in dynamic inference (NVIDIA#1927) * Stop Nemo_CICD_Test from failing in forks (NVIDIA#2024) * Clean up dynamic inference step (NVIDIA#1992) Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * ci: Auto-update copy-pr-bot vetters (NVIDIA#1850) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> * Have datasets account for tokenizers which incorrectly define PAD (NVIDIA#2017) * ci: Enable integration tests (NVIDIA#2023) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix build-push-wheel workflow (NVIDIA#2022) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update tooling for interactive jobs (NVIDIA#2032) Signed-off-by: oliver könig <okoenig@nvidia.com> * revert(hotfix): ci: trustees_override (NVIDIA#2041) Signed-off-by: oliver könig <okoenig@nvidia.com> * add missing warnings import in model parallel config (NVIDIA#2039) Signed-off-by: ykarnati <ykarnati@nvidia.com> * Reduce-scatter implementation with FP32 accumulation (NVIDIA#1967) Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> * ci(fix): Workflows on `main` (NVIDIA#2045) Signed-off-by: oliver könig <okoenig@nvidia.com> * build: Bump modelopt (NVIDIA#2046) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove TestCaptureFreezeGC unit test. (NVIDIA#1978) * ci: Add multi-approval action (NVIDIA#2051) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Repair codeowners file * ci(hotfix): Set docs allowed to fail Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/test iteration time (NVIDIA#2067) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove performance for ckpt-resume Signed-off-by: oliver könig <okoenig@nvidia.com> * Allow inference test throughput to vary by 10% (NVIDIA#2070) * ci(hotfix): Inference test pipeline Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Fix autoformatter (NVIDIA#2073) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove iteration-time from t5 Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): disable inference test Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Disable inference test Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Bypass approvalbot in merge-queue (NVIDIA#2082) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Enable merge-group for approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update local tooling (NVIDIA#2066) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add extra RL files (NVIDIA#2077) Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Prevent summary jobs from running in forks (NVIDIA#2083) Co-authored-by: oliver könig <okoenig@nvidia.com> * ci: Fix test scope (NVIDIA#2091) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove publish workflows Signed-off-by: oliver könig <okoenig@nvidia.com> * Refactor the attention metadata into separate classes (NVIDIA#2001) Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Guard against incorrectly using MoE prefill graphs (NVIDIA#2030) Co-authored-by: oliver könig <okoenig@nvidia.com> * Revert "Refactor the attention metadata into separate classes (NVIDIA#2001)" This reverts commit a652e2c. * Run mr-slim tests in lightweight-mode (NVIDIA#2106) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Inference | Lazy compile UVM allocator. (NVIDIA#1977) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * chore: Reenable trustees (NVIDIA#2108) Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "Inference | Lazy compile UVM allocator. (NVIDIA#1977)" This reverts commit 7487c53. * ci(fix): Changeset of copyright checker (NVIDIA#2110) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/chore/update release settings (NVIDIA#2097) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove unnecessary check on rotary_pos_cos (NVIDIA#2003) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * (Reverted) Inference | Lazy compile UVM allocator. (NVIDIA#2125) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Refactor Attention Metadata to Separate Classes (NVIDIA#2112) Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Refactor model_provider to model_builder format for ModelOpt examples (NVIDIA#2107) * wandb Inference stats logging (NVIDIA#2026) Co-authored-by: root <root@gpu-h100-0058.cm.cluster> Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster> Co-authored-by: root <root@gpu-h100-0220.cm.cluster> * Make `PipelineParallelLayout` always return str from ` __repr__` (NVIDIA#2055) Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Add flash_attn_3 as first option for FA3 import (NVIDIA#2010) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> * Add debugging hint for case when cudagraphs are created but no matching runner is found (NVIDIA#2129) * ci: LTS container (NVIDIA#2133) Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "ci: LTS container (NVIDIA#2133)" This reverts commit eb48e81. * Fix param init (NVIDIA#2033) Signed-off-by: Chen Cui <chcui@nvidia.com> * Hotfix to unit tests on hopper FA3 (NVIDIA#2143) * Add BytesIO to safe_globals (NVIDIA#2074) * add deprecation warning for legacy tokenizer system (NVIDIA#2145) Signed-off-by: dimapihtar <dpihtar@gmail.com> * replay: ci: Bump LTS container (NVIDIA#2157) Signed-off-by: oliver könig <okoenig@nvidia.com> * Hotfix to unit tests on hopper FA3 (bis) (NVIDIA#2179) * Fix has_modelopt_state() for native Torch checkpoint format (NVIDIA#2160) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * chore: Remove codeowners (NVIDIA#2175) Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix FP8 inference with sequence parallelism (NVIDIA#2009) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> * Replace ModelOpt generation server (NVIDIA#2147) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * Add hybrid model support for dynamic inference engine (NVIDIA#1907) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> * Async task and event loop safety in Megatron Core (NVIDIA#2025) Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> * Rename skip_prompt_log_probs (NVIDIA#2181) * Dynamic inference context | UVM only. (NVIDIA#1983) Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> * Update copy-pr-bot.yaml [skip ci] Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "Dynamic inference context | UVM only. (NVIDIA#1983)" This reverts commit d6979d6. * ci: Run `auto-update-copy-pr-bot` only on forks (NVIDIA#2191) Signed-off-by: oliver könig <okoenig@nvidia.com> * Inference throughput tests: refactor goldens to be in list format (NVIDIA#2072) * Enable TE custom quantization recipe (NVIDIA#2005) Signed-off-by: Evgeny <etsykunov@nvidia.com> Signed-off-by: root <Evgeny> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: root <Evgeny> * Add MoE parameters to ModelOpt pruning example + conf fixes (NVIDIA#2205) Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Add repr to pg collection class (NVIDIA#2089) Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com> * Move `data_samplers.py` from `legacy` to `training.datasets` & add `DistributedSignalHandler` to DataLoader workers (NVIDIA#2068) * Fix Megatron-FSDP checkpoint save failure (NVIDIA#2138) --------- Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: ykarnati <ykarnati@nvidia.com> Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> Signed-off-by: Evgeny <etsykunov@nvidia.com> Signed-off-by: root <Evgeny> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com> Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com> Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com> Co-authored-by: Deepak Narayanan <2724038+deepakn94@users.noreply.github.com> Co-authored-by: helen ngo <helenn@nvidia.com> Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: kanz-nv <kanz@nvidia.com> Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com> Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com> Co-authored-by: root <root@gpu-h100-0058.cm.cluster> Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster> Co-authored-by: root <root@gpu-h100-0220.cm.cluster> Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> Co-authored-by: Evgeny Tsykunov <e.tsykunov@gmail.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com> Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>

Signed-off-by: Evgeny <etsykunov@nvidia.com> Signed-off-by: root <Evgeny> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: root <Evgeny>

negvet added 3 commits September 25, 2025 17:10

Enable custom recipe in TE

48e3b48

Signed-off-by: Evgeny <etsykunov@nvidia.com>

Merge main

cdab16f

Signed-off-by: Evgeny <etsykunov@nvidia.com>

Support fp4 recipe to be custom as well as fp8

f81eb9f

Signed-off-by: Evgeny <etsykunov@nvidia.com>

negvet requested review from a team as code owners October 28, 2025 17:14

Merge branch 'main' into etsykunov/custom_quant_recipe

36dbfbf

ChenhanYu reviewed Oct 29, 2025

View reviewed changes

deepakn94 approved these changes Oct 30, 2025

View reviewed changes

Merge branch 'main' into etsykunov/custom_quant_recipe

bd21b45

copy-pr-bot Bot temporarily deployed to nemo-ci October 30, 2025 19:11 Inactive

copy-pr-bot Bot had a problem deploying to nemo-ci October 30, 2025 19:11 Failure

copy-pr-bot Bot temporarily deployed to nemo-ci October 30, 2025 19:11 Inactive

ko3n1g added this to the Core 0.16 milestone Oct 30, 2025

copy-pr-bot Bot temporarily deployed to nemo-ci October 30, 2025 19:12 Inactive

copy-pr-bot Bot temporarily deployed to public October 30, 2025 19:15 Inactive

ChenhanYu self-requested a review November 5, 2025 00:22

ChenhanYu approved these changes Nov 5, 2025

View reviewed changes

Fix linters

64a5a52

Signed-off-by: Evgeny <etsykunov@nvidia.com>

ananthsub reviewed Nov 6, 2025

View reviewed changes

root and others added 3 commits November 7, 2025 15:29

Move validations to transformer_config post_init

2245c22

Signed-off-by: root <Evgeny>

Merge branch 'main' into etsykunov/custom_quant_recipe

9c29ab1

Merge branch 'main' into etsykunov/custom_quant_recipe

43e01a3

negvet requested a review from ananthsub November 8, 2025 10:58

copy-pr-bot Bot temporarily deployed to public November 10, 2025 10:32 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci November 10, 2025 10:35 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci November 10, 2025 10:36 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci November 10, 2025 10:50 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci November 11, 2025 09:27 Inactive

ko3n1g added this pull request to the merge queue Nov 11, 2025

Merged via the queue into NVIDIA:main with commit 1f6cde8 Nov 11, 2025
47 of 49 checks passed

timmoon10 mentioned this pull request Aug 7, 2026

Add missing config options for TE quantization recipes NVIDIA-NeMo/Megatron-Bridge#5408

Open

5 tasks

Navigation Menu

Uh oh!

Enable TE custom quantization recipe - #2005

Enable TE custom quantization recipe#2005
ko3n1g merged 9 commits into
NVIDIA:mainfrom
negvet:etsykunov/custom_quant_recipe

negvet commented Oct 28, 2025

Uh oh!

copy-pr-bot Bot commented Oct 28, 2025

Uh oh!

ChenhanYu Oct 29, 2025

Uh oh!

negvet Oct 30, 2025

Uh oh!

ko3n1g commented Oct 30, 2025

Uh oh!

copy-pr-bot Bot commented Oct 30, 2025

Uh oh!

ko3n1g commented Oct 30, 2025

Uh oh!

ananthsub Nov 6, 2025

Uh oh!

negvet Nov 7, 2025

Uh oh!

ananthsub commented Nov 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		return fn


		def _get_custom_recipe(quantizer_factory_python_path: str) -> Union[Fp8Recipe, Fp4Recipe]:

Uh oh!

Conversation

negvet commented Oct 28, 2025

What does this PR do ?

Contribution process

Pre-checks

Code review

(Step 1): Add PR label Expert Review

(Step 2): Collect the expert reviewers reviews

(Step 3): Final Review

(Optional Step 4): Cherry-pick into release branch

Merging your PR

Uh oh!

copy-pr-bot Bot commented Oct 28, 2025

Uh oh!

ChenhanYu Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

negvet Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

ko3n1g commented Oct 30, 2025

Uh oh!

copy-pr-bot Bot commented Oct 30, 2025

Uh oh!

ko3n1g commented Oct 30, 2025

Uh oh!

ananthsub Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

negvet Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

ananthsub commented Nov 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

(Step 1): Add PR label `Expert Review`