fp8 online quant: split out Fp8OnlineLinearMethod

[vkuzo]

Summary:

Split out Fp8OnlineLinearMethod from Fp8LinearMethod to more clearly separate online quant from offline quant logic, following a similar PR recently landed for Fp8OnlineMoEMethod.

In the same PR, beef up testing for online quant in integration tests a bit so we can depend on tests for testing future functionality for online quant. Specifically, extend the online fp8 quant test to also include a small moe model, and also extend it to run inference with a couple of tokens.

Test Plan:

// on a NVIDIA B200
// run online quant test (dense + moe smoke tests)
with-proxy pytest tests/quantization/test_fp8.py -s -x -k online_quantization
// run entire fp8.py test suite
with-proxy pytest tests/quantization/test_fp8.py -s -x

Reviewers:

Subscribers:

Tasks:

Tags:

Purpose

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

---

Note

^{Cursor Bugbot is generating a summary for commit d209af298f5d6601a9328c21c252c4fc11de1911. Configure here.}

---

Note

Cleanly separates online FP8 linear quantization from offline (serialized) flow and strengthens smoke test coverage.

• Adds Fp8OnlineLinearMethod that patches weight loading to quantize fp16/bf16 weights on-the-fly via ops.scaled_fp8_quant, with optional Marlin preparation
• Updates Fp8Config.get_quant_method to choose Fp8OnlineLinearMethod vs Fp8LinearMethod based on is_checkpoint_fp8_serialized; simplifies Fp8LinearMethod to the serialized-FP8 path
• Keeps MoE split (Fp8OnlineMoEMethod/Fp8MoEMethod) unchanged, integrates with selection logic
• Expands tests: new test_online_quantization runs on facebook/opt-125m and Qwen/Qwen1.5-MoE-A2.7B, parameterizes KV cache (auto/fp8) and Marlin/ROCm flags, and performs short greedy generation

^{Written by Cursor Bugbot for commit d209af298f5d6601a9328c21c252c4fc11de1911. This will update automatically on new commits. Configure here.}

---

Note

Cleanly separates online FP8 linear quantization from the serialized (offline) flow and strengthens smoke test coverage.

• Adds Fp8OnlineLinearMethod that patches weight loading to quantize fp16/bf16 weights via ops.scaled_fp8_quant, with optional Marlin prep
• Updates Fp8Config.get_quant_method to choose Fp8OnlineLinearMethod vs Fp8LinearMethod based on is_checkpoint_fp8_serialized; Fp8LinearMethod now focuses on fp8-serialized path
• Keeps MoE split (Fp8OnlineMoEMethod/Fp8MoEMethod) and integrates it into the same selection logic
• Expands test_online_quantization: runs on facebook/opt-125m and Qwen/Qwen1.5-MoE-A2.7B, parameterizes KV cache (auto/fp8) and Marlin/ROCm flags, and validates a short greedy generation

^{Written by Cursor Bugbot for commit b9d2f36f55819d195b00afd8efabf99ab9824a22. This will update automatically on new commits. Configure here.}

[vkuzo]

this is outdated, deleting

[vkuzo]

all of these changes is just moving online quant code to the new child class

[vkuzo]

all of these changes is just moving online quant code to the new child class

[gemini-code-assist]

Code Review

This pull request is a well-executed refactoring that splits the online quantization logic for FP8 linear layers into a new Fp8OnlineLinearMethod class. This change significantly improves code clarity and maintainability by separating the concerns of online and offline quantization, following the pattern established for MoE layers. The implementation is clean, and the logic has been moved correctly. The tests have also been updated to cover both dense and MoE models for online quantization, which is a great improvement. Overall, this is a solid contribution that enhances the codebase.

[robertgshaw2-redhat]

This looks good. Can you make a new directory called online where this sits?

[mergify]

Hi @vkuzo, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[kylesayrs]

Agree with @robertgshaw2-redhat, it'd be great to break these out into separate files/directories

Looks great in concept, thanks for looking at this

[mergify]

Hi @vkuzo, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mgoin]

LGTM as just structural changes, thanks!

[vkuzo]

CI is failing with an OOM on QWen 1.5B on a 24GB NVIDIA L4 machine, my best guess is that we actually need #31914 to land for memory usage to be sane. I'm going to remove Qwen 1.5B from this PR (since the fp8.py changes here do not touch MoEs) to unblock, and we can revisit online quant + moe in CI once the memory issue is fixed.

[vkuzo]

@mgoin thanks, CI is green after the latest fix

[vkuzo]

rebasing manually since automatic rebase failed on permissions

[vkuzo]

rebased on top of #27814 which just landed