Skip to content

fix: Add support for using MXFP8 datatype in flashinfer A2A communicator - #30972

Closed
djns99 wants to merge 1 commit into
sgl-project:mainfrom
djns99:djns99/mxfp8-a2a
Closed

fix: Add support for using MXFP8 datatype in flashinfer A2A communicator#30972
djns99 wants to merge 1 commit into
sgl-project:mainfrom
djns99:djns99/mxfp8-a2a

Conversation

@djns99

@djns99 djns99 commented Jul 13, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #32316431606
Latest PR Test (Extra): ❌ Run #32316431413

@gemini-code-assist

gemini-code-assist Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions github-actions Bot added documentation Improvements or additions to documentation quant LLM Quantization sgl-kernel labels Jul 13, 2026
@djns99
djns99 force-pushed the djns99/mxfp8-a2a branch 4 times, most recently from 101d0f2 to c6767c8 Compare July 14, 2026 02:01
@djns99
djns99 marked this pull request as ready for review July 14, 2026 21:25
@gemini-code-assist

gemini-code-assist Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@b8zhong b8zhong self-assigned this Jul 14, 2026
@djns99 djns99 mentioned this pull request Jul 16, 2026
Closed
5 tasks
@djns99
djns99 force-pushed the djns99/mxfp8-a2a branch from c6767c8 to d6bdc57 Compare July 16, 2026 03:32
@djns99
djns99 marked this pull request as draft July 20, 2026 03:18
@djns99
djns99 marked this pull request as ready for review July 20, 2026 03:19
@gemini-code-assist

gemini-code-assist Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@djns99
djns99 force-pushed the djns99/mxfp8-a2a branch from d6bdc57 to 8d35444 Compare July 20, 2026 03:19
@nvpohanh

nvpohanh commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

@djns99 could you fix the linter issue?

@djns99
djns99 force-pushed the djns99/mxfp8-a2a branch from 8d35444 to 8119d3b Compare July 30, 2026 03:58
@djns99
djns99 force-pushed the djns99/mxfp8-a2a branch from 8119d3b to 07c1a19 Compare August 6, 2026 02:35
@djns99
djns99 marked this pull request as draft August 6, 2026 05:40
@djns99
djns99 marked this pull request as ready for review August 9, 2026 23:36
@djns99

djns99 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@nvpohanh I think this should be ready now

@nvpohanh

nvpohanh commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Still seeing lint issue: https://github.com/sgl-project/sglang/actions/runs/31343321057/job/93320571089?pr=30972

@djns99

djns99 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Hmm odd it passed locally when I ran it earlier today. Must have had a out of sync local state. Pushed a fix

nvpohanh
nvpohanh reviewed Aug 12, 2026
View reviewed changes
Comment thread python/sglang/srt/server_args.py
"Select DeepEP dispatcher output dtype",
NS("exec.moe"),
] = "auto"
flashinfer_a2a_dispatch_type: A[

@nvpohanh nvpohanh Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering why we need this flag. We should always just dispatch with the lowest precision supported by the quant config, right?

@nvpohanh nvpohanh Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I even think SGLANG_MOE_NVFP4_DISPATCH is not needed

@djns99 djns99 Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree in most cases we want to do this. Im not familiar enough to say definitively that there are no cases that this is disadvantageous.
I can see low batch cases which are latency bound being better to use BF16 and quantizing fused into the MOE kernel operations so we dont pay the cost of a separate quantization kernel

nvpohanh
nvpohanh reviewed Aug 12, 2026
View reviewed changes
Comment thread python/sglang/srt/server_args.py
else:
if self.moe_runner_backend == "flashinfer_trtllm":
self.moe_runner_backend = "flashinfer_trtllm_routed"
logger.warning(

@nvpohanh nvpohanh Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this warning for?

@djns99 djns99 Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is making sure the user is warned if they ask for the wrong flashinfer_trtllm backend. Otherwise, it will get a less clear error in _handle_flashinfer_a2a_dispatch_type
We can make this an error if that would be more clear

@nvpohanh

nvpohanh commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

@djns99 could you fix the conflicts?

@djns99 djns99 closed this Sep 7, 2026
@djns99

djns99 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Changes will be merged via #31470

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nvpohanh nvpohanh nvpohanh left review comments
@ch-wan ch-wan Awaiting requested review from ch-wan ch-wan is a code owner
@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner
@Edwardf0t1 Edwardf0t1 Awaiting requested review from Edwardf0t1 Edwardf0t1 is a code owner
@FlamingoPg FlamingoPg Awaiting requested review from FlamingoPg FlamingoPg is a code owner
@AniZpZ AniZpZ Awaiting requested review from AniZpZ AniZpZ is a code owner
@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner
@b8zhong b8zhong Awaiting requested review from b8zhong b8zhong is a code owner
@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner
@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner
@Fridge003 Fridge003 Awaiting requested review from Fridge003 Fridge003 is a code owner
@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner
@wisclmy0611 wisclmy0611 Awaiting requested review from wisclmy0611 wisclmy0611 is a code owner
@zijiexia zijiexia Awaiting requested review from zijiexia zijiexia is a code owner
@JustinTong0323 JustinTong0323 Awaiting requested review from JustinTong0323 JustinTong0323 is a code owner
@sogalin sogalin Awaiting requested review from sogalin sogalin is a code owner
@OrangeRedeng OrangeRedeng Awaiting requested review from OrangeRedeng OrangeRedeng is a code owner
@Alisehen Alisehen Awaiting requested review from Alisehen Alisehen is a code owner
@mmangkad mmangkad Awaiting requested review from mmangkad mmangkad is a code owner

Assignees

@b8zhong b8zhong

Labels

documentation Improvements or additions to documentation quant LLM Quantization sgl-kernel

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@djns99 @nvpohanh @b8zhong