Draft: FP4 Disagg + MTP Configs

[trevor-m]

Adds two configs:

• 1p2d-mtp - Required minimal changes from original 1p2d config. I just had to add the speculative args and reduce the memory fraction on the decode nodes.
• max-tpt-2-mtp - This one required a lot of workarounds:
  • To avoid SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK cap of 1024, I basically halved all of the settings related to tokens to account for speculative-num-steps=1.
    • max running requests: 67584->33792
    • cuda graph max bs: 1024->512 (later reduced to 256)
    • num reserved decode tokens 112->224
  • Use shell script patch for --speculative-moe-a2a-backend (remove once support mtp with deepseek r1 nvfp4 model sgl-project/sglang#13115 is merged)
  • After those changes I encountered OOMs on the decode side during draft mode cuda capture. I reduced the mem fraction from 0.83->0.73 but it didn't appear to make a difference. I ultimately just reduced the cuda graph bs to 256.
  • On the prefill side, there is a bug with single batch overlap and the speculative layer. I just disabled SBO for now

[ishandhanani]

Can you rename that shell script and add a comment for why its used and when we should remove it. Im assumign we wil remove once we ship sgl > 0.5.6

[trevor-m]

Thanks, yes it can be removed once sgl-project/sglang#13115 is merged.

I also updated the PR description with all of the changes I made to the configs.

[Fridge003]

@trevor-m Is there any accuracy or performance data

[ishandhanani]

can we rename this file to gb200-fp4-mtp-setup.sh

i dont have a good way of organizing these lol so descriptive naming is probably the best

[trevor-m]

What about checkout-pr-13115.sh?
I also found out we need to disable the engine patch since this PR is based on main. I tried rebasing to 0.5.5 but it might have some dependencies on other PRs.

[ishandhanani]

I see. I can assist with this tomorrow

[trevor-m]

@Fridge003

@trevor-m Is there any accuracy or performance data

This is the pareto with the low latency config:

Let me try to check the accuracy.
The high throughput one is still not ready.

[Fridge003]

@trevor-m @ishandhanani
sgl-project/sglang#13115 is merged