[Performance]: Reevaluate padding requirement for Sequence Parallelism

[ProExpertProg]

Proposal to improve performance

Currently, we pad num_tokens to a multiple of TP (SP) size when Sequence Parallelism (SP) is enabled. However, we should benchmark to see the performance reduction in padding num_tokens and compare it to just manually padding with -num_tokens % tp_size around the sequence parallel section, or by doing uneven work across TP ranks by manipulating the sizes returned by reduce_scatter (more complicated but theoretically the best performance).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[baonudesifeizhai]

i will have a try ... currently
17 vs 16 tokens +10.58%.... on this https://paste.ubuntu.com/p/K3TSjBG6dR/ script

[baonudesifeizhai]

A small microbench (16 vs 17 tokens, where 17 pads to 24 under TP=8+SP) does show ~10% higher latency.
This happens because:

the prefill token count goes from 16 → 24 (a 50% increase),

but prefill FLOPs only account for a small fraction of the total request latency,
so the wall-clock increase ends up around ~10% rather than ~50%.

In a workload closer to real inference scenarios (TP=8, SP enabled, Llama-3.1-8B, batch=128, seq_len≈2048, gen_len=16):

padding only adds 8 / 2048 ≈ 0.4% extra prefill tokens,

and the measured throughput stays in the 68k–69k tokens/s range with no consistent trend related to the padding remainder.

https://paste.ubuntu.com/p/PJNSzgDQqQ/

[ProExpertProg]

The approach makes sense! But I'm a little worried about the methodology.

Can you report some vllm bench latency results for different size models? You can use 1024 vs 1025 input tokens, 1 output token, batch 1, that way it'll be a single batch of size 1024.

As seen in #27700, the threshold will be 8 MiB for hidden size 8192 which gives num_tokens>=512 for 16bit models and num_tokens>=1024 for 8bit tokens.

So the cases that would be good to check:

• llama3-70B-FP8, 1024 vs 1025
• llama3-70B, 1024 vs 1025
• llama3-70B, 512 vs 513

Does that make sense?

[ProExpertProg]

Huh maybe try running with larger --num-iters?

[baonudesifeizhai]

https://paste.ubuntu.com/p/6DktnrqHdG/ i run with vllm benchmark
python benchmarks/bench_sp_padding_vllm_bench.py --run-all
--tensor-parallel-size 8
--num-iters 50
--num-iters-warmup 20
--gpu-memory-utilization 0.85

... but

================================================================================
Results for meta-llama/Llama-3.1-70B-Instruct
================================================================================
  Base (512 tokens):     2.3126s
  Base+1 (513 tokens): 2.3397s
  Overhead: +1.17%
================================================================================

================================================================================
SUMMARY
================================================================================
Model                                              Len      Base(s)    Base+1(s)    Overhead
--------------------------------------------------------------------------------
RedHatAI/Meta-Llama-3.1-70B-Instruct-FP8           1024     5.0433     4.9668         -1.52%
meta-llama/Llama-3.1-70B-Instruct                  1024     4.9141     5.2309         +6.45%
meta-llama/Llama-3.1-70B-Instruct                  512      2.3126     2.3397         +1.17%
================================================================================
  

Huh maybe try running with larger --num-iters?