FlashInfer 0.2.3+ does not support per-request generators

[sirouk]

I'm using FlashInfer 2.5 attention backend with vLLM 0.9.0.1 on CUDA 12.8 and with VLLM_USE_FLASHINFER_SAMPLER=1, when temperature deviates between requests, this is what happens:

...
1|vllm-server-7011  | (VllmWorker rank=2 pid=16417) WARNING 05-31 03:56:50 [topk_topp_sampler.py:99] FlashInfer 0.2.3+ does not support per-request generators. Falling back to PyTorch-native implementation.
1|vllm-server-7011  | (VllmWorker rank=3 pid=16418) WARNING 05-31 03:56:50 [topk_topp_sampler.py:99] FlashInfer 0.2.3+ does not support per-request generators. Falling back to PyTorch-native implementation.
1|vllm-server-7011  | (VllmWorker rank=7 pid=16422) WARNING 05-31 03:56:50 [topk_topp_sampler.py:99] FlashInfer 0.2.3+ does not support per-request generators. Falling back to PyTorch-native implementation.
1|vllm-server-7011  | (VllmWorker rank=6 pid=16421) WARNING 05-31 03:56:50 [topk_topp_sampler.py:99] FlashInfer 0.2.3+ does not support per-request generators. Falling back to PyTorch-native implementation.
1|vllm-server-7011  | (VllmWorker rank=5 pid=16420) WARNING 05-31 03:56:50 [topk_topp_sampler.py:99] FlashInfer 0.2.3+ does not support per-request generators. Falling back to PyTorch-native implementation.
...

I am able to set it to VLLM_USE_FLASHINFER_SAMPLER=1 and it falls back to Flash Attention sampler.

Please let me know if this is a WIP, thanks!

Cross post:
vllm-project/vllm#18811

[yzh119]

The warning means flashinfer sampler is not enabled.

I wonder why do we need a standalone generator when temperature deviates between requests? I think temperature is irrelavant to random generator?

Per request generator is something feature we can support, but it seems vllm using a dictionary to store the generators https://github.com/vllm-project/vllm/blob/0f71e24034263363e48cdc6ae036e7ca057a4f44/vllm/v1/sample/ops/topk_topp_sampler.py#L87, which will introduce some overhead of turning these generators to cuda kernel inputs. @WoosukKwon if we can make it a cuda tensor of Philox seed a tensor of Philox offset, it will be more efficient.

[xidulu]

Following up on this issue:

Is there some simple example on what are per-request generators?

This is what my code look like

def get_multi_completion_vllm(
        model,
        context,
        num_generations=1,
        temperature=0.6,
        top_p=0.95,
        top_k=-1,
        max_tokens=2000,
        seed=1,
        sample_params = None,
    ):
    if not sample_params:
        sample_params = SamplingParams(
            temperature=temperature,
            top_p=top_p,
            top_k=top_k,
            max_tokens=max_tokens,
            n=num_generations,
            seed=seed
        )
    
    contiunation = model.generate(
        context, sample_params,
        use_tqdm=False
    )
    return [
        o.text for o in contiunation[0].outputs
    ]

vllm_model = Model(....)

sample_params_completion = SamplingParams(
        n=16,
        temperature=0.6,
        max_tokens=1000,
        top_p=0.95,
        top_k=-1,
        seed=seed
 )

answers = []
for prompts in DATASET:
    early_stop_answers = get_multi_completion_vllm(
                vllm_model,
                prompts,
                sample_params=sample_params_completion
            )
    answers.append(early_stop_answers)

And then I also received the warning

WARNING 06-10 16:04:25 [topk_topp_sampler.py:99] FlashInfer 0.2.3+ does not support per-request generators. Falling back to PyTorch-native implementation.

I would like to fix it but I don't know why my code involves a per request generator