[FT] Propagate batch size control for vLLM backend

[alvin319]

Issue encountered

With vLLM backend, currently there's no way for us to control the batch size defined in here and the vLLM model config does not have ways to determine a specific batch size. However, we can control the maximum number of sequences (batch size) in vLLM directly from examples such as this.

Solution/Feature

• Propagate the max_num_seqs parameter into the initialization of the vLLM model.

Possible alternatives

• Other alternatives are to implement batching ourselves, which is an overkill since the vLLM backend already supports that.

[alvin319]

I'm happy to take this on, but I figured it warrants a discussion since vLLM leverages continuous batching, which behaves differently than the common understanding of a "fixed" batch size in the LightEval world.

[NathanHB]

hey ! it would be usefull to expose the max_num_seqs argument. you can add it to the generation config and propagate it to LLM :)

[alvin319]

Hi @NathanHB -- I just pushed a PR out #588 and please review it when you get the chance!