Feature request: Batched inference for llama.cpp models

[ggbetz]

Hi Luca,

llama.cpp / llama-cpp-python are apparently going to allow for batched inference [1] [2].

Is this something you've on your radar, and are planning to reflect in the lmql-server, as well? So far, we have batch_size=1 for llama.cpp models, right?

Thanks again for creating and maintaining this great project!

Gregor

[lbeurerkellner]

Thanks for raising this, we will keep it on our radar. It should be simple to add support for this, once it is upstreamed in llama.cpp/llama-cpp-python.

[reuank]

Just a quick update on this topic. It looks like llama-cpp-python will add that feature very soon: abetlen/llama-cpp-python#951.

[lbeurerkellner]

Marking this as a good first issue for backend work. The llama.cpp backend lives in https://github.com/eth-sri/lmql/blob/main/src/lmql/models/lmtp/backends/llama_cpp_model.py and is currently limited to max_batch_size of 1.

[Saibo-creator]

It looks like the batch inference feature mentioned earlier is still in development. I'll hold off for now.

[balu54]

Is parallel inference will comes with batche inference feature?

Marking this as a good first issue for backend work. The llama.cpp backend lives in https://github.com/eth-sri/lmql/blob/main/src/lmql/models/lmtp/backends/llama_cpp_model.py and is currently limited to max_batch_size of 1.

Is parallel inference includes in batch inference.