Implement Multi-Process Handling (#7) #15
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change adds functionality for llama-swap to manage multiple llama-server backends. This is an advanced use case that requires enough GPU resources to load multiple models. An example is when programming and a larger model like qwen-coder-32B is used for code generation and qwen-coder-1.5B is used for auto-completion.
This implements the design from issue #7. A new
groups
configuration is introduced:When a request to
/v1/chat/completions
with a{"model":"coding/qwen-coder-1.5b"}
is sent llama-swap will load both models into the GPU(s). When a request for "coding/qwen-coder-32b" is requested the request will be served immediately. This can greatly reduces the time to first token for a request for both models in the configuration.It is up to the operator to ensure the group configuration is viable. Generally watch out for these things:
port
conflicts in the configurations