[Doc] update(example model): for OpenAI compatible serving - #4503
Merged
Merged
Conversation
The previous model was in a gated repo and not available
fpaupier
changed the title
Member
|
FWIW the |
Collaborator
|
When we added Mistral it doesn't have the gate :(((( I think NousResearch is a good alternative, another one is Zephyr which we uses for testing. |
mgoin
reviewed
May 1, 2024
mgoin
left a comment
mgoin
left a comment
Member
There was a problem hiding this comment.
Should be good to go after this model alias switch
| You can start the server using Python, or using [Docker](deploying_with_docker.rst): | ||
| ```bash | ||
| python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-Instruct-v0.2 --dtype auto --api-key token-abc123 | ||
| python -m vllm.entrypoints.openai.api_server --model unsloth/llama-3-8b-Instruct --dtype auto --api-key token-abc123 |
Member
There was a problem hiding this comment.
Suggested change
| python -m vllm.entrypoints.openai.api_server --model unsloth/llama-3-8b-Instruct --dtype auto --api-key token-abc123 | |
| python -m vllm.entrypoints.openai.api_server --model NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123 |
|
|
||
| completion = client.chat.completions.create( | ||
| model="mistralai/Mistral-7B-Instruct-v0.2", | ||
| model="unsloth/llama-3-8b-Instruct", |
Member
There was a problem hiding this comment.
Suggested change
| model="unsloth/llama-3-8b-Instruct", | |
| model="NousResearch/Meta-Llama-3-8B-Instruct", |
| ```python | ||
| completion = client.chat.completions.create( | ||
| model="mistralai/Mistral-7B-Instruct-v0.2", | ||
| model="unsloth/llama-3-8b-Instruct", |
Member
There was a problem hiding this comment.
Suggested change
| model="unsloth/llama-3-8b-Instruct", | |
| model="NousResearch/Meta-Llama-3-8B-Instruct", |
| specifies how are roles, messages, and other chat-specific tokens are encoded in the input. | ||
|
|
||
| An example chat template for `mistralai/Mistral-7B-Instruct-v0.2` can be found [here](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2#instruction-format) | ||
| An example chat template for `unsloth/llama-3-8b-Instruct` can be found [here](https://colab.research.google.com/drive/1Aau3lgPzeZKQ-98h69CCu1UJcvIBLmy2?usp=sharing#scrollTo=vITh0KVJ10qX) |
Member
There was a problem hiding this comment.
Suggested change
| An example chat template for `unsloth/llama-3-8b-Instruct` can be found [here](https://colab.research.google.com/drive/1Aau3lgPzeZKQ-98h69CCu1UJcvIBLmy2?usp=sharing#scrollTo=vITh0KVJ10qX) | |
| An example chat template for `NousResearch/Meta-Llama-3-8B-Instruct` can be found [here](https://colab.research.google.com/drive/1Aau3lgPzeZKQ-98h69CCu1UJcvIBLmy2?usp=sharing#scrollTo=vITh0KVJ10qX) |
instead of unsloth model
Contributor
Author
|
good suggestion @mgoin, I updated with the |
robertgshaw2-redhat
pushed a commit
to neuralmagic/nm-vllm
that referenced
this pull request
May 6, 2024
z103cb
pushed a commit
to z103cb/opendatahub_vllm
that referenced
this pull request
May 7, 2024
dtrifiro
pushed a commit
to opendatahub-io/vllm
that referenced
this pull request
May 7, 2024
Draft
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previous reference model model
mistralai/Mistral-7B-Instruct-v0.2in getting started doc required special access to the model card on hugging face, leading to an exception when starting the reference example.The error with was:
The new model proposed for the getting started - unsloth/llama-3-8b-Instruct - is publicly available with reasonable performance and runs on commodity hardware. The getting started proposed in the vLLM Quickstart now works.