Skip to content

Commit

Permalink
update: modify parameter for the LoRA adapter on the vllm page
Browse files Browse the repository at this point in the history
  • Loading branch information
Um Changyong committed Nov 5, 2024
1 parent 66966a6 commit 70e326f
Showing 1 changed file with 12 additions and 2 deletions.
14 changes: 12 additions & 2 deletions docs/docs/integrations/llms/vllm.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -266,8 +266,18 @@
"from langchain_community.llms import VLLM\n",
"from vllm.lora.request import LoRARequest\n",
"\n",
"llm = VLLM(model=\"meta-llama/Llama-2-7b-hf\", enable_lora=True)\n",
"\n",
"llm = VLLM(\n",
" model=\"meta-llama/Llama-3.2-3B-Instruct\",\n",
" max_new_tokens=300,\n",
" top_k=1,\n",
" top_p=0.90,\n",
" temperature=0.1,\n",
" vllm_kwargs={\n",
" \"gpu_memory_utilization\": 0.5,\n",
" \"enable_lora\": True,\n",
" \"max_model_len\": 350,\n",
" },\n",
")\n",
"LoRA_ADAPTER_PATH = \"path/to/adapter\"\n",
"lora_adapter = LoRARequest(\"lora_adapter\", 1, LoRA_ADAPTER_PATH)\n",
"\n",
Expand Down

0 comments on commit 70e326f

Please sign in to comment.