Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

请增加下参数num_ctx用来对接ollama本地模型 #2812

Closed
goactiongo opened this issue Sep 26, 2024 · 2 comments
Closed

请增加下参数num_ctx用来对接ollama本地模型 #2812

goactiongo opened this issue Sep 26, 2024 · 2 comments

Comments

@goactiongo
Copy link

goactiongo commented Sep 26, 2024

oneapi v0.6.9-alpha版本,增加了2个处理ollama上下文长度和max_token的参数

支持通过max_tokens(对应ollama的num_predict)参数限制输出token数
支持通过num_ctx(ollama原生参数)参数设定更改ollama默认context window大小(默认仅1k/2k上下文)

麻烦fastgpt增加下这个参数,多谢

参考oneapi issue .
songquanpeng/one-api#1694

curl如下
通过curl测试新增加的参数num_ctx,可以正常传递给ollama
curl --location --request POST 'http://172.22.1.39:3001/v1/chat/completions'
--header 'Authorization: Bearer sk-iYrST123'
--header 'Content-Type: application/json'
--data '{
"model": "qwen2:latest",
"stream": false,
"temperature": 1,
"max_tokens": 2000,
"num_ctx":7777,
"messages": [
{
"role": "user",
"content": "你是谁"
}
]
}'

image

@goactiongo goactiongo changed the title 请增加下num_ctx用来对接ollama本地模型 请增加下参数num_ctx用来对接ollama本地模型 Sep 27, 2024
@c121914yu
Copy link
Collaborator

c121914yu commented Sep 28, 2024

参考文档,可以直接添加自定义 body
image

@goactiongo
Copy link
Author

goactiongo commented Sep 28, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants