Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Issues ablout openai /v1/chat/completions interface in streaming mode #130

Open
sunxichen opened this issue Sep 13, 2023 · 2 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@sunxichen
Copy link

Issue description:
感觉目前的实现跟openai标准的输出的不太一样:

  1. finish_reason全都是null,即使生成到最后一个字符了也是null,正常应该是"stop"或"length"吧
  2. index全是0
  3. stop参数目前不支持:"The stop parameter is not currently supported"
  4. 在启动服务时,已经设置--eos_id 151645的情况下,生成的内容虽然在<|im_end|>之后终止了,但还是会返回<|im_end|>,正常情况下这个字符不应该返回的吧

Steps to reproduce:
请求示例:
{
"model": "Qwen",
"messages": [
{
"role": "user",
"content": "你好"
}
],
"stream":true,
"top_p":0.7,
"presence_penalty":1.2,
"max_tokens":512
}

返回示例:
data: {"id": "chatcmpl-e93674ffedbb465e9ac7d3b503e29f13", "object": "chat.completion.chunk", "created": 1694575540, "model": "Qwen", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "你好"}, "finish_reason": null}]}

data: {"id": "chatcmpl-e93674ffedbb465e9ac7d3b503e29f13", "object": "chat.completion.chunk", "created": 1694575540, "model": "Qwen", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "!"}, "finish_reason": null}]}

data: {"id": "chatcmpl-e93674ffedbb465e9ac7d3b503e29f13", "object": "chat.completion.chunk", "created": 1694575540, "model": "Qwen", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "很高兴"}, "finish_reason": null}]}

data: {"id": "chatcmpl-e93674ffedbb465e9ac7d3b503e29f13", "object": "chat.completion.chunk", "created": 1694575540, "model": "Qwen", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "为你"}, "finish_reason": null}]}

data: {"id": "chatcmpl-e93674ffedbb465e9ac7d3b503e29f13", "object": "chat.completion.chunk", "created": 1694575540, "model": "Qwen", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "提供"}, "finish_reason": null}]}

data: {"id": "chatcmpl-e93674ffedbb465e9ac7d3b503e29f13", "object": "chat.completion.chunk", "created": 1694575540, "model": "Qwen", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "帮助"}, "finish_reason": null}]}

data: {"id": "chatcmpl-e93674ffedbb465e9ac7d3b503e29f13", "object": "chat.completion.chunk", "created": 1694575540, "model": "Qwen", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "。"}, "finish_reason": null}]}

data: {"id": "chatcmpl-e93674ffedbb465e9ac7d3b503e29f13", "object": "chat.completion.chunk", "created": 1694575540, "model": "Qwen", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "<|im_end|>"}, "finish_reason": null}]}

Expected behavior:

感觉openai接口的功能还没有很完善,后续还会有继续更新的计划吗?

@sunxichen sunxichen added the bug Something isn't working label Sep 13, 2023
@hiworldwzj hiworldwzj added the enhancement New feature or request label Sep 13, 2023
@hiworldwzj
Copy link
Collaborator

@sunxichen 会持续进行更新,多谢你的反馈。

@WANDY666
Copy link
Contributor

index的话,是当有多个choice的时候用来区分和标记的,现在n只支持1,只有一个choice,所以都是0

hiworldwzj added a commit that referenced this issue Jan 8, 2024
Return a string to describe about the finish reason.

Related to #130

Due to in codebase there exists the usage like

```
req.aborted and req.req_status == ReqRunStatus.WAIT_IN_QUEUE
```

It is not suitable to merge `ReqRunStatus` and `FinishStatus`, so I
separate an individual class to indicate the finish reason.

---------

Co-authored-by: hiworldwzj <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants