Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Gemini streaming return being truncated by FinishReason. #1477

Merged
merged 1 commit into from
Jun 13, 2024

Conversation

mxdlzg
Copy link
Contributor

@mxdlzg mxdlzg commented May 28, 2024

close #1408
close #1386

我已确认该 PR 已自测通过,相关截图如下:
(此处放上测试通过的截图,如果不涉及前端改动或从 UI 上无法看出,请放终端启动成功的截图)


理论上早期的版本应该没有返回FinishReason?所以流不会被截断。
但是在v0.6.7左右的版本修复stream流问题时这部分被修正了#1382,引入了FinishReason字段。涉及问题#1345
#1328
但是有个比较尴尬的地方在于,实际上Gemini的API本身就在每个流message中返回STOP(看起来跟早期的Gemini的表现不太一致,不知道为什么会出现这个问题)。
image
1716881148212

因为是流式返回,没法准确预知哪个是最后一条message,只能暂时去掉FinishReason字段了。


启动:

[SYS] 2024/05/28 - 15:26:32 | One API v0.0.0 started 
[SYS] 2024/05/28 - 15:26:32 | running in debug mode 
[SYS] 2024/05/28 - 15:26:32 | SQL_DSN not set, using SQLite as database 
[SYS] 2024/05/28 - 15:26:32 | database migration started 
[SYS] 2024/05/28 - 15:26:32 | database migrated 
[SYS] 2024/05/28 - 15:26:32 | REDIS_CONN_STRING not set, Redis is not enabled 
[SYS] 2024/05/28 - 15:26:32 | using theme default 
[SYS] 2024/05/28 - 15:26:32 | initializing token encoders 
[SYS] 2024/05/28 - 15:26:33 | token encoders initialized 
[DEBUG] 2024/05/28 - 15:27:06 | 2024052815270615146503779860442 | request body: {
    "model": "gemini-pro",
    "temperature": 0,
    "messages": [
        {
            "role": "user",
            "content": "Why sky is blue?"
        }
    ],
    "stream": true
} 
[SYS] 2024/05/28 - 15:27:06 | failed to get token encoder for model gemini-pro: no encoding for model gemini-pro, using encoder for gpt-3.5-turbo 
[INFO] 2024/05/28 - 15:27:06 | 2024052815270615146503779860442 | user 1 has enough quota 499999999963990, trusted and no need to pre-consume 
[DEBUG] 2024/05/28 - 15:27:06 | 2024052815270615146503779860442 | converted request: 
{"contents":[{"role":"user","parts":[{"text":"Why sky is blue?"}]}],"safety_settings":[{"category":"HARM_CATEGORY_HARASSMENT","threshold":"BLOCK_NONE"},{"category":"HARM_CATEGORY_HATE_SPEECH","threshold":"BLOCK_NONE"},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","threshold":"BLOCK_NONE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","threshold":"BLOCK_NONE"}],"generation_config":{}} 
[GIN] 2024/05/28 - 15:27:12 | 2024052815270615146503779860442 | 200 |  6.303667817s |       127.0.0.1 |    POST /v1/chat/completions
[INFO] 2024/05/28 - 15:27:12 | 2024052815270615146503779860442 | record consume log: userId=1, channelId=1, promptTokens=12, completionTokens=337, modelName=gemini-pro, tokenName=Test, quota=1023, content=模型倍率 1.00,分组倍率 1.00,补全倍率 3.00 

@mxdlzg mxdlzg mentioned this pull request May 28, 2024
5 tasks
@mxdlzg mxdlzg marked this pull request as ready for review May 28, 2024 08:05
@manjieqi
Copy link
Contributor

希望能尽快合并

@songquanpeng songquanpeng merged commit b1520b3 into songquanpeng:main Jun 13, 2024
@songquanpeng
Copy link
Owner

Thx~

sunls24 pushed a commit to sunls24/one-api that referenced this pull request Jun 22, 2024
SheldonLiu0412 pushed a commit to SheldonLiu0412/one-api that referenced this pull request Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Gemini的流式返回存在问题 v0.6.6-alpha.11 Gemini回复被截断
3 participants