-
Notifications
You must be signed in to change notification settings - Fork 22.2k
fix(metadata): Some providers use requestbody(body.metadata.user_id) to enable cache features Fixes #8195 #11276
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
e1e356c
7afbc51
00637c0
2627831
8d7c2fb
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,3 @@ | ||
| import os from "os" | ||
| import { Installation } from "@/installation" | ||
| import { Provider } from "@/provider/provider" | ||
| import { Log } from "@/util/log" | ||
|
|
@@ -9,7 +8,6 @@ import { | |
| type StreamTextResult, | ||
| type Tool, | ||
| type ToolSet, | ||
| extractReasoningMiddleware, | ||
| tool, | ||
| jsonSchema, | ||
| } from "ai" | ||
|
|
@@ -64,6 +62,13 @@ export namespace LLM { | |
| Provider.getProvider(input.model.providerID), | ||
| Auth.get(input.model.providerID), | ||
| ]) | ||
| if (provider.options?.enableMeta) { | ||
| input.model.options.metadata = { | ||
| user_id: `user_${Instance.project.id ?? "unknown"}_account__session_${input.sessionID}`, | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Currently, the purpose of this combination is to simulate the behavior in the Claude Code CLI, which is why this approach is adopted
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If it's a variable name, it could indeed be optimized to |
||
| project_id: Instance.project.id, | ||
| session_id: input.sessionID, | ||
| } | ||
| } | ||
| const isCodex = provider.id === "openai" && auth?.type === "oauth" | ||
|
|
||
| const system = [] | ||
|
|
@@ -261,7 +266,6 @@ export namespace LLM { | |
| return args.params | ||
| }, | ||
| }, | ||
| extractReasoningMiddleware({ tagName: "think", startWithReasoning: false }), | ||
| ], | ||
| }), | ||
| experimental_telemetry: { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps "the train has already left the station", but I would prefer not adding opencode specific headers that give away where my traffic is coming from like this. Also, should we be using a metadata
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This addresses the issue where many model providers cannot properly enable caching functionality. I attempted to add parameters like user_id in the metadata.
However, the model providers cannot recognize them correctly, including AWS Claude Code and several third-party provided Claude models.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think some API endpoint like AWS bedrock/Azure are alao require this fix to get better prompt cache support
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, I ran into this issue while using AWS