Repository navigation
fix(chat): reasoning_content parsing, image passthrough, model-change race #104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| ## Fixes | ||
|
|
||
| - **Image attachments now reach every chat model.** The vision gate previously guessed vision support from a hardcoded model-name allow-list (`gpt-4o`, `claude-3`, `gemini`, …), so images were silently stripped for any model whose id didn't match — including custom OpenAI-compatible endpoints (e.g. a local proxy serving `playground-claude-opus-4-8`) and vision-capable on-device models like Gemma. Cairn now always forwards attached images to the model and lets it decide — a non-vision model simply says it can't see them. Images also now flow through the subagent dispatch path, which previously dropped them. | ||
| - **Changing the chat model now applies to the very next message.** A background re-hydrate (triggered after write-tool turns and on every `db:changed` event) could read a stale backend settings cache and silently revert a model you'd just picked — the "have to clear chat twice for the new model to take effect" bug. Refresh hydrates now prefer the live in-memory config over the cache, so a model change sticks immediately. | ||
| - **Qwen/DeepSeek/GLM-style reasoning now shows in the thinking panel on desktop.** The desktop agent stream parsers only read `delta.reasoning`, but these models emit their thinking on `delta.reasoning_content` (and `message.reasoning_content` on non-streaming responses). Cairn now reads `reasoning_content ?? reasoning` in the chat loop, the coding-agent loop, and the subagent dispatch loop — matching mobile — and strips `reasoning_content` before re-sending assistant messages (re-sending it triggers 400s on DeepSeek). | ||
| - **Token counting no longer inflates the context ring for image attachments.** The prompt-token breakdown stringified the whole multimodal message — including the image's base64 data URL — so a single screenshot could add tens of thousands of phantom tokens to the "conversation" segment and spike the context gauge. Images now contribute a flat per-image estimate; only the text parts are tokenised (matching mobile). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| /** | ||
| * Token-breakdown image-content guard. | ||
| * | ||
| * A multimodal user message carries an OpenAI parts array | ||
| * (`[{type:"text"}, {type:"image_url", image_url:{url:"data:...base64..."}}]`) | ||
| * rather than a plain string. `calculatePromptBreakdown` used to stringify that | ||
| * whole array, so the giant base64 data URL was tokenised verbatim — inflating | ||
| * the conversation count (and the context ring) by orders of magnitude. | ||
| * | ||
| * These tests lock in that image parts contribute a small flat estimate, NOT | ||
| * the length of their base64 payload, and that text parts are still counted. | ||
| */ | ||
| import { describe, it, expect } from "vitest"; | ||
| import { calculatePromptBreakdown, type OpenAIMessage } from "./llm"; | ||
|
|
||
| // A ~40KB base64 data URL — realistic for a small screenshot. If this were | ||
| // tokenised, it would be tens of thousands of tokens. | ||
| const BIG_DATA_URL = "data:image/png;base64," + "A".repeat(40_000); | ||
|
|
||
| function multimodalUser(text: string): OpenAIMessage { | ||
| return { | ||
| role: "user", | ||
| content: [ | ||
| { type: "text", text }, | ||
| { type: "image_url", image_url: { url: BIG_DATA_URL } }, | ||
| ], | ||
| } as unknown as OpenAIMessage; | ||
| } | ||
|
|
||
| describe("calculatePromptBreakdown image handling", () => { | ||
| it("does not tokenise the base64 data URL of an image part", () => { | ||
| const b = calculatePromptBreakdown(undefined, [multimodalUser("what is this?")]); | ||
| // The 40KB base64 blob alone would be well over 10k tokens if counted. | ||
| expect(b.conversation).toBeLessThan(2_000); | ||
| }); | ||
|
|
||
| it("still counts the text part of a multimodal message", () => { | ||
| const withText = calculatePromptBreakdown(undefined, [multimodalUser("describe the architecture diagram in detail please")]); | ||
| const noText = calculatePromptBreakdown(undefined, [multimodalUser("")]); | ||
| expect(withText.conversation).toBeGreaterThan(noText.conversation); | ||
| }); | ||
|
|
||
| it("counts a plain-string message the same as before", () => { | ||
| const b = calculatePromptBreakdown(undefined, [ | ||
| { role: "user", content: "hello world" }, | ||
| ]); | ||
| expect(b.conversation).toBeGreaterThan(0); | ||
| expect(b.conversation).toBeLessThan(20); | ||
| }); | ||
|
|
||
| it("is dominated by the flat per-image estimate, not payload size", () => { | ||
| const small = calculatePromptBreakdown(undefined, [ | ||
| { | ||
| role: "user", | ||
| content: [ | ||
| { type: "text", text: "hi" }, | ||
| { type: "image_url", image_url: { url: "data:image/png;base64," + "A".repeat(100) } }, | ||
| ], | ||
| } as unknown as OpenAIMessage, | ||
| ]); | ||
| const big = calculatePromptBreakdown(undefined, [multimodalUser("hi")]); | ||
| // A 400x larger base64 payload must not change the count — images are flat. | ||
| expect(big.conversation).toBe(small.conversation); | ||
| }); | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.