feat(local-inference): stream IMAGE_DESCRIPTION token-by-token through the chat pipe (#9105) - #9289
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
e4a195d to
0785f13
Compare
There was a problem hiding this comment.
Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.
…h the chat pipe (#9105) Route the local vision describe (IMAGE_DESCRIPTION) onto the SAME onTextChunk -> onStreamChunk -> SSE -> frontend streaming pipe as chat text, so a vision description renders token-by-token in the dashboard. Gated on a new fused ABI-v13 streaming-describe entrypoint with graceful fallback to the buffered one-shot describe on <=v12 libs (the cascade probes visionStreamSupported()). - core: additive stream?/onStreamChunk? on ImageDescriptionParams + TranscriptionParams (the runtime already injects onStreamChunk into local model params when a chat streaming context is active). - ffi-bindings: ABI v13 cascade attempt + visionStreamSupported() + describeImageStreamOpen(); degrades to the buffered describeImage on <=v12 libs. - desktop-fused-ffi-backend-runtime: describeImage() decodes token-by-token via describeImageStreamOpen + the EXISTING llmStreamNext loop when onTextChunk is set. - thread onTextChunk + maxTokensPerStep through backend -> engine -> service (arbiter vision capability) -> the IMAGE_DESCRIPTION provider handler. Validated live on Windows CPU with SmolVLM-500M: token-by-token describe with OCR (read "ELIZA OCR" / "Total: $42.00" off a rendered image). The native ABI-v13 describe_image_stream_open lands separately to the elizaOS/llama.cpp fork; this JS change is safe before it (graceful degradation). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0785f13 to
0b836b7
Compare
There was a problem hiding this comment.
Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
…oken vision describe) (#9507) Points the fused-lib submodule at the fork commit that adds eliza_inference_describe_image_stream_open (ABI 12->13), on top of develop's current pinned base (keeps the iOS fused-slice + diarizer fixes). Activates the JS vision streaming landed in #9289 (it gracefully fell back to one-shot describe until now). Validated on Windows CPU: real eliza-1-0_8b bundle + published mmproj streams token-by-token with OCR. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
What
Routes the local vision describe (
IMAGE_DESCRIPTION) onto the same streaming pipe as chat text —onTextChunk → onStreamChunk → SSE → frontend— so a vision description renders token-by-token in the dashboard instead of arriving as one blob.Gated on a new fused ABI-v13 streaming-describe entrypoint, with graceful fallback to the buffered one-shot
describe_imageon ≤v12 libs (the JS cascade probesvisionStreamSupported()), so this change is safe to land before the native side ships.Changes
types/model.ts): additivestream?/onStreamChunk?onImageDescriptionParams+TranscriptionParams. The runtime already injectsonStreamChunkinto local model params when a chat streaming context is active (runtime.ts).visionStreamSupported()+describeImageStreamOpen(); degrades todescribeImageon ≤v12.describeImage()decodes token-by-token viadescribeImageStreamOpen+ the existingllmStreamNextloop whenonTextChunkis set (no event-loop blocking — yields between steps); else the buffered path.onTextChunk+maxTokensPerStepthroughbackend → engine → service(arbiter vision capability) → theIMAGE_DESCRIPTIONprovider handler.Validation
Validated live on Windows CPU with SmolVLM-500M (mtmd): token-by-token describe with real OCR — read "ELIZA OCR" / "Total: $42.00" off a rendered image, streaming 256 chunks at ~21 tok/s. Evidence (video + stills) attached to #9105.
The native
eliza_inference_describe_image_stream_open(ABI v13, reusesllm_stream_next) lands separately to theelizaOS/llama.cppfork + a gitlink bump; this JS change degrades gracefully until then.Part of the "every model — including vision — through the same streaming pipe" effort (#9105).
🤖 Generated with Claude Code