fix: image generation with nearai models - #1819
Conversation
github-actions
Bot
added
scope: agent
f5c6c42 to
e1d8bd3
Compare
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements a comprehensive system for handling, persisting, and displaying generated images within the agentic loop and web interface. Key changes include the introduction of a GeneratedImageSentinel to manage image metadata, logic to omit large base64 data URLs from LLM context while preserving them for frontend reconstruction, and automatic media type inference (JPEG, PNG, GIF, WebP) for generated assets. The web frontend was updated to cache generated images per thread and handle history reconstruction more effectively. Review feedback highlights opportunities to optimize frontend memory usage, refine scrolling behavior during history loads, and ensure consistent state synchronization during pagination.
zmanian
left a comment
zmanian
left a comment
There was a problem hiding this comment.
Review -- COMMENT (leaning APPROVE)
Well-structured fix for image persistence and LLM context pollution. Good test coverage (7 new tests). URL construction fix is correct. Both DB backends work without schema changes.
Medium (worth addressing)
- Unbounded image storage in DB:
tool_result_content_for_persistencestores full base64 data URLs (1-5 MB each) with no truncation, unlike other tool results (1000 char limit). Acknowledge the storage growth implication or add a size cap. - Duplicated
endpoint_url(): Exact same logic inImageGenerateToolandImageEditTool. Extract to a shared helper.
Low (non-blocking)
- Frontend
generatedImagesByThreadMap grows unbounded (cleared on page refresh) - DOM dedup compares full data URL strings (a Set of IDs would be cleaner)
normalize_embedded_jsonlimit of 3 parse rounds is undocumented
github-actions
Bot
added
contributor: regular
hanakannzashi
changed the title
# Conflicts: # src/channels/web/handlers/chat.rs # src/lib.rs
zmanian
left a comment
zmanian
left a comment
There was a problem hiding this comment.
Re-review -- APPROVE
The three new commits (dce9e8cb, a02a87bc, 89b3a911) address all medium-priority items from the previous review and most low-priority ones:
Previously raised -- status
| Item | Status |
|---|---|
| Unbounded image storage in DB | Fixed -- MAX_PERSISTED_IMAGE_SENTINEL_BYTES (8 MiB) cap with clear fallback message |
Duplicated endpoint_url() |
Fixed -- extracted to image_api_endpoint_url() in src/tools/builtin/mod.rs |
Frontend generatedImagesByThread unbounded |
Fixed -- LRU eviction with GENERATED_IMAGE_THREAD_CACHE_CAP = 20 / GENERATED_IMAGES_PER_THREAD_CAP = 8 |
| Scroll-per-image during history load | Fixed -- addGeneratedImage(url, path, false) + single scroll after loop |
normalize_embedded_json limit undocumented |
Fixed -- comment explaining the 3-round cap |
Pagination not calling rememberGeneratedImage |
Fixed -- older-history path now calls rememberGeneratedImage |
Redundant renderGeneratedImagesForCurrentThread() |
Removed; replaced with a single container.scrollTop = container.scrollHeight |
| DOM dedup via full data URL string comparison | Not addressed (low priority, acceptable for now) |
New changes look correct
ironclaw_safety::SafetyLayerimport migration indispatcher.rsfollows the project convention (CLAUDE.md: "new code should import fromironclaw_safetydirectly").- Module declaration ordering fix in
lib.rsis cosmetic and correct. - The merge from staging resolved cleanly based on the diff.
One minor observation (non-blocking)
infer_generated_image_media_type decodes only the first 64 characters of base64 (not 64 bytes). Due to base64 encoding ratios, this yields ~48 decoded bytes, which is enough for all four magic-number checks (JPEG needs 3, PNG needs 8, GIF needs 6, WebP needs 12). Just noting this is fine as-is.
Good work addressing the feedback thoroughly.
# Conflicts: # src/agent/dispatcher.rs
There was a problem hiding this comment.
Pull request overview
Fixes image generation and rendering across the tool execution layer, persisted history reconstruction, SSE eventing, and the web UI—specifically addressing NearAI STG /v1/v1 endpoint construction issues and ensuring generated images persist and render correctly after refresh.
Changes:
- Centralized image API endpoint URL construction to avoid duplicated
/v1and updated the default fallback image model. - Standardized “image_generated” sentinel parsing/handling end-to-end (agent ↔ persistence ↔ web history), including summarizing tool results to avoid feeding base64/data URLs back into the model.
- Updated web UI history rendering and CSS so generated image cards render as stable standalone blocks and don’t collapse when additional assistant messages are appended.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/tools/builtin/mod.rs | Adds shared helper to build image API endpoints without /v1 duplication. |
| src/tools/builtin/image_gen.rs | Uses shared endpoint helper; infers media type from base64 prefix; emits data URLs with detected media type; adds tests. |
| src/tools/builtin/image_edit.rs | Uses shared endpoint helper for edits/fallback; applies media-type inference; adds tests. |
| src/lib.rs | Registers new internal generated_images module. |
| src/generated_images.rs | Introduces GeneratedImageSentinel parser/normalizer to reliably detect sentinels across stringification layers. |
| src/channels/web/util.rs | Collects generated images from persisted tool results; adds tool result preview that hides sentinel payloads. |
| src/channels/web/types.rs | Extends TurnInfo with generated_images and defines GeneratedImageInfo for API responses. |
| src/channels/web/static/style.css | Adjusts generated image card/layout styling to prevent collapse and keep sizing stable. |
| src/channels/web/static/app.js | Renders generated image blocks from SSE + history; adds de-dupe/caching helpers and scroll behavior changes. |
| src/channels/web/server.rs | Uses new preview + generated image collection when building history responses. |
| src/channels/web/handlers/chat.rs | Same as server history path: uses preview + generated image collection for responses. |
| src/app.rs | Updates fallback image model to black-forest-labs/FLUX.2-klein-4B. |
| src/agent/thread_ops.rs | Persists full image sentinel payloads (with size cap) and summarizes sentinel tool results when rebuilding model context. |
| src/agent/dispatcher.rs | Uses shared sentinel detection; emits image_generated; replaces tool-message content with a summary that omits base64/data URLs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
think-in-universe
changed the title
Co-authored-by: Robert Yan <46699230+think-in-universe@users.noreply.github.com>
Summary
/v1/v1/...URL construction and switching the fallback image model toblack-forest-labs/FLUX.2-klein-4B.image_generatedevents, and avoid feeding raw base64/data URLs back into the model.generated_imagesso images survive page refresh and render as standalone UI blocks.Change Type
Linked Issue
Fixes #1810
Validation
cargo fmt --all -- --checkcargo clippy --all --benches --tests --examples --all-features -- -D warningscargo buildcargo test -q test_build_turns_collects_generated_images_from_persisted_tool_results --libcargo test -q test_rebuild_chat_messages_summarizes_image_generated_result --libcargo test -q test_image_generation_summary_tool_message_omits_data_url --libcargo test -q test_parse_image_generated_sentinel_accepts_stringified_json --libcargo test -q parses_double_stringified_sentinel --libcargo test --features integrationif database-backed or integration behavior changed<img src="data:...">review-prorpr-shepherd --fixwas run before requesting reviewSecurity Impact
None. No auth, secrets, sandbox, or permission model changes. This PR does stop raw generated-image base64 from being echoed back into model-visible tool results.
Database Impact
None. No schema or migration changes. This does change persisted
tool_calls.resultcontent for generated-image tool results so history can reconstruct images, but it remains within existing JSON/text storage and is compatible with both PostgreSQL and libSQL.Blast Radius
Touches image tool execution, agent tool-result handling, persisted tool-call history, and web chat/history rendering. Regressions would most likely affect generated-image display, tool-result previews, or refresh/history reconstruction for image turns.
Review Follow-Through
Review track: B