Skip to content

fix: preserve local image attachment metadata across vision fallback paths - #34315

Open
sunzd-1997 wants to merge 3 commits into
NousResearch:mainfrom
sunzd-1997:feat-fix-image-attachment-metadata
Open

fix: preserve local image attachment metadata across vision fallback paths#34315
sunzd-1997 wants to merge 3 commits into
NousResearch:mainfrom
sunzd-1997:feat-fix-image-attachment-metadata

Conversation

@sunzd-1997

Copy link
Copy Markdown

What does this PR do?

This PR preserves deterministic local image attachment metadata across Hermes image routing paths.

Gateways such as Feishu may correctly download an image into the local Hermes cache, but the agent-visible text does not always retain stable local file metadata. When the main model does not support native vision, or when vision_analyze fails or raises, downstream skills/tools can lose the exact local path they need to upload or reprocess the cached image.

This PR adds a machine-readable attachment block for local images:

[Hermes image attachment]
local_image_path=...
size_bytes=...
mime_type=...
[/Hermes image attachment]

Models still receive native image parts when supported. Text fallback now keeps the local cached image path, byte size, and MIME type before vision analysis runs, so the metadata remains available whether vision succeeds, fails, or throws.

This is not Feishu-specific. It makes local image attachments deterministic for any gateway, CLI, TUI, skill, or tool workflow that needs to operate on the exact cached file.

Related Issue

Fixes #

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • agent/image_routing.py

    • Added image_attachment_metadata(path) to collect stable local image metadata.
    • Added format_image_attachment_block(image_paths) to emit machine-readable attachment blocks.
    • Updated native image content parts to append local metadata blocks while continuing to pass native image_url parts.
    • Preserved existing remote image URL hint behavior.
  • gateway/run.py

    • Injects local image attachment metadata before running vision_analyze, so metadata survives success, failure, and exception paths.
  • cli.py

    • Applies the same metadata injection behavior for CLI image preprocessing.
  • tui_gateway/server.py

    • Applies the same metadata injection behavior for TUI gateway image preprocessing.
  • tests/agent/test_image_routing.py

    • Covers metadata block formatting and native image routing behavior.
  • tests/gateway/test_image_attachment_metadata.py

    • Adds regressions for gateway text fallback when vision succeeds, fails, or raises.
  • tests/hermes_cli/test_kanban_worker_image_extraction.py

    • Updates the direct image routing expectation to assert the new local metadata block.

How to Test

  1. Attach or send a local image through a text fallback path where the main model does not receive native vision input.

  2. Verify the agent-visible text includes the attachment block with local_image_path, size_bytes, and mime_type.

  3. Run the focused regression suite:

/Users/sunyusong1/.hermes/hermes-agent/venv/bin/python -m pytest tests/agent/test_image_routing.py tests/gateway/test_image_attachment_metadata.py tests/tools/test_clipboard.py::TestPreprocessImagesWithVision::test_vision_failure_includes_path tests/tools/test_clipboard.py::TestPreprocessImagesWithVision::test_vision_exception_includes_path tests/hermes_cli/test_kanban_worker_image_extraction.py

Local result: 92 passed.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Focused regression suite: 92 passed

@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation P2 Medium — degraded but workaround exists labels May 29, 2026
sunyusong1 and others added 2 commits June 2, 2026 17:51
Move npm install, uv sync, playwright install, and frontend build out of
the Dockerfile and into a new cont-init.d/00-bootstrap-deps script that
runs on the first container boot. The Docker build now only installs
system tools, s6-overlay, and copies source code, completing in < 5 min
instead of timing out at 20–35 min on the licloud CI platform.

- Remove all COPY manifest + npm install + uv sync + frontend build layers
- Add docker/cont-init.d/00-bootstrap-deps: first-boot install script
  with .bootstrap_done marker for fast subsequent starts
- stage2-hook.sh: guard .venv/node_modules/.playwright chowns with [ -d ]
  since they no longer exist at build time; add .venv guard to skills_sync;
  remove ui-tui references (no frontend in this deployment)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for preserving attachment context across the gateway, CLI, and TUI text-preprocessing paths.

Problems

  • agent/image_routing.py's proposed format_image_attachment_block writes raw paths into newline-delimited fields. A valid POSIX filename can contain a newline, which makes the claimed machine-readable format ambiguous and can forge subsequent fields. Use an escaped/structured representation and cover that input.
  • The stated cross-fallback guarantee is incomplete: run_agent.py:5045-5074 sends non-vision image content through _describe_image_for_anthropic_fallback(), whose output at run_agent.py:4905-4909 has only a prose path note. This PR does not add size/MIME metadata there.

Suggested changes

  • Serialize attachment metadata in a robust structured format and test newline-containing paths.
  • Cover the run_agent.py fallback or narrow the declared scope; add a regression through tests/run_agent/test_vision_aware_preprocessing.py.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants