fix(telegram): fail fast when user images cannot be accessed (rebase for moved file)

[HiddenPuppy]

Summary

Rebase of PR #22413 onto current main, resolving the file-move conflict.

The original PR modified gateway/platforms/telegram.py which no longer exists on main — the Telegram adapter was moved to plugins/platforms/telegram/adapter.py. This PR transplants the MIME fix to the new location.

Changes

File	Change
plugins/platforms/telegram/adapter.py	Normalize MIME type image/jpg → image/jpeg using standard MIME types (transplanted from old gateway/platforms/telegram.py)
gateway/run.py — _build_media_placeholder	Return neutral fallback when no media_urls exist, preventing empty user input
gateway/run.py — _enrich_message_with_vision	Track all_failed across all images. When ALL analyses fail AND the user provided no caption, return a clear fail-fast message
tests/gateway/test_vision_memory_leak.py	Add 4 tests: all-failed/no-caption, all-failed/with-caption, exception/no-caption, mixed success/failure

Addresses Review Feedback

Per @teknium1's review on PR #22413:

• ✅ MIME fix transplanted to plugins/platforms/telegram/adapter.py
• ✅ 4 new regression tests added in test_vision_memory_leak.py

Fixes #22385
Closes #22413 (replaced by this rebased version)

[teknium1]

Thanks for rebasing the fix onto the moved Telegram adapter. The no-caption fail-fast branch addresses a behavior that still exists on current main: gateway/run.py:15329-15340 tells the model to retry vision_analyze after a failed analysis, and text-mode image routing calls that helper at gateway/run.py:10566-10574.

Problems

• The new captioned-failure test says it returns only the original user text, but the changed condition only returns the fail-fast response when not user_text; with a caption, the existing failure placeholders remain. Align the test description/assertions with the intended behavior, or change the implementation accordingly.
• plugins/platforms/telegram/adapter.py:8132 is the direct .jpg MIME emission site, but the PR adds no regression test for the new image/jpeg normalization.

Suggested changes

• Add a .jpg photo-handler test asserting event.media_types == ["image/jpeg"].
• Make the captioned all-failure test assert the actual intended output contract.

Automated hermes-sweeper review.

[egilewski]

suggesting changes

The all-failed captioned-image path still preserves the retry loop this change is meant to stop. With vision_analyze_tool returning {"success": false} and user_text="please describe this", _enrich_message_with_vision() still returns:

You can try looking at it yourself with vision_analyze using image_url: /cache/img.jpg

followed by the caption. That happens because the new fail-fast branch is gated by not user_text.strip(). It leaves a common Telegram request such as a captioned “please describe this” image telling the agent to retry the same unavailable analysis.

The new test_all_failed_with_caption_returns_original docstring says there should be “no enriched parts,” but the assertions only check that the caption is present and the new fail-fast phrase is absent, so the old retry instruction passes unnoticed. Please suppress the retry hint/cache path whenever all analyses fail (while preserving the caption as appropriate), and assert that the captioned result contains neither vision_analyze nor the image path.

Security evidence:

• trust boundary: a failed Telegram image analysis becomes model-visible gateway input.
• source/sink/invariant: all failed analyses must not return an instruction to retry the same unavailable image.
• current-main reproduction: failed captioned analysis returns the existing vision_analyze retry instruction.
• PR-head or patch-replay validation: the commits replay cleanly on current main, but failed captioned analysis still returns that instruction and cache path.
• positive/negative cases: 7 focused tests pass; the explicit failed-analysis probe shows no-caption is fixed and captioned input is not.
• residual bypass search: every non-whitespace caption bypasses the new all_failed fail-fast branch.
• reviewer validation: the probe imported the replayed gateway.run and checked the exact captioned and uncaptioned outputs.

Signed: GPT-5.6-sol-xhigh in Codex

[GottZ]

This was generated by AI during triage.

Summary

Fourteen PRs address or reference this issue complex across four distinct causes: verbose or capped auxiliary-vision output, lost system prompts, transient auto-vision failures, and inaccessible-image retry loops. The diffs range from internal concise preprocessing and provider-cap propagation (#10852), persona forwarding (#25656), retries (#28999/#29263), fail-fast handling (#22413/#63635), user-facing token configuration (#15430/#29600/#38004/#73438/#74939/#74945), and removal of vision caps (#75253).

Related pull requests

• fix(gateway): keep auto vision preprocess concise #10852 best fix — (+139/-5) — n/a: The diff shortens automatic gateway descriptions, applies an internal 500-token cap, and preserves that explicit cap through custom-provider, retry, and fallback requests with wire-level tests; it is the recorded best fix for Default vision pre-process prompt generates overly long descriptions (~2000 chars), significantly slowing down image-bearing requests on local models #10809 and [Bug/Enhancement] Hardcoded max_tokens and overly verbose prompt in vision_tools.py causes severe latency for reasoning models #29590, and the earlier keep-open concern was addressed in commit 1f898ab.
• feat(tools): read vision max_tokens from auxiliary.vision config #15430 [closed] partial — (+18926/-3839) — n/a: This closed PR reads auxiliary.vision.max_tokens from configuration, but remains relevant as the broad predecessor of the config-knob proposals and was closed as not planned under the documented max-tokens-knob policy.
• fix(telegram): fail fast when user images cannot be accessed #22413 duplicate — (+26/-2) — n/a: This open PR adds failed-analysis fail-fast handling and a media placeholder, but targets the moved Telegram adapter and lacks the requested regression tests, so it is superseded by fix(telegram): fail fast when user images cannot be accessed (rebase for moved file) #63635.
• fix(vision): preserve auxiliary system prompts #25656 best fix — (+143/-8) — n/a: The diff forwards session system prompts into auxiliary vision and keys Anthropic fallback caching by prompt, making it the recorded best fix for [Bug]: Vision fallback (auxiliary.vision) ignores SOUL.md system prompt — session context lost during image analysis #24697, but the keep-open review identifies failing bare-agent tests, stale gateway threading, and unthreaded CLI/TUI paths requiring author action.
• fix(gateway): retry auto-vision on transient failure (#28972) #28999 [closed] best fix — (+617/-6) — n/a: The closed recorded best fix for [Bug]: [Discord] Auto-vision returns success=false on attachments, forcing duplicate vision_analyze call per image #28972 adds classified bounded retries and configuration-backed coverage; despite its earlier keep-open review, the later automated close verdict cites equivalent transient and empty-response retries on main in fb403a3a730f2a4eab1cfc60ffe854f7ea7c4159.
• fix(discord): retry auto-vision once after first failure to avoid duplicate vision_analyze (#28972) #29263 [closed] fixes — (+40/-0) — n/a: This closed PR hardcodes one delayed retry for the same [Bug]: [Discord] Auto-vision returns success=false on attachments, forcing duplicate vision_analyze call per image #28972 failure and remains relevant as the smaller duplicate that its author closed in favor of fix(gateway): retry auto-vision on transient failure (#28972) #28999.
• fix(vision): honor auxiliary.vision.max_tokens from config in vision_tools #29600 [closed] best fix — (+47/-2) — n/a: This closed recorded best fix for [Bug/Enhancement] Hardcoded max_tokens and overly verbose prompt in vision_tools.py causes severe latency for reasoning models #29590 reads image and video token caps from auxiliary.vision with focused tests, but it duplicates feat(tools): read vision max_tokens from auxiliary.vision config #15430's user-facing config approach and was closed by a contributor as no longer relevant.
• fix: respect auxiliary.vision.max_tokens from config.yaml (#29590) #38004 [closed] partial — (+22/-4) — n/a: This closed PR implements the same configurable image/video caps, plus an unrelated conversation-loop backoff, and remains relevant as a later duplicate closed with a reference to main commit db472f5df.
• Cap gateway vision preprocessing output #61187 [closed] fixes — (+145/-26) — n/a: This closed duplicate of fix(gateway): keep auto vision preprocess concise #10852 adds the same concise 500-token gateway path and provider propagation, but changes general explicit-cap behavior more broadly; contributor triage identified fix(gateway): keep auto vision preprocess concise #10852 as the earlier PR.
• fix(telegram): fail fast when user images cannot be accessed (rebase for moved file) #63635 best fix — (+62/-2) — n/a: This recorded best fix for Telegram: images sent by user are not accessible — agent spends 30+ min failing to read them #22385 rebases fix(telegram): fail fast when user images cannot be accessed #22413 onto the active Telegram adapter and adds fail-fast tests, but the keep-open verdict and contributor review block completion until captioned all-failure output removes the retry hint/path and .jpg normalization has a handler regression test.
• feat(vision): make max_tokens configurable via auxiliary.vision.max_tokens #73438 [closed] partial — (+13/-2) — n/a: This closed PR exposes configurable caps for vision and browser vision with documentation, and remains relevant as a policy-rejected predecessor to the no-config-knob re-scope in fix(vision): stop capping aux vision output with hardcoded max_tokens #75253.
• fix(vision): read max_tokens from auxiliary.vision config instead of hardcoded 2000 #74939 [closed] partial — (+14/-4) — n/a: This closed PR duplicates the configurable image/browser cap change and remains relevant only as the stale predecessor superseded first by fix(vision): read max_tokens from auxiliary.vision config instead of hardcoded 2000 #74945 and then by fix(vision): stop capping aux vision output with hardcoded max_tokens #75253.
• fix(vision): read max_tokens from auxiliary.vision config instead of hardcoded 2000 #74945 [closed] partial — (+14/-4) — n/a: This closed successor still adds the prohibited config knob, but its discussion documents the policy-compliant re-scope that became fix(vision): stop capping aux vision output with hardcoded max_tokens #75253.
• fix(vision): stop capping aux vision output with hardcoded max_tokens #75253 related — (+17/-3) — n/a: The diff removes hardcoded caps from image, video, and browser-vision calls without adding configuration and now asserts their absence in all three kwargs-capture tests, directly addressing the visible keep-open review while targeting truncation rather than Default vision pre-process prompt generates overly long descriptions (~2000 chars), significantly slowing down image-bearing requests on local models #10809's verbosity latency.

Duplicates

#22413 is superseded by #63635; #61187 substantially duplicates #10852; #29263 duplicates #28999; #15430, #29600, #38004, #73438, #74939, and #74945 share the rejected user-facing auxiliary.vision.max_tokens approach, with #74939 superseded by #74945 and #74945 re-scoped as #75253.

Suggested consolidation

Keep #63635 open with a salvage path, consistent with its automated keep-open verdict: the author should suppress vision_analyze and cached-path retry hints for captioned as well as uncaptioned all-failure cases, preserve the caption, and add the requested .jpg handler test before re-review; this explicitly addresses the contributor's blocking objection rather than merging over it. Close #22413 as a duplicate of #63635; retain #10852, #25656, and #75253 only on their distinct salvage paths, with #25656 requiring the listed test and call-site repairs, and leave the already-closed config-knob and retry duplicates closed.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I22385(["issue #22385 (closed)"])
    I28972(["issue #28972 (open)"])
    subgraph Dup22413 ["PRs duplicating each other"]
        P22413["PR #22413 (open)"]
        P63635["PR #63635 (open)"]
    end
    P63635 -->|best fix| I22385
    P63635 -.->|partial| I28972
    class I22385 closed
    class I28972 open
    class P22413 open
    class P63635 open
    class P63635 best
    class P63635 target
    click I22385 "https://github.com/NousResearch/hermes-agent/issues/22385"
    click I28972 "https://github.com/NousResearch/hermes-agent/issues/28972"
    click P22413 "https://github.com/NousResearch/hermes-agent/pull/22413"
    click P63635 "https://github.com/NousResearch/hermes-agent/pull/63635"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 14 pull requests and 5 issues in this complex. Diffs were read for 13 of 14 PRs (rest unavailable); Assessment working set: 110 kB of PR diffs, 48 kB of issue/PR text, 15 kB of discussion (22 comments), 29 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.