fix(gateway): salvage MEDIA: false-positive guards (#16721 + #35699)

[kshitijk4poor]

Summary

User-requested batch salvage of two tiny, related MEDIA: false-positive fixes for the gateway. Both target the same problem class — MEDIA: example strings in ordinary text getting mis-delivered as real attachments — on two different code paths.

Original PRs salvaged (commits cherry-picked, original authorship preserved):

• fix(gateway): restrict auto-appended media to producer tools #16721 by @VinciZhu — restrict the gateway auto-append path to producer tools (gateway/run.py)
• fix(gateway): skip MEDIA: tags inside code blocks and blockquotes #35699 by @liuhao1024 — skip MEDIA: tags inside fenced code blocks, inline code, and blockquotes (gateway/platforms/base.py)

What each fix does

#16721 — producer-tool allowlist for auto-append (gateway/run.py)

When the model omits a MEDIA: tag from its final reply, the gateway scans the turn's tool results and auto-appends any media it finds. Previously it scanned all tool output, so a literal MEDIA:/path/... example string in skill_view docs, logs, or search results could be delivered as a bogus attachment. This adds _collect_auto_append_media_tags() keyed by a producer-tool allowlist (text_to_speech), so only tools that actually emit deliverables are eligible.

#35699 — mask protected spans in extract_media (gateway/platforms/base.py)

extract_media ran MEDIA_TAG_CLEANUP_RE over the raw reply text, so an example MEDIA:/path.png inside a ``` fenced block, `inline code`, or a > blockquote was extracted as a real attachment. This adds _mask_protected_spans() which blanks those regions (preserving offsets) before matching, while keeping backtick-quoted MEDIA: paths working.

Current-main reconciliation (in #16721's cherry-pick)

#16721 was opened against an older gateway/run.py. Current main has since gained the #34608 current-turn slice isolation and a compression-safe fallback (#160). The cherry-pick was reconciled so the salvaged helper keeps both:

• current-turn isolation via history_offset ([Bug]: MEDIA tag from stale tool output in session history leaks into later text-only responses #34608),
• the compression-safe fallback that scans the full message list (relying on _history_media_paths dedup) when mid-run compression shrinks the list below the history length (Telegram: All voice messages are attached to replies and send multiple times #160),
• the extension-anchored _TOOL_MEDIA_RE already on main (instead of the looser MEDIA:(\S+) in the original PR),

and layers on #16721's producer-tool allowlist as the primary guard. Net result is strictly tighter than current main with no regression to the compression or slicing behavior.

Follow-up commit on top

• chore(release): add @VinciZhu (haaasined@gmail.com) to scripts/release.py AUTHOR_MAP so the contribution is credited in release notes. (@liuhao1024 was already mapped.)

Verification

• tests/gateway/test_platform_base.py + tests/gateway/test_media_extraction.py — 139 passed, 2 skipped (both PRs' new tests included).
• Import-level E2E smoke covering: code-block/inline MEDIA: not extracted, real prose tag still extracted; non-producer tool output not auto-appended, real TTS output still auto-appended and preserved under the compression-safe fallback path.

Scope

Two concerns, both narrow and in the same subsystem; touched files limited to the two production files, their two test files, and the AUTHOR_MAP. A third related PR (#34388, serialized-JSON MEDIA: boundary) is intentionally not in this batch — it will be handled separately because its line-start-anchor approach needs reworking against current main.

[kshitijk4poor]

CI note: the two red shards (test (1), test (3)) are pre-existing failures unrelated to this PR, confirmed by checking out this branch's base commit (f106e58af, zero changes from here) and reproducing both:

• test (1) → test_model_catalog.py::test_in_repo_lists_match_manifest — website/static/api/model-catalog.json is out of sync with _PROVIDER_MODELS['nous'] / OPENROUTER_MODELS (needs scripts/build_model_catalog.py regen). This PR does not touch the catalog or model lists.
• test (3) → test_gui_command.py::{test_gui_installs_packages_and_launches_desktop_app, test_gui_forwards_desktop_environment_overrides} — StopIteration in the desktop-launch mocks. This PR does not touch any GUI/desktop code.

This PR only changes gateway/run.py, gateway/platforms/base.py, and the two gateway test files (+ AUTHOR_MAP). The gateway media test suite — tests/gateway/test_platform_base.py + tests/gateway/test_media_extraction.py — is 139 passed, 2 skipped locally, and e2e, ruff, check-attribution, and the nix builds are all green.

[tonydwb]

Review: APPROVE ✅

This PR salvages two related fixes for MEDIA: false-positive guards in the gateway, cherry-picked from #16721 and #35699 with current-main reconciliation.

Correctness

• _mask_protected_spans(): correctly masks fenced code blocks, inline code, and blockquotes before MEDIA tag extraction, while preserving backtick-quoted paths in MEDIA: tags. Character count is preserved so regex offsets stay valid.
• extract_media(): masked content is used for both matching and cleaning, preventing bogus extractions while keeping real tags working.
• _collect_auto_append_media_tags(): extracts inline scan logic into a named function with proper tool-name-to-call-ID mapping, producer-tool allowlist (text_to_speech/text_to_speech_tool), current-turn isolation via history_offset, and a compression-safe fallback path. All three concerns from the original code are preserved or improved.

Security

No new security concerns. The existing filter_local_delivery_paths handles path safety.

Code Quality

• Clean refactoring — the inline scan block in gateway/run.py is extracted into a well-documented helper function.
• _TOOL_MEDIA_RE now compiled once at module level (previously recompiled on every scan loop iteration).
• Edge case handling for backtick-quoted MEDIA: paths inside inline code is careful and correct.

Testing

• 7 new tests across test_platform_base.py and test_media_extraction.py covering:
  • MEDIA in fenced code blocks ignored
  • MEDIA in inline code ignored
  • MEDIA in blockquotes ignored
  • Real MEDIA outside protected regions still extracted
  • Mixed code+prose extraction
  • Skill docs not auto-appended
  • Real TTS output still auto-appended
• All existing tests pass (139 passed, 2 skipped).

Performance

No concerns. _mask_protected_spans() does a few linear passes over reply text, acceptable for typical message lengths.

Documentation

• PR body is thorough with per-fix breakdown, reconciliation notes, and verification results.
• Code docstrings reference the relevant PRs.
• AUTHOR_MAP updated to credit the original contributor.

Verdict

Clean fix, well-tested, well-documented. Approving.

[kshitijk4poor]

Self-review (hermes-pr-review)

Ran the regex-defense review workflow (empirical false-positive + bypass sweeps against the live functions, plus an adversarial correctness pass). One real bug found and fixed in this PR; remaining items dispositioned below.

🔴 Fixed in this PR (commit "keep code blocks verbatim in cleaned text when media present")

extract_media blanked code blocks/inline-code/blockquotes in the delivered text. The cleanup path did cleaned = media_pattern.sub('', _mask_protected_spans(cleaned)) — it assigned the masked (space-blanked) copy back to cleaned. So whenever a real MEDIA: tag was delivered (the if media: branch), every fenced code block, inline `code`, and > blockquote in the reply was wiped to whitespace in the user-visible message.

The existing tests missed it because all the code-block cases had empty media, which skips the if media: branch. Empirical repro before the fix:

input:   "Here is your file:\nMEDIA:/real/out.png\nExample code:\n```python\nprint(\"hello\")\n```\nThanks!"
cleaned: 'Here is your file:\n\nExample code:\n         \n              \n   \nThanks!'   ← code block gone

Fix: mask only a length-equal copy of cleaned to locate the real tag spans, then delete those spans from the unmasked cleaned. Masking is a locator, not a text rewrite — protected spans now survive verbatim. Strengthened test_media_mixed_code_and_prose (it only asserted "Done." survived, not the code block) and added test_inline_code_survives_when_real_media_present. Both fail on the old .sub-based code and pass now.

🟡 Known limitation (not a regression — pre-existing main behavior, preserved)

Auto-append compression-safe fallback dedups by exact path string. When mid-run context compression shrinks the message list below history_offset, _collect_auto_append_media_tags falls back to scanning all messages and relies on history_media_paths (string-equality) to avoid re-appending a prior-turn tag. This is exactly what origin/main already does — the reconciled cherry-pick preserved it. With the new producer-tool allowlist layered on top, the fallback is now tighter than main (only text_to_speech results are scanned, not every tool). Not introduced here; flagging for awareness.

🟢 Verified clean (empirical sweeps)

• False-positive sweep (9 cases): bare line-start, prose-same-line, indented, MEDIA:"quoted", MEDIA:`backtick`, two-line TTS, two real tags, real-tag-then-code-example, Output: prefix — all extract correctly.
• Producer-allowlist (Probe C): non-producer (skill_view) → not appended; real TTS → appended; compression fallback (offset > len) → TTS still found; empty call_id → correctly excluded; mixed TTS+doc in one assistant turn → only the TTS path appended. image_generate/video_generate are intentionally not in the allowlist (their paths are model-emitted at the dispatch site).
• Regex backtracking: all masking patterns are linear (no nested quantifiers) — no catastrophic-backtracking risk.

Minor / acknowledged

~~~ (tilde) code fences and 4-space-indented markdown code blocks are not masked by _mask_protected_spans (it covers ```, inline, and >). Pre-existing gap in the original change; the common fenced/inline/blockquote forms are covered. Not blocking.

Tests: tests/gateway/test_platform_base.py + tests/gateway/test_media_extraction.py — 140 passed, 2 skipped.

[kshitijk4poor]

CI after the review-fix push: same two pre-existing unrelated red shards as before — test (1) (model-catalog.json drift) and test (3) (test_gui_command.py StopIteration mocks). Both reproduce on the unmodified base commit and neither imports the files this PR touches. e2e, ruff, nix (both OS), check-attribution, and the gateway test shards are green. The new cleaned-text regression test(s) pass.