Skip to content

fix(gateway): build _TOOL_MEDIA_RE from shared MEDIA_EXT_ALTERNATION (#37318) - #37347

Closed
temalo wants to merge 1 commit into
NousResearch:mainfrom
temalo:fix/37318-media-tag-extensions
Closed

fix(gateway): build _TOOL_MEDIA_RE from shared MEDIA_EXT_ALTERNATION (#37318)#37347
temalo wants to merge 1 commit into
NousResearch:mainfrom
temalo:fix/37318-media-tag-extensions

Conversation

@temalo

@temalo temalo commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Why

The agent emits MEDIA:/tmp/report.md (or any of a dozen common deliverable extensions). The dispatch-site regex in gateway/platforms/base.py strips the tag and returns {success: true} — but the file is never actually shipped to the user. The reporter found this on Feishu after extensive debugging that ruled out Feishu, streaming, and the platform adapter; the root cause was a regex whitelist mismatch one layer down.

gateway/run.py's auto-append + history-dedup path uses its own _TOOL_MEDIA_RE that hand-rolled the extension list. The list was a strict subset of the canonical MEDIA_DELIVERY_EXTS tuple that gateway/platforms/base.py uses, so the two patterns drifted:

extension MEDIA_DELIVERY_EXTS _TOOL_MEDIA_RE (run.py)
.md
.json, .yaml, .yml, .xml
.html, .htm, .svg
.bmp, .tiff
.tsv, .odt, .ods, .odp, .key
.tar, .bz2, .xz

Result: _collect_auto_append_media_tags silently dropped them on the way out, and the in-line copy of the same regex inside the history scan silently dropped them again on the way in.

What

  • Export MEDIA_EXT_ALTERNATION (was _MEDIA_EXT_ALTERNATION, kept as a backwards-compat alias) from gateway/platforms/base.py so other dispatch sites can build their regex off the same source-of-truth alternation that drives MEDIA_TAG_CLEANUP_RE and extract_media.
  • Rebuild gateway/run.py's module-level _TOOL_MEDIA_RE from that shared alternation.
  • Replace the duplicate, locally-redeclared _TOOL_MEDIA_RE inside the history-scan loop with a reference to the module-level pattern, so a future drift can only happen in one place instead of three.
  • Add .markdown and .toml to MEDIA_DELIVERY_EXTS (called out by the reporter, lossless additions).
  • Explicit comment in base.py noting that .py / .js / .sh are deliberately not in the tuple — the existing test_no_media_extensions contract makes clear that bare-path mentions of arbitrary source files should not auto-ship. The reporter asked for them; the constitution says no, and I preserved the constitution.

Behaviour change footprint

  • extract_media (base.py) gains .markdown and .toml as recognised extensions. All other behaviour identical.
  • _TOOL_MEDIA_RE in gateway/run.py now matches the full MEDIA_DELIVERY_EXTS set instead of its old subset. This was the bug — fixing the regex strictly broadens what the auto-append path will deliver, which is exactly what the dispatch path already does.
  • Two producer-tool layers in gateway/run.py (the _collect_auto_append_media_tags allowlist and the per-tool MEDIA: detection) still gate which tools are eligible to emit MEDIA: tags in the first place, so the broadened regex doesn't open new attack surface — it only stops dropping deliverable tags that the dispatch layer was already approving.

Test coverage

New file: tests/gateway/test_run_tool_media_ext_parity.py

Test class What it asserts
test_every_delivery_ext_matched_by_run_pattern Parametric over every extension in MEDIA_DELIVERY_EXTS. Locks in the parity contract so a future contributor who adds an extension to the shared tuple gets a failing test if they forget to keep the patterns in sync.
test_issue_37318_named_extensions_match Each extension specifically named in the issue (.md, .markdown, .json, .yaml, .yml, .toml, plus the silently-dropped pre-existing ones like .html/.svg/.bmp/...) now matches.
test_run_pattern_uses_shared_alternation Walks the shared alternation and verifies the run.py regex accepts each branch — defends against re-introducing a hand-rolled subset.
test_existing_paths_still_match Unix, home-relative, and Windows path forms from the existing #34632 tests still match.
test_invalid_inputs_still_rejected Relative paths, missing anchor, unknown extension, and bare MEDIA: tokens are still rejected — the fix didn't loosen anchoring or accept arbitrary extensions.
test_md_tag_extracted End-to-end: BasePlatformAdapter.extract_media strips a MEDIA:/tmp/report.md tag cleanly.

Pre-existing tests (test_run_tool_media_re.py, test_media_extraction.py, test_extract_local_files.py) continue to pass, including test_no_media_extensions which guards the deliberate .py / .log exclusion.

Full tests/gateway/ directory: baseline on upstream/main has 39 failures (pre-existing missing-deps / Python 3.14 / flake — unrelated). My branch has 39 failures, the same set; new failures introduced: 0.

Out of scope

  • I did not add .py / .js / .sh to MEDIA_DELIVERY_EXTS even though the issue asked for them, because the existing test_no_media_extensions contract intentionally excludes them from the bare-path auto-detector. Adding them would surprise existing users whose agents mention /tmp/script.py in prose. Users who want source-file delivery can still use the explicit send_message MEDIA: flow with a glob, but the auto-detector should stay narrow. The comment in base.py near MEDIA_DELIVERY_EXTS documents this.
  • I did not refactor the broader auto-append flow or the history-dedup path. The fix is the minimum needed: one shared alternation, one module pattern, two call sites using it.
  • The issue references "the same issue exists in gateway/run.py (line ~17121)" — that line is the history-dedup _TOOL_MEDIA_RE, and it's the second of the two patterns I fixed.

Fixes #37318

The auto-append and history-dedup regexes in gateway/run.py
hand-rolled their own extension whitelist that drifted behind
the source-of-truth list MEDIA_DELIVERY_EXTS in
gateway/platforms/base.py. Common deliverable extensions —
.md, .markdown, .json, .yaml, .yml, .toml, .html, .htm, .svg,
.bmp, .tiff, .tsv, .xml, .odt, .ods, .odp, .key, .tar, .bz2,
.xz — were silently dropped from the auto-append path, and
the dispatch regex in base.py would happily strip the MEDIA:
tag before delivery returned a misleading success.

User-visible symptom (from the issue): asking the agent to
send a .md file via Feishu/Telegram/etc returned
{success: True} but the file never reached the user.

Fix:
- Export MEDIA_EXT_ALTERNATION (was _MEDIA_EXT_ALTERNATION,
  kept as alias) from gateway/platforms/base.py so other
  dispatch sites can reuse the same alternation.
- Rebuild gateway/run.py's module-level _TOOL_MEDIA_RE from
  the shared alternation.
- Replace the duplicate locally-redeclared _TOOL_MEDIA_RE
  inside the history scan with a reference to the module
  pattern, so the two patterns can never drift again.
- Add .markdown and .toml to MEDIA_DELIVERY_EXTS (called out
  by the reporter, lossless additions). Deliberately keep
  .py/.js/.sh out — the existing test_no_media_extensions
  contract makes clear those should not be auto-shipped from
  bare-path mentions.

Test coverage (tests/gateway/test_run_tool_media_ext_parity.py):
- Every extension in MEDIA_DELIVERY_EXTS is matched by
  _TOOL_MEDIA_RE (parity guard against future drift).
- Each extension named in the issue now matches.
- Existing Unix/home-relative/Windows path matching unchanged.
- Invalid paths (relative, no anchor, unknown ext) still rejected.
- End-to-end: BasePlatformAdapter.extract_media now strips a
  MEDIA:/tmp/report.md tag cleanly.

Regression check: full tests/gateway/ suite — baseline 39 failures
on upstream/main (pre-existing missing-deps / py3.14 / flake), my
branch 39 failures, new failures: zero.

Fixes NousResearch#37318
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #37342 — both fix #37318 by expanding the MEDIA: delivery extension whitelist (base.py + run.py regexes). Part of the saturated MEDIA-regex cluster; canonical dynamic approach is #29609.

@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists duplicate This issue or pull request already exists labels Jun 2, 2026
@temalo

temalo commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

Closing as confirmed duplicate of #37342 per @alt-glitch's triage. Both PRs took the static-whitelist approach to fixing #37318; #29609 has the canonical dynamic fix and is the better path forward.

Thanks for the catch — should've grepped open PRs against the same issue before opening.

@temalo temalo closed this Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MEDIA: tag fails silently for .md files — extract_media() regex whitelist missing common extensions

2 participants