Skip to content

Fix(yuanbao): quote file media resolve - #22742

Closed
libo1106 wants to merge 6 commits into
NousResearch:mainfrom
YuanbaoTeam:fix/yuanbao-quote-file-media-resolve
Closed

libo1106 wants to merge 6 commits into
NousResearch:mainfrom
YuanbaoTeam:fix/yuanbao-quote-file-media-resolve

Conversation

@libo1106

@libo1106 libo1106 commented May 9, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fix Yuanbao bot's inability to read quoted file and image messages in group chats. When a user quotes a file/image message and @mentions the bot, the bot previously could not access the media content — it would report "file not found" because the quote metadata ( ⁠cloud_custom_data⁠ ) doesn't carry the ⁠resourceId⁠ needed for downloading.
This PR introduces a transcript fallback lookup mechanism: when the quote's ⁠desc⁠ field doesn't contain ybres references (which is always the case for file quotes, ⁠type=3⁠ ), the system looks up the quoted message by ⁠message_id⁠ in the session transcript to retrieve the ⁠resourceId⁠ that was stored when the message was first observed.

Related Issue

Fixes #

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

File: ⁠gateway/platforms/yuanbao.py⁠

  1. ⁠_parse_resource_id⁠ + ⁠_extract_text⁠ ybres anchors — When ⁠TIMFileElem⁠ messages are observed, store ⁠[file:name|ybres:resourceId]⁠ in transcript so the resourceId is available for later lookup.
  2. ⁠QuoteContextMiddleware._extract_quote_context()⁠ — Returns a 3-tuple ⁠(id, text, media_refs)⁠ extracting any ybres references from the quote desc. Added ⁠ctx.quote_media_refs⁠ field to ⁠InboundContext⁠ .
  3. ⁠DispatchMiddleware._dispatch_inbound_event()⁠ — When a user quotes a message:
    ⦁If ⁠quote_media_refs⁠ is empty (file quotes only have filename in desc), performs a reversed transcript lookup by ⁠message_id⁠ to extract ybres anchors from the observed message content.
    ⦁Resolves and downloads quote media refs exclusively (skips unrelated history backfill).
    ⦁Overrides ⁠message_type⁠ to ⁠DOCUMENT⁠ when resolved media includes document MIME types ( ⁠application/*⁠ , ⁠text/*⁠ ) so ⁠gateway/run.py⁠ 's file injection logic properly provides the content to the agent.
  4. ⁠_RESOLVABLE_MEDIA_KINDS⁠ constant — Extracted ⁠frozenset({"image", "file"})⁠ to eliminate duplicated kind-checking logic across 4 call sites.

How to Test

File quote (quote_type=3):

  1. In a Yuanbao group chat, send a ⁠.txt⁠ file (do NOT @bot)
  2. Wait for bot to observe the message (gateway log shows ⁠Group message observed⁠ )
  3. Quote/reply to that file message, @bot and ask "这个文件里有什么?"
  4. ✅ Expected: Bot reads and responds with file content

Image quote (quote_type=2):

  1. In a Yuanbao group chat, send an image (do NOT @bot)
  2. Quote/reply to that image message, @bot and ask "这是什么?"
  3. ✅ Expected: Bot analyzes and describes the image

Regression — non-quote messages:

  1. Send a message directly @bot without quoting → should work as before
  2. Send a file directly @bot (no quote) → should download and read normally

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Linux

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

  • This skill is broadly useful to most users (if bundled) — see Contributing Guide
  • SKILL.md follows the standard format (frontmatter, trigger conditions, steps, pitfalls)
  • No external dependencies that aren't already available (prefer stdlib, curl, existing Hermes tools)
  • I've tested the skill end-to-end: hermes --toolsets skills -q "Use the X skill to do Y"

Screenshots / Logs

libo1106 added 5 commits May 10, 2026 00:17
…ote desc lacks ybres

When a user quotes a file message (type=3) and @bot, the quote's desc field
only contains the filename without a ybres:// resource reference. The existing
QuoteContextMiddleware only extracted media refs from desc using the ybres regex,
which always returned empty for file quotes.

Fix: add a transcript lookup fallback in QuoteContextMiddleware.handle() —
when quote_media_refs is empty but reply_to_message_id is set, search the
session transcript for the quoted message_id and extract ybres anchors from
its content.

Also fix message_type classification: when quote media resolves non-image files,
override message_type to DOCUMENT so gateway/run.py's document injection logic
properly prepends the file path and content for the agent.
@libo1106 libo1106 changed the title Fix/yuanbao quote file media resolve Fix(yuanbao) quote file media resolve May 9, 2026
@libo1106 libo1106 changed the title Fix(yuanbao) quote file media resolve Fix(yuanbao): quote file media resolve May 9, 2026
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery labels May 9, 2026

@beforeload beforeload left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've encountered this issue as well. when a user quotes a file or image message and @mentions the bot in a group chat, the bot can't access the media content because the quote metadata doesn't carry the resourceId.
I've tested this PR hands-on and it works well. Would love to see it merged. Thanks!

@libo1106

Copy link
Copy Markdown
Contributor Author

Hi @alt-glitch

Just wanted to follow up on this PR — is there anything else I can help with to move it forward?

This fix is scoped only to the Yuanbao platform and shouldn’t affect other platforms. I’ve tested it locally (including file/image quote scenarios), and everything is working as expected.

Happy to make any adjustments or add more tests if needed. Thanks!

@loongfay

Copy link
Copy Markdown
Contributor

This is our team's developer. This fix will optimize the dialogue experience of Yuanbao.
@alt-glitch @kovyrin @jbarket

@teknium1

Copy link
Copy Markdown
Collaborator

Salvaged onto current main via #26216 and merged. Your 5 commits were cherry-picked with your authorship preserved (rebase merge), now visible on main as d57a4b3..0086cda. Thanks for the fix!

One conflict at _collect_observed_media was resolved in favor of your _RESOLVABLE_MEDIA_KINDS constant — current main had migrated the literal {"image", "file"} to a set via PR #23937 (ruff PLR6201), so your constant landed as the canonical version. Same intent, cleaner code.

@libo1106
libo1106 deleted the fix/yuanbao-quote-file-media-resolve branch May 20, 2026 02:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants