RFC-0061: Information-Preserving Degradation for ModalityFilter (design discussion) - #377
Merged
Million-mo merged 9 commits intoAug 21, 2026
Conversation
…ty filter * docs(rfc): design discussion for modality_filter degradation Proposes replacing the bare MIME placeholder ([image/png]) that ModalityFilterCapability emits when degrading unsupported multimodal content with an information-preserving placeholder carrying retrievable metadata (filename/mime/source), and optionally a session-scoped persisted reference so a text-only model can delegate the image to a vision-capable subagent or file tool. The problem is grounded in opencode ecosystem findings (HEAD 040b856): bare placeholders were rejected in PR #29279 on hallucination risk, are reported as a defect in issue #42758 (no way to access actual content), and the community consensus (#29216) is to preserve the reference rather than destroy it. Auto model-switching is deliberately out of scope (NOT_PLANNED upstream). Surveys 4 options (status quo / metadata-only / persist+reference / hybrid) with an evaluation matrix and recommends the hybrid: upgrade describe() to honest metadata now, add an opt-in reference strategy for manifests that declare vision-capable consumers.
Phase 1 (RFC-0061): describe_multimodal_content() no longer emits a bare
[image/png] placeholder. Binary content now states its media type, that
direct model processing is unsupported, and whether a file identifier is
available. Control characters in caller-supplied identifiers are escaped
to prevent prompt injection via malformed filenames. URL content keeps its
[image: url] / [audio: url] form since the URL is already retrievable.
Phase 2: new opt-in 'reference' strategy persists binary content to a
per-session scratch directory and replaces it with a [file: <path>]
reference that a vision-capable subagent or the agent's read tool can
open. Scratch dirs live under tempfile.gettempdir()/wolfharness-modality/
{session_id}/; after_node_run() removes them. URL and UploadedFile content
has no local bytes and falls back to describe. The strategy is wired
through the ModalityFilterCapabilityConfig schema.
Million-mo
force-pushed
the
docs/rfc-0061-modality-filter-information-preserving-degradation
branch
from
August 19, 2026 07:41
efbf553 to
d065144
Compare
…ersions - mcp_server/conversions.py: pass BlobResourceContents.mimeType through to BinaryContent instead of hardcoding application/octet-stream - acp_server/converters.py: drop _DOCUMENT_FORMATS whitelist gate in resource_to_content so arbitrary-mime embedded blobs reach the model, with octet-stream fallback; modality filtering stays downstream in ModalityFilterCapability - add unit tests for both conversions (mime passthrough, fallback, image binary preservation)
- Current State: note original bare placeholder vs info-preserving impl landed - Goals: mark 'no protocol server rewrites' superseded; document converter-layer mime passthrough rationale - Decision Record: add 2026-08-19 entry for describe+reference+mime fix commits
pydantic-ai falls back to a short content hash (sha1[:6]) when no explicit identifier is set. A bare hash is not a retrievable reference, so presenting it as 'File: <hash>' misled text-only models into thinking they could open it. Hash-shaped identifiers now degrade to the 'no file reference available' branch; real paths from fsspec-style tools still surface.
…oldown list_resources/list_resource_templates now query providers concurrently under a 10s timeout; McpServerCap caches listings and invalidates on reconnect (servers rarely send resources/list_changed) and on exit; failed MCP connections enter a 30s cooldown so retries aren't re-paid per call; viking per-directory ls runs in parallel; http(s) URIs are forwarded to providers (MCP resources may legally use any RFC3986 scheme) instead of being rejected up front.
…e cap read_resource returns image bytes (base64 blob + MIME) for vision-capable models, gated on _should_return_image_bytes() and viking:// scheme; a 5MB cap degrades oversize images to a text URI hint on both read_resource and viking_read; ToolReturn truncation preserves binary content instead of serializing to bytes repr or char-splitting string content.
…erving-degradation Resolve conflict in mcp_server_cap.py: integrate PR wolf1069b#384's direct client fallback and _connect_via_pool() extraction with our connect cooldown. The cooldown check now lives inside _connect_via_pool() (pool-based retry path) while the direct-client fallback (no pool) proceeds without cooldown.
Million-mo
deleted the
docs/rfc-0061-modality-filter-information-preserving-degradation
branch
August 21, 2026 01:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RFC design discussion PR for
ModalityFilterCapabilitydegradation, now with a working implementation on this branch. The motivating defect: when the active model does not support a modality, the capability degrades an image to a bare MIME placeholder ([image/png]), which:Motivation (real deployment)
Encountered while debugging a text-only model (
glm52/ kimi-k2) failing on pasted images through the opencode server — the same downstream scenario as opencode issue #42758.What this RFC does
Surveys 4 options with an evaluation matrix:
[image/png]placeholderreferencestrategyRecommends Option 4: upgrade
describe()to honest metadata immediately (cheap, in-core), add an opt-inreferencestrategy for manifests that declare a vision-capable subagent/tool consumer.Grounding: opencode ecosystem (HEAD
040b856)Implementation on this branch (2026-08-19)
Landing prototype of Option 4 + converter-layer mime integrity so binary content reaches the filter intact:
d06514464— information-preservingdescribe(honest metadata placeholder with anti-hallucination guardrail) + opt-inreferencestrategy (session-scoped scratch persistence undertempfile).115ca7784—BlobResourceContents.mimeTypepassthrough in MCP conversions; removal of the_DOCUMENT_FORMATSwhitelist in ACPresource_to_contentso arbitrary-mime embedded blobs surface asBinaryContent(octet-stream fallback).ModalityFilterCapabilityremains the single decision point for unsupported modalities.Discussion ask
referencestrategy (differentiating from opencode's unresolved PRs)?Files
docs/rfcs/draft/RFC-0061-modality-filter-information-preserving-degradation.md(new, under review)src/wolfharness/capabilities/modality_filter.py(implementation)src/wolfharness/capabilities/modality_utils.py(implementation)src/wolfharness/mcp_server/conversions.py(mime passthrough)src/wolfharness_server/acp_server/converters.py(drop_DOCUMENT_FORMATSwhitelist)