feat(agent-core-v2): accept HEIC, HEIF, and BMP images for Kimi models - #3649
Conversation
Key the accepted image MIME set by the provider the agent is bound to instead of a single global allowlist: Kimi accepts PNG/JPEG/GIF/WebP plus BMP/HEIC/HEIF, every other provider keeps the baseline set. The provider image policy lives in the human layer (`llm/media/image-formats`) so both the agent-side gates and the Anthropic protocol lowering share it. - thread the provider type through every ingestion gate: ReadMediaFile, prompt entry, MCP tool results, session media references, kap-server prompt/skill routes; expose it via IAgentProfileService.getModelProviderType - ReadMediaFile sends accepted formats it cannot re-encode locally inline up to the provider's per-image budget (5MB for Kimi, 3.75MB otherwise) and points at a conversion command above that; full_resolution uses the same provider budget - acp-server no longer gates formats itself and leaves the decision to the engine prompt pipeline, which knows the bound provider - the Anthropic lowering allowlist follows the bound provider so Kimi over the Anthropic-compatible protocol no longer rejects HEIC client-side
# Conflicts: # packages/agent-core-v2/src/agent/prompt/promptService.ts
🦋 Changeset detectedLatest commit: afbc8e9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04c878377e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… paths Follow-ups from review of the provider-aware image formats: - the Anthropic lowering read the policy from `LlmModel.provider`, which the adapter registry sets to the route id (`anthropic`) rather than the model's provider type, so Kimi over the Anthropic-compatible protocol still rejected HEIC client-side; the accepted set now comes from a `ProtocolTrait.acceptedImageMimes` hook that the Kimi traits implement, and the test goes through the real trait with the routed provider id - kap-server prompt ingestion gated media against the currently bound model even when the same request selected another model; the profile service now resolves the provider type for a requested alias - the session media memo served a cached inline image to a provider that does not accept its format; memo hits are re-judged against the current provider before being returned - the media tools registrar keyed ReadMediaFile only by alias and capabilities, so a reload that changed the alias's provider type or protocol kept the stale tool; both now take part in the key
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: afbc8e9865
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (inlineOnly) { | ||
| const inlineLimit = Math.max(readByteBudget, this.inlineImageByteBudget); | ||
| if (compressed.finalByteLength > inlineLimit) { |
There was a problem hiding this comment.
When an accepted non-recodable image is within the inline byte limit but exceeds maxEdge—for example, a wide GIF, BMP, or animated WebP—the inlineOnly branch checks only its byte length and skips the dimension check on lines 458–460. ReadMediaFile therefore sends it unchanged even in the default mode, whereas the previous path rejected images exceeding the configured/default edge ceiling and reserved native delivery for full_resolution; keep enforcing maxEdge here and return conversion guidance when the format cannot be resized.
Useful? React with 👍 / 👎.
Sync 147 upstream commits onto the fork. Notable upstream changes: - Remove the protocol package and relocate wire types into agent-core-v2/src/events.ts (MoonshotAI#3646) - Absorb stepRetry into the LLM requester turn state machine (already followed in the 2026-09-07 sync) - NotifyUser tool + mid-turn update panel (MoonshotAI#3524) - HEIC/HEIF/BMP image support (MoonshotAI#3649), resumable large file reads (MoonshotAI#3645), reasoning_details round-trip (MoonshotAI#3492), zstd updater manifests (MoonshotAI#3669) Conflict resolutions (all by hand, fork features preserved): - package.json: drop the empty simple-git-hooks block (upstream cleanup; not a fork feature). - docs/config-manifest.toml: take the union, then regenerate with pnpm gen:config-manifest (33 sections, fork sections intact). - test/agent/stepRetry/stepRetry.test.ts: keep upstream's slimmed retryBackoffDelays test — the fork had deleted the old fat version when stepRetry was absorbed; the util is fork-used code with no other coverage. - protocol/src/events.ts: accept upstream's package removal; the fork's compaction.started model/model_display fields already auto-merged into their new home (agent/fullCompaction/compactionOps.ts). No source still imports @moonshot-ai/protocol. Merge follow-ups: - Regenerate state/wire/config manifests; freshness tests pass. - node-sdk update-all-session-models flag test: drop the v1-engine parity half (legacy agent-core v1 is removed upstream); keep the v2 registration pin. - Fix pre-existing lint errors in merge-touched files so the lint-staged pre-commit gate passes: unused imports/variables (sdk-rpc-client-v2.ts, tui/commands/config.ts, check-import-boundaries.mjs), floating promises (sdk-rpc-client-v2.ts, fullCompaction.test.ts). - Add compactionOps.ts (model_display) to .github/FORK_OWNED_FILES so the relocated compaction-model fields are guard-listed. Verification: - pnpm run typecheck green; pnpm run lint at fork baseline (no new findings vs pre-merge HEAD). - FORK_OWNED_FILES markers all resolve. - Fork suites pass: TUI commands + session CLI (75 tests), x-opencode-session (7), update-all-session-models flag, oauth openai-compatible. - fullCompaction.test.ts: 2 failures, both pre-existing at pre-merge HEAD (baseline had 5; upstream's macOS stabilization fixed 3. EOF )
1 participant
Problem
ReadMediaFilerefused iPhone photos (.heic) with "which the provider does not accept. Convert it to JPEG first", even though the Kimi API accepts HEIC/HEIF/BMP natively (formats are detected server-side from magic bytes, and the upload pipeline re-encodes everything to WebP anyway). The refusal came from a single global image-format allowlist (PNG/JPEG/GIF/WebP) that was shared by every ingestion point and did not know which provider the agent was talking to.Two further blockers sat behind that gate:
ReadMediaFilecannot re-encode HEIC locally, and its default path capped the delivered image at the 256KB read budget, so a typical 1–4MB iPhone photo would have failed with "too large to send safely" even once the format was allowed.PNG/JPEG/GIF/WebPallowlist, so Kimi reached over the Anthropic-compatible protocol would still have rejected HEIC client-side.What changed
Provider-keyed image policy. A small vocabulary module in the human layer (
llm/media/image-formats) defines aProviderImagePolicyper provider type: Kimi acceptsPNG/JPEG/GIF/WebP/BMP/HEIC/HEIFwith a 5MB per-image inline budget; every other provider keeps the baseline four formats and the existing 3.75MB budget. The agent-side policy helpers (isModelAcceptedImageMime,gateImageFormatParts,buildUnsupportedImageNotice,unsupportedImageMimeFromUrl,compressImageContentParts) take an optionalproviderTypeand consult it, and the refusal notice now names the current provider's accepted formats.Provider type threaded to every gate.
IAgentProfileService.getModelProviderType()exposes the bound model's provider type; it is passed intoReadMediaFileby the media tools registrar (same pattern asinlineVideoSupported), into the prompt-entry gate in the prompt service, into MCP tool results (looked up at call time so a later model switch is honored), into session media references (from the requester's model), and into the kap-server prompt and skill routes via a newproviderTypeoption onresolvePromptMediaFiles.Inline passthrough for formats we cannot re-encode. In
ReadMediaFile, an accepted format that the local compressor cannot decode (HEIC/HEIF/BMP/GIF/animated WebP) is now sent untouched up tomax(read byte budget, provider inline budget)instead of the 256KB read budget; above that the tool returns a conversion command (sips/heif-convert/magick) since converting to JPEG re-enables local downsampling. Thefull_resolutioncap uses the same provider budget, so it stays at 3.75MB for non-Kimi providers and becomes 5MB for Kimi. Rationale for 5MB: 3.75MB was derived from Anthropic's 5MB limit minus base64 overhead and has no basis for Kimi; 5MB covers 12MP and 24MP iPhone HEICs while keeping the per-turn resend cost bounded.Anthropic lowering follows the bound provider. The base64 image allowlist in the Anthropic lowering now comes from the same policy keyed by
ctx.model.provider, so Kimi over the Anthropic-compatible protocol forwards HEIC while real Anthropic still fails fast.acp-server leaves format gating to the engine. The ACP edge does not know the bound provider, and the engine's prompt pipeline already gates every prompt image (with the same notices and alias canonicalization), so the redundant edge gate was removed and the stale comment claiming the engine never gates was corrected. Compression still happens at the edge.
The runtime safety net is unchanged: if a provider still rejects an image format, the error is classified as
image_formatand the request is resent with media stripped.Tests. Red-first for each slice: policy helpers (
image-compress.test.ts),ReadMediaFilewith and without a Kimi provider including the 5MB boundary, the registrar handing the provider type through,getModelProviderType, prompt-entry gating, MCP resource links and inline images (including call-time lookup), session media references, kap-server inline HEIC on a Kimi-typed provider, the Anthropic lowering, and the klient e2e matrix (an unaccepted-everywhere format is still replaced on every provider; a Kimi-only format reaches Kimi and is replaced elsewhere). Existing tests that used BMP as the "unaccepted" sample were switched to TIFF.Follow-up after review
LlmModel.provideris the route id (anthropic) on the production path, not the model's provider type, so the lowering still refused HEIC for Kimi over the Anthropic-compatible protocol. The accepted set now comes from aProtocolTrait.acceptedImageMimeshook implemented by the Kimi traits; the test goes through the realkimiAnthropicTraitwith the routed provider id, including a requester-level check of the wire body.model, media preparation now resolves that alias's provider type (getModelProviderType(alias)); a request that only switchesprofilestill uses the bound model's provider.Not changed on purpose: re-gating historical images after a provider switch (the existing
image_formatrecovery already strips media and resends, so the session stays usable; a provider-aware history projection is a separate change), and enforcing the 5MB inline budget at prompt/MCP ingress (that budget isReadMediaFile's decision boundary where the model can convert the file; user-supplied attachments keep the existing 64MB/10MB caps, as GIF always has).