feat: multimodal images through the v1 message graph + color-codeword-v1

[mikasenghaas]

Summary

Multimodal (VLM) image support through the verifiers v1 message graph, so a VLM trains end to end.

• Ingress: a content-part union (TextContentPart / ImageUrlContentPart, MessageContent) lets UserMessage/SystemMessage hold images; Task.instruction: str | Messages seeds an image-bearing initial prompt (the default harness opts in via SUPPORTS_MESSAGE_INSTRUCTION, passed as INITIAL_MESSAGES). The interception server and the v0 legacy bridge preserve image parts (shared content_to_parts) instead of flattening to text; message_hash hashes list content stably.
• Egress: the renderer's MultiModalData rides on TurnTokens / MessageNode as a transient, serialization-excluded sidecar (offsets stored node-local); add_turn attributes each image to the node that introduced it; Branch.multi_modal_data merges the nodes' items and rebases offsets to branch-global. Pixel tensors never reach the wire or results.jsonl.
• User colocation: UserConfig.colocated (default True) runs the vf.User inside the agent's already-started runtime instead of spawning one per rollout — removes the per-rollout runtime churn (and the startup races it caused) for multi-turn tasksets.
• color-codeword-v1 taskset: the v1 port of the v0 color-codeword env (multi-turn VLM codeword decoding; turn-0 squares in the Messages instruction, later turns via a colocated vf.User).
• Review-pass cleanups: capability flags (APPENDS_SYSTEM_PROMPT, SUPPORTS_*) as docstrings + tightened resolve_prompt; trimmed the mm transient-carrier comments; color-codeword-v1 hard-codes MAX_TURNS/SEED as module constants with a Field(ge=...) validator.

Pairs with prime-rl#2766 (the training consumer + config).

Verification

• v0 ↔ v1 parity: short 8-step 2-GPU runs of both paths train end to end — Trainable 256/256 every step (mm_kwargs reach the Qwen3-VL forward), Error 0.0%, clean finish; the two paths behave equivalently (v1 taskset rewards 0.86/0.58/0.68/0.55/0.46/0.54/0.39/0.38; v0 bridge 0.83/0.68/0.70/0.43/0.57/0.36/0.53/0.35).
• Reward is noisy over short runs (both start ~0.85 and drift down over 8 steps — a short-run/lr artifact present identically in the v0 baseline, not a v1 regression). The CI nightly (25 steps × 8 GPU) is the convergence test (reward → 0.85).
• A unit test (in the prime-rl PR) pins the node-local → branch-global placeholder rebase and the exclude-from-wire invariant.

🤖 Generated with Claude Code

---

Note

Medium Risk
Widens Task.instruction and changes ingress/persistence boundaries for images; harnesses without SUPPORTS_MESSAGE_INSTRUCTION will error on multimodal tasks, and training consumers depend on correct mm attribution.

Overview
This PR wires multimodal images through verifiers v1 end-to-end and adds a reference VLM taskset.

Message model & ingress: UserMessage / SystemMessage bodies become MessageContent (str or typed text + image_url parts). Shared content_to_parts replaces flattening images to text in the interception server and v0 legacy bridge. message_to_wire emits native OpenAI multimodal shapes. Task.instruction is now str | Messages; harnesses opt in with SUPPORTS_MESSAGE_INSTRUCTION (default harness passes pre-built turns via INITIAL_MESSAGES).

Training graph: Renderer MultiModalData flows on transient TurnTokens / MessageNode (excluded from persistence). add_turn attributes each image to the node that introduced it; Branch.multi_modal_data merges along the path for mm_kwargs. List content is hashed canonically for graph dedup.

Rollout ergonomics: User simulators default to colocated in the agent runtime (UserConfig.colocated, serve_user(agent_runtime=...)), avoiding a separate runtime per rollout. Episodes call trim_memory_periodically (glibc malloc_trim every N rollouts) to cap RSS after large base64 image bodies.

New example: color-codeword-v1 — multi-turn color-square → letter codeword task; turn 0 images in Messages instruction, later turns via colocated vf.User MCP simulator.

^{Reviewed by Cursor Bugbot for commit 15ee62a. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add multimodal image support to the v1 message graph and introduce color_codeword_v1 taskset

• Widens UserMessage, SystemMessage, and Task.instruction to accept lists of typed content parts (TextContentPart, ImageUrlContentPart) in addition to plain strings, propagating multimodal content through the OpenAI wire serialization path.
• Updates message_hash and add_turn in graph.py to correctly hash and attribute multimodal content parts to the graph nodes that introduced them.
• Adds SUPPORTS_MESSAGE_INSTRUCTION to harness.py and updates the default harness to serialize a Messages instruction as INITIAL_MESSAGES env, with the program reading it on startup.
• Introduces the color_codeword_v1 taskset: a multi-turn task where the model decodes a sequence of colored image squares into a letter codeword, with exact and partial match rewards.
• Adds colocated support to UserConfig and serve_user so the user simulator can run inside the agent runtime instead of a separate process.
• Risk: Task.instruction type is now str | Messages; harnesses that don't set SUPPORTS_MESSAGE_INSTRUCTION=True will raise ValueError if passed a Messages instruction.

^{Macroscope summarized 15ee62a.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces multimodal image support as a significant new feature, changing core message types and adding new processing logic across multiple framework files. An unresolved review comment also identifies a potential environment variable leakage bug that could affect runtime behavior.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 15ee62a. Configure here.}

[cursor]

Stale INITIAL_MESSAGES env leaks

Medium Severity

For string instructions, DefaultHarness never sets INITIAL_MESSAGES, while program.py treats any non-empty parsed value as the chat seed. On the subprocess runtime, the child inherits the host os.environ, so a leftover INITIAL_MESSAGES from an earlier multimodal run can override the argv user prompt.

Additional Locations (1)

• packages/harnesses/harnesses/default/program.py#L119-L124

 

^{Reviewed by Cursor Bugbot for commit 15ee62a. Configure here.}