Skip to content

chore(configs)!: drop vlm attr-path fields and the unread orchestrator.model.vlm - #3852

Closed
faresobeid wants to merge 1 commit into
refactor/trainer-vlmfrom
refactor/drop-vlm-attr-config
Closed

faresobeid wants to merge 1 commit into
refactor/trainer-vlmfrom
refactor/drop-vlm-attr-config

Conversation

@faresobeid

@faresobeid faresobeid commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #3861 (refactor/trainer-vlm: fold multimodal into trainer/vlm.py; one VLM registry). That PR removes the last readers of these fields. Merge it first.

Summary

Breaking config change.

  • [model.vlm] loses vision_encoder_attr and language_model_attr. Only freeze_vision_encoder is left. The trainer now finds the modules by convention (model.visual, and model.language_model with a fallback to model.model). The only trainable VLM families (Qwen3.5 dense and MoE) already used exactly these paths.
  • orchestrator.model.vlm is removed. Nothing in the orchestrator read it, because images are detected from message content parts. vlm moves from BaseModelConfig to the trainer ModelConfig. The shared RL [model.vlm] now propagates only to trainer.model.vlm.
  • Updated to match:
    • examples/extra/vlm/{sft,sft-moe}.toml and configs/ci/nightly/multimodal_color_codeword.toml: the color-codeword config now has a bare [model.vlm].
    • docs/advanced.md: the family table loses the attr columns, and the wrong "prefix derived from language_model_attr" sentence is gone.
    • tests/unit/test_configs.py.
  • Removed test_orchestrator_vlm_requires_renderer. test_orchestrator_renderer_none_rejected already covers the renderer=None rejection, and the VLM part no longer exists.

Migration: delete vision_encoder_attr / language_model_attr lines and any [orchestrator.model.vlm] block. Old keys now fail with "Extra inputs are not permitted". No compatibility shim, by design.

Behavior change

Config-schema only, as described above. Resolved configs were dumped with model_dump(mode="json") for every TOML under configs/, examples/, k8s/ (91 entries, with overlays composed on sibling bases). Every difference is a removed key:

  • orchestrator.model.vlm (always null)
  • the two attr strings in the 3 VLM TOMLs

No value changed.

Checks

  • pytest tests/unit/test_configs.py tests/unit/orchestrator tests/unit/train/test_multimodal.py -m "not gpu": 292 passed
  • ruff check, ruff format --check: clean
  • Old keys and [orchestrator.model.vlm] are rejected at parse time (checked by hand).

GPU checks (not run): the nightly configs/ci/nightly/multimodal_color_codeword.toml RL run, and examples/extra/vlm/sft.toml for a few steps. These are the same model paths that the base PR already validates on GPU.

Conflicts

Text-only conflicts:

No logical conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants