Conversation
`_common.run_one()` has required `anthropic_intermediate_system_expectation` for every Anthropic run since #2711. `test_qwen38small.py`, added by #2760, never declared it, so its `stage-c-2-gpu-h200` lane raises before doing any work: ValueError: Anthropic per-model verification requires an intermediate-system expectation #2760 branched before that check existed, and its own CI had `stage-c-2-gpu-h200` skipped, so the combination was never exercised. `"forbidden"` is the derived value, not a guess. `_verify_intermediate_system()` returns `route_supports and "system" in fixed_template_append_roles(tito_model)`, and `Qwen38SmallTITOTokenizer.FIXED_TEMPLATE.allowed_append_roles` is `{"tool", "user", "assistant"}`, so the live capability is always False. The sibling models whose fixed templates also omit `system`, `qwen35` and `qwen36`, declare `"forbidden"` too. Nothing enumerated the real `ModelConfig` objects, which is why a missing required field could only surface from a GPU lane. Add a CPU regression test that parses each model config in the directory and asserts that every Anthropic-verified one declares a valid contract. It reads the files with `ast` instead of importing them, so it does not trigger the e2e CI registrations.
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Evidence that
|
With the intermediate-system contract declared, the lane gets past `run_one()`
and reaches the Anthropic leg, which then trips the soft gate:
AssertionError: Session multi-role e2e: assistant_text mismatch ratio
32/32=1.000 exceeds threshold 0.2
Every hard gate stayed at 0 in that run: `special_token_count`,
`special_token_type` and `non_assistant_text` were all `0.0`, and only
`assistant_text` was `1.0`. That is the endpoint-only serialization
difference the Anthropic tool-call conversion introduces, not a TITO bug.
Both sibling models that pair `tool_call_parser="qwen3_coder"` with the
Anthropic endpoint, `qwen35` and `qwen36`, already carry
`anthropic_assistant_text_threshold=1.0` for exactly this reason, as does
`qwen3` on `qwen25`. Give `qwen38small` the same treatment, with the same
comment, so the OpenAI-side threshold and every hard gate stay untouched.
Second commit: the lane needs a second field tooPushed Once the intermediate-system contract is declared, the lane gets past The metrics from that same run show every hard gate at zero: Only the soft This is a solved problem for this exact model family. Every sibling that pairs
So |
_common.run_one()has requiredanthropic_intermediate_system_expectationfor every Anthropic run since #2711.test_qwen38small.py, added by #2760, never declared it, so itsstage-c-2-gpu-h200lane raises before doing any work:#2760 branched before that check existed, and its own CI had
stage-c-2-gpu-h200skipped, so the combination was never exercised. Observed on the currentmaincommit6b7ac37283in https://github.com/radixark/miles/actions/runs/33730559131."forbidden"is derived, not guessed._verify_intermediate_system()returnsroute_supports and "system" in fixed_template_append_roles(tito_model), andQwen38SmallTITOTokenizer.FIXED_TEMPLATE.allowed_append_rolesis{"tool", "user", "assistant"}, so the live capability is alwaysFalseand only"forbidden"satisfies_assert_intermediate_system_expectation. The sibling models whose fixed templates also omitsystem,qwen35andqwen36, declare"forbidden"as well; the ones that allow appendingsystemdeclare"required".Nothing enumerated the real
ModelConfigobjects, which is why a missing required field could only surface from a GPU lane. The second commit hunk adds a CPU regression test that parses every model config in that directory and asserts each Anthropic-verified one declares a valid contract. It reads the files withastrather than importing them, so it does not trigger the e2e CI registrations. Verified both ways on a devbox: 9/9 parametrized cases pass as committed, and dropping the new line again fails exactly[test_qwen38small].