fix: handle non-tool appended messages in TITO incremental tokenization

[guapisolo]

Summary

This PR fixes TITO incremental tokenization for non-assistant appends by replacing the old single dummy-diff approach with role-aware segmentation.

Previously, append tokenization could become unstable when appending user/system messages (and mixed tool + non-tool sequences), because boundary tokens were inferred from a single synthetic context.
Now we tokenize appended content segment-by-segment using role-specific synthetic prefixes.

What Changed

• Refactored tokenize_additional_non_assistant in tito_tokenizer.py to a role-segmented pipeline.
• Added segment splitting rules:
  • contiguous tool messages are grouped and tokenized together. use [dummy_system, dummy_assistant] in additional tokenzer without dummy_user to avoid any cut think issue across models.
  • each user and system message is tokenized as a singleton segment, and use [dummy_system] in additional tokenzer
• Keep Qwen-3.5 template ban system message behavior, and revert previous modifications.
• Modify Qwen-3.5 chat template logic to skip no user message check.

Tests

Updated unit tests (test_tito_tokenizer.py)

• Add coverage for user message.
• Add coverage for intermediate system check, which can detect templates that ban intermediate system like qwen3.5
• test_tito_tokenizer_model_matrix.py run many additional tokenize checks across different models.

New model-matrix tests (test_tito_tokenizer_model_matrix.py)

• Added cross-model matrix cases and existing failure explanation.
• Assertions focus on using TokenSeqComparator to check mismatches except assistant-text diff.

[gemini-code-assist]

Code Review

This pull request updates the _build_dummy_assistant function to correctly handle leading tool messages within appended_messages, ensuring proper turn-transition tokens are rendered. Feedback suggests restoring the reasoning_content field to maintain consistency with reasoning models and simplifying the tool_calls list comprehension using enumerate on a slice of the messages.

[gemini-code-assist]

The reasoning_content field was removed from the dummy assistant message. This field was present in the previous implementation (line 32) and is often necessary for reasoning models to correctly render turn boundaries (e.g., to ensure the reasoning block is closed). Unless its removal was intentional to fix a specific issue, it should be restored.

Additionally, the tool_calls generation can be simplified using enumerate on a slice of appended_messages.

Suggested change

	assistant: dict[str, Any] = {"role": "assistant", "content": ""}
	if num_leading_tools > 0:
	assistant["tool_calls"] = [
	{
	"id": resp.get("tool_call_id") or f"call0000{i}",
	"id": appended_messages[i].get("tool_call_id") or f"call0000{i}",
	"type": "function",
	"function": {
	"name": resp.get("name") or "dummy_func",
	"name": appended_messages[i].get("name") or "dummy_func",
	"arguments": {},
	},
	}
	for i, resp in enumerate(tool_responses)
	],
	}
	for i in range(num_leading_tools)
	]
	assistant: dict[str, Any] = {
	"role": "assistant",
	"content": "",
	"reasoning_content": " ",
	}
	if num_leading_tools > 0:
	assistant["tool_calls"] = [
	{
	"id": msg.get("tool_call_id") or f"call0000{i}",
	"type": "function",
	"function": {
	"name": msg.get("name") or "dummy_func",
	"arguments": {},
	},
	}
	for i, msg in enumerate(appended_messages[:num_leading_tools])
	]

[maocheng23]

For my sanity check, now our TITO would only support GLM style? Can we still keep it flexible somehow?

If it would be too hard to support Qwen3 chat template, I think it's still good here

[maocheng23]

And we don't need reasoning_content any more?

[guapisolo]

For my sanity check, now our TITO would only support GLM style? Can we still keep it flexible somehow?

If it would be too hard to support Qwen3 chat template, I think it's still good here

This walkaround for glm 4.7 break qwen impl. Need fix to qwen3 chat template. Unblock TITO dev.

[guapisolo]

Remove user in tool to avoid boundary issue around think.

[jybsuper]

LGTM