Fix GLM-4.6v flash tool calling in transformers 5.x

[baonudesifeizhai]

Purpose

fix #31485

Test Plan

export CUDA_VISIBLE_DEVICES=0,1,2,3

vllm serve "zai-org/GLM-4.6V-Flash" \
  --tensor-parallel-size 4 \
  --enable-auto-tool-choice \
  --tool-call-parser glm45 \
  --limit-mm-per-prompt '{"image": 1, "video": 0}' \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 8000

cat > repro_glm46v_toolcall.py <<'EOF'
import base64, json, requests

def b64_image(path: str) -> str:
    data = open(path, "rb").read()
    return base64.b64encode(data).decode("utf-8")

img_b64 = b64_image("test.jpg")

payload = {
  "model": "zai-org/GLM-4.6V-Flash",
  "temperature": 0.2,
  "max_tokens": 256,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }
  ],
  "tool_choice": "auto",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Look at the image, then call get_weather for Tokyo. Only call the tool; do not answer in text."},
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}}
      ]
    }
  ]
}

r = requests.post("http://127.0.0.1:8000/v1/chat/completions", json=payload, timeout=300)
print("status:", r.status_code)
print(json.dumps(r.json(), indent=2, ensure_ascii=False))
EOF

python repro_glm46v_toolcall.py

main branch

python repro_glm46v_toolcall.py
status: 200
{
  "id": "chatcmpl-86e1c31493b07575",
  "object": "chat.completion",
  "created": 1767344497,
  "model": "zai-org/GLM-4.6V-Flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Got it, let's see. The user wants me to call the get_weather function for Tokyo. The image is of a cat, but the instruction is to look at the image then call the tool for Tokyo. So I need to proceed with that. The function call should be get_weather with location \"Tokyo\".\nI will now call the get_weather function for Tokyo.\nget_weather\nlocation\nTokyo\n",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": null,
        "reasoning_content": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": 151338,
      "token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 3436,
    "total_tokens": 3529,
    "completion_tokens": 93,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "kv_transfer_params": null
}

this branch

python repro_glm46v_toolcall.py
status: 200
{
 "id": "chatcmpl-b7ee6c5807040746",
 "object": "chat.completion",
 "created": 1767345552,
 "model": "zai-org/GLM-4.6V-Flash",
 "choices": [
   {
     "index": 0,
     "message": {
       "role": "assistant",
       "content": "<think>Got it, let's see. The user wants me to call the get_weather function for Tokyo. The image is of a cat, but the instruction is to look at the image then call the tool for Tokyo. So I need to proceed with that. The function call should be get_weather with location \"Tokyo\".</think>\nI will now call the get_weather function for Tokyo.\n",
       "refusal": null,
       "annotations": null,
       "audio": null,
       "function_call": null,
       "tool_calls": [
         {
           "id": "chatcmpl-tool-a7572587686f09fd",
           "type": "function",
           "function": {
             "name": "get_weather",
             "arguments": "{\"location\": \"Tokyo\"}"
           }
         }
       ],
       "reasoning": null,
       "reasoning_content": null
     },
     "logprobs": null,
     "finish_reason": "tool_calls",
     "stop_reason": 151338,
     "token_ids": null
   }
 ],
 "service_tier": null,
 "system_fingerprint": null,
 "usage": {
   "prompt_tokens": 3436,
   "total_tokens": 3529,
   "completion_tokens": 93,
   "prompt_tokens_details": null
 },
 "prompt_logprobs": null,
 "prompt_token_ids": null,
 "kv_transfer_params": null
}

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Documentation preview: https://vllm--31622.org.readthedocs.build/en/31622/

[gemini-code-assist]

Code Review

This pull request fixes an issue with tool calling for GLM-4.6v models by ensuring tool call tokens are not skipped during decoding. The core logic of setting skip_special_tokens=False is correct. However, I've identified a potential high-severity issue where the call to super().adjust_request() could enable structured output generation, which conflicts with this model's text-based tool call format and could break tool calling for certain tool_choice options. I have provided a suggestion to rectify this.

[chaunceyjiang]

thanks~