[BUG] Kimi K2 Thinking doesn't work very well with Chutes.AI

[NoobixCube]

Problem (one or two sentences)

There seems to be an issue with tool calls, prompts or API usage for Kimi K2 Thinking on Chutes.AI.
I can sometimes get good results using the model, but it seems sensitive to prompt wording or roo response.

I'm not sure if that means we need to call tools a different way or what.

It sounds like Roo Code online and OpenRouter already know how to properly use this model so we just need to follow that approach.

Context (who is affected and when)

I assume anyone trying to use Kimi K2 Thinking through chutes.ai will be affected.

Reproduction steps

Using the Roo Code VS code extension, select Chutes.AI as a provider and Kimi K2 Thinking as the model.
Generate a request from a mode similar to Orchestrator and I would expect the error to occur again.
Typically this manifests in a "retry" prompt from Roo Code.

Expected result

Roo takes my prompt and uses Kimi K2 Thinking to generate a result

Actual result

Roo takes my prompt and I end up in an endless chain of "retry" prompts but can see the Kimi K2 Thinking output which appears to be the next step.

Variations tried (optional)

No response

App Version

Version: 3.32.1

API Provider (optional)

Chutes AI

Model Used (optional)

Kimi K2 Thinking

Roo Code Task Links (optional)

No response

Relevant logs or errors (optional)

Roo said
I'll help you upgrade the TeamSelect scene by analyzing both scenes and creating a comprehensive plan. Let me start by exploring the codebase to find these scenes.  <|tool_calls_section_begin|>   <|tool_call_begin|>   functions.codebase_search:12  <|tool_call_argument_begin|>  {"query": "TeamSelect scene", "path": "ouroboros"}  <|tool_call_end|> <|tool_calls_section_end|>


API Request
$0.0000


Roo said
I'll help you upgrade the TeamSelect scene by analyzing both scenes and creating a comprehensive plan. Let me start by exploring the codebase to find these scenes.  <|tool_calls_section_begin|> <|tool_call_begin|>  functions.codebase_search:13 <|tool_call_argument_begin|> {"query": "TeamSelect scene", "path": "ouroboros"} <|tool_call_end|> <|tool_calls_section_end|>


API Request
$0.0000


Roo said
I'll help you upgrade the TeamSelect scene by analyzing both scenes and creating a comprehensive plan. Let me start by exploring the codebase to find these scenes.  <|tool_calls_section_begin|>   <|tool_call_begin|>   functions.codebase_search:14  <|tool_call_argument_begin|>  {"query": "TeamSelect scene", "path": "ouroboros"}  <|tool_call_end|> <|tool_calls_section_end|>


Roo is having trouble...
This may indicate a failure in the model's thought process or inability to use a tool properly, which can be mitigated with some user guidance (e.g. "Try breaking down the task into smaller steps").


API Request
$0.0000

I'll help you upgrade the TeamSelect scene by analyzing both scenes and creating a comprehensive plan. Let me start by exploring the codebase to find these scenes.  <|tool_calls_section_begin|> <|tool_call_begin|>  functions.codebase_search:15 <|tool_call_argument_begin|> {"query": "TeamSelect scene", "path": "ouroboros"} <|tool_call_end|> <|tool_calls_section_end|>


API Request
$0.0000


Roo said
I'll help you upgrade the TeamSelect scene by analyzing both scenes and creating a comprehensive plan. Let me start by exploring the codebase to find these scenes.  <|tool_calls_section_begin|>   <|tool_call_begin|>   functions.codebase_search:15  <|tool_call_argument_begin|> {"query": "TeamSelect scene", "path": "ouroboros"}  <|tool_call_end|> <|tool_calls_section_end|>


API Request
$0.0000


Roo said
I'll help you upgrade the TeamSelect scene by analyzing both scenes and creating a comprehensive plan. Let me start by exploring the codebase to find these scenes.   <|tool_calls_section_begin|> <|tool_call_begin|>  functions.codebase_search:16 <|tool_call_argument_begin|> {"query": "TeamSelect scene", "path": "ouroboros"} <|tool_call_end|> <|tool_calls_section_end|>


API Request
$0.0000


Roo said
I'll help you upgrade the TeamSelect scene by analyzing both scenes and creating a comprehensive plan. Let me start by exploring the codebase to find these scenes.   <|tool_calls_section_begin|>   <|tool_call_begin|>   functions.codebase_search:17 <|tool_call_argument_begin|> {"query": "TeamSelect scene", "path": "ouroboros"} <|tool_call_end|> <|tool_calls_section_end|>


API Request
$0.0000


Error
Unexpected API Response: The language model did not provide any assistant messages. This may indicate an issue with the API or the model's output.


API Request
$0.0000

Rate limiting for 1 seconds...

[roomote]

I'm exploring a potential fix for the Kimi K2 Thinking model compatibility issue with Chutes.AI. The model appears to be using a different tool calling format that needs proper parsing. I'll open a PR to share progress.

[roomote]

Opened PR #9368 with my attempt at addressing this issue. The PR adds a specialized parser for the Kimi K2 tool call format that uses the <|tool_calls_section_begin|> style markers. Feedback welcome!

[Lissanro]

Very recently in ik_llama.cpp it was fixed by ikawrakow/ik_llama.cpp#958 - before the patch I had the same issue in Roo Code, after the patch it was fixed. I was running the model locally though.

I do not know what backend Chutes.AI is using but there is a possibility they need to update it (unless it is possible to workaround somehow on Roo's side?). For example, llama.cpp also had similar bug and it was fixed by ggml-org/llama.cpp#16932. So multiple backends were affected.