Output is truncated when using Kimi K2

[greyishsong]

App Version

3.25.10 (c75549c)

API Provider

OpenAI Compatible

Model Used

Kimi K2

Roo Code Task Links (Optional)

No response

🔁 Steps to Reproduce

1. Use the latest VS Code (on Arch Linux) and the latest stable version of Roo Code
2. Switch to Kimi K2 (use the official Moonshot provider, both kimi-k2-0711-preview and kimi-k2-turbo-preview are ok), following the official instructions.
3. Prepare a task that is big enough, e.g., implement a Vue component requiring 200+ lines of code
4. Roo Code is able to create or modify the target file, but the output is truncated

💥 Outcome Summary

The newly created file is truncated. In my Moonshot console, I can see that the model's output is truncated exactly to 1024 tokens even if I set the max token limit to -1 or 32000.

What's more, I have also tried to set max_tokens: 32000 through a custom header according to Moonshot's Chat API doc, but this doesn't work at all. During the whole process, no error is reported.

It seems that a wrong max_tokens parameter is very likely to be the reason for truncation. I have tried to use VS Code's DevTool to check the header of the API request, but I cannot find Roo Code's request on it.

📄 Relevant Logs or Errors (Optional)

[roomote]

Hi @greyishsong! I'm here to help out the maintainers and am going to investigate this issue with the Kimi K2 model output truncation. I'll look into why the max_tokens parameter isn't being properly passed through the OpenAI Compatible API provider and work on a fix. Thanks for the detailed bug report!

[greyishsong]

I see the PR automatically generated by roomote bot.

The fix modifies the BaseOpenAiCompatibleProvider class to only include the max_tokens parameter when the includeMaxTokens option is explicitly set to true. This should resolve the truncation issue you were experiencing with the Kimi K2 model.

However, I think this PR goes the wrong way. According to Moonshot's document about the max_tokens parameter, 1024 is likely to be the default value if max_tokens is not included in the header of API request:

It is recommended to provide a reasonable value as needed. If not provided, we will use a good default integer like 1024.

And the document says,

The maximum number of tokens to generate for the chat completion. If the result reaches the maximum number of tokens without ending, the finish reason will be "length"; otherwise, it will be "stop"

So it may be a good choice to review the response from Moonshot's server and check whether the value of finish_reason is "length". Because I could not find the network request in VS Code DevTool yesterday, I tried to use Wireshark to capture all requests to api.moonshot.cn, then found all the requests and responses were encrypted and so invisible. I am not familiar with Web development. What can I do to help you debug this problem on my machine?

[greyishsong]

Moonshot provides an official Anthropic style proxy through https://api.moonshot.cn/anthropic, I tried these settings in Roo Code and found that the output is no more truncated:

• provider: Anthropic
• API key: my Kimi API key
• use custom base URL https://api.moonshot.cn/anthropic (I guess https://api.moonshot.ai/anthropic will also works if your Kimi API key is created on platform.moonshot.ai instead of platform.moonshot.cn)
• pass the API key as Authorization header, not X-Api-key
• model: any model in the list should be ok because kimi-k2-0711-preview actually responses behind the proxy
• turn off browser use

I see that when using Anthropic as provider, Roo Code will use Anthropic SDK instead of OpenAI SDK for API request. So now I believe the problem should be related with the "OpenAI compatible" provider.