[FEAT]: OpenAI Harmony support

[wcwong]

What would you like to see?

The OpenAI open weight models (gpt-oss-20b, gpt-oss-120b) are using the OpenAI Harmony Response Format. This does not appear to be compatible with the generic OpenAI response format as connecting AnythingLLM to these models does not result in properly parsed responses.

[Nikvothe]

I am looking for a frontend that can support this, as OpenWebUI doesn't support it either.

[chaserhkj]

What inference back end are you guys using? I have been using llama.cpp's OAI-compatible interface and it is running gpt-oss-20b fine with both OpenWebUI and AnythingLLM since the merge of ggml-org/llama.cpp#15181

Edit: This post is a good summary of what flags need to be used with llama.cpp to makes this work, if you want to try that route.

[wcwong]

First, apologies for an error on my part. I should have looked more closely and the garbage that was being generated was directly from the model -- turns out you can't run the NVIDA NIM for oss120b on an A100 even though the self-check says that there is a supported profile...

After getting it running on an H100 and verifying that I'm getting coherent output, the issue with AnythingLLM is that it doesn't appear to be recognizing all the tags.

It looks like 'analysis' may be used instead of the other thinking tab. Sample output from AnythingLLM 1.8.5-42

analysisUser asks a tongue twister/riddle. We can answer humorously. Provide the classic answer: "A woodchuck would chuck as much wood as a woodchuck could chuck if a woodchuck could chuck wood." Or mention the study: about 700 pounds. The classic answer: "It would chuck approximately 700 pounds (or 317 kg) of wood." Provide both. Since user not asking medical, but we can answer. So respond with fun answer.assistantfinalIt’s a classic tongue‑twister, so the “official” answer is just as whimsical as the question:

**“A woodchuck would chuck as much wood as a woodchuck could chuck if a woodchuck could chuck wood!”**

That said, a playful little study by a wildlife biologist once estimated that, based on the amount of dirt a woodchuck moves when digging a burrow, a woodchuck could theoretically move roughly **700 pounds (about 317 kg) of wood** if it were inclined to do so. 

Either way, it’s a fun riddle—no need to worry about any actual wood‑chucking critters in the neighborhood!