OpenAI /completions route fails

[iverly]

Hello,

Thank you the new release including the OpenAI routes but after a try, it always returns the following error using the raw request of the README.md:

llama.cpp/server/json.h:21313: assert(it != m_value.object->end()) failed (cosmoaddr2line /Users/maxime-georide/Downloads/llamafile-server-0.2 1000000fe3c 1000001547c 100000162e8 10000042748 1000004ffdc 10000050cb0 1000005124c 100000172dc 1000001b370 10000181e78 1000019d3d0)

curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{
    "role": "system",
    "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
},
{
    "role": "user",
    "content": "Write a limerick about python exceptions"
}
]
}'

I'm on M1 with mistral-7b-instruct-v0.1.Q5_K_M.gguf and llama-2-7b-chat.Q5_K_S.gguf models.

I didn't try with the OpenAI SDK.

[iverly]

I've the same behaviour with the OpenAI SDK.

With the following Python script :

import openai

client = openai.OpenAI(
    base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
    api_key = "sk-no-key-required"
)

completion = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
    {"role": "system", "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."},
    {"role": "user", "content": "Write a limerick about python exceptions"}
]
)

print(completion.choices[0].message)

[lucido-simon]

Hi. I have the same behavior on Linux/x86

[iverly]

Still the same behaviour with 2.1.0.

[Mardak]

It looks like "stop" is required. Try passing in null. This works for me:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "stop": null,
  "messages": [{
    "role": "user",
    "content": "Write a limerick about python exceptions"
  }]
}'

@ggerganov https://github.com/Mozilla-Ocho/llamafile/blob/73ee0b1e66a89fba701843e9be067c24e8fe93d7/llama.cpp/server/server.cpp#L2402-L2408