feat(provider/mistral): response_format.type: 'json_schema'
#8130
Conversation
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
|
Hm it currently fails when running
The response does not include any data, it only has a single request body{
"model": "open-mistral-7b",
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"recipe": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"ingredients": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"amount": {
"type": "string"
}
},
"required": [
"name",
"amount"
],
"additionalProperties": false
}
},
"steps": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"name",
"ingredients",
"steps"
],
"additionalProperties": false
}
},
"required": [
"recipe"
],
"additionalProperties": false
},
"strict": false,
"name": "response"
}
},
"messages": [
{
"role": "system",
"content": "JSON schema:\n{\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"properties\":{\"recipe\":{\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"},\"ingredients\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"},\"amount\":{\"type\":\"string\"}},\"required\":[\"name\",\"amount\"],\"additionalProperties\":false}},\"steps\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}}},\"required\":[\"name\",\"ingredients\",\"steps\"],\"additionalProperties\":false}},\"required\":[\"recipe\"],\"additionalProperties\":false}\nYou MUST answer with a JSON object that matches the JSON schema above."
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Generate a lasagna recipe."
}
]
}
]
}curl request
|
| async function main() { | ||
| const result = await generateObject({ | ||
| model: mistral('open-mistral-7b'), | ||
| model: mistral('mistral-small-latest'), |
There was a problem hiding this comment.
see #8130 (comment). Fails with open-mistral-7b
gr2m
changed the title
response_format.type: 'json_schema'
|
Converted back to draft because I got some insight from the kind folks at Mistral
request body{
"model": "open-mistral-7b",
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"type": "object",
"properties": {
"recipe": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"ingredients": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"amount": {
"type": "string"
}
},
"required": [
"name",
"amount"
],
"additionalProperties": false
}
},
"steps": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"name",
"ingredients",
"steps"
],
"additionalProperties": false
}
},
"required": [
"recipe"
],
"additionalProperties": false
},
"strict": false,
"name": "response"
}
},
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Generate a lasagna recipe."
}
]
}
]
} |
…pe` is `json_object`
…e` is `json_object`
gr2m
force-pushed
the
gr2/8127-mistral-json-schema-response
branch
from
a0ff323 to
4ef15d2
Compare
|
The reproduce with curlworks if
|
| // For Mistral we need to need to instruct the model to return a JSON object. | ||
| // https://docs.mistral.ai/capabilities/structured-output/structured_output_overview/ | ||
| if (responseFormat?.type === 'json') { | ||
| if (responseFormat?.type === 'json' && !responseFormat?.schema) { |
There was a problem hiding this comment.
| if (responseFormat?.type === 'json' && !responseFormat?.schema) { | |
| if (responseFormat?.type === 'json' && (!responseFormat?.schema || !structuredOutputs)) { |
When structuredOutputs is false but a JSON schema is provided, the model won't receive any schema information, causing it to generate unstructured JSON instead of following the schema.
View Details
Analysis
The logic for JSON instruction injection has a bug when structuredOutputs is disabled but a JSON response format with schema is requested. Currently, the code only injects JSON instructions when there's no schema (!responseFormat?.schema on line 118), but when structuredOutputs is false, the code falls back to using { type: 'json_object' } format without the schema information (line 151).
This means when:
responseFormat.type === 'json'responseFormat.schemaexistsstructuredOutputs === false
The condition on line 118 evaluates to false (because schema exists), so no JSON instructions are injected, and the condition on line 141 also evaluates to false (because structuredOutputs is false), so it uses json_object format without schema information. The model receives no guidance about the expected JSON structure, likely producing unstructured output instead of following the provided schema.
Recommendation
Update the condition on line 118 to inject JSON instructions when either there's no schema OR when there's a schema but structuredOutputs is disabled:
if (responseFormat?.type === 'json' && (!responseFormat?.schema || !structuredOutputs)) {This ensures that JSON instructions with schema information are injected when structuredOutputs is disabled, providing the model with the necessary schema guidance through the fallback instruction method.
We could leave out the |
Note to self: was told that "this would not happen if you set strict == True". Will have a look |
… by Mistral team
done via 178031f. This should be good to go now. |
|
will default |
) ## Background We are currently not exporting a type for `providerOptions` ## Summary We have to take into account that provider options can be different depending on what is used: - language model - image model - embedding So instead of `MistralProviderOptions` we settled on `MistralLanguageModelOptions`. We considered `MistralLanguageModelProviderOptions` but rejected it due to its length. ## Manual Verification [examples/ai-core/src/generate-text/mistral-provider-options.ts](https://github.com/vercel/ai/blob/a380bdc31176cf81efe5964e75fbc389d134e435/examples/ai-core/src/generate-text/mistral-provider-options.ts#L9-L17) ## Tasks - [ ] ~~Tests have been added / updated (for bug fixes / features)~~ we don't currently do test of module exports, we use examples instead - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Related Issues Addresses #8130 (comment)
gr2m
added
ai/provider
provider/mistral
Background
The mistral API now supports
json_schemaas response format.Summary
If
responseFormat.typeisjsonandresponseFormat.schemais set, then take advantage of Mistral's support ofjson_schemaresponse formatManual Verification
Build first
Run test
Tasks
pnpm changesetin the project root)pnpm prettier-fixin the project root)Related Issues
closes #8127