Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion apps/docs/content/features/routing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ curl -X POST "https://api.llmgateway.io/v1/chat/completions" \

## Optimized Auto Routing

Our most powerful feature is **optimized auto routing**, which automatically selects the best model for your specific use case without you having to specify a model at all.
Auto routing automatically selects the best model for your specific use case without you having to specify a model at all.

### Current Implementation

Expand Down Expand Up @@ -120,6 +120,44 @@ curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
the API will return an error.
</Callout>

### Reasoning models only

Just specify the `reasoning_effort` value and only a model which supports reasoning will be chosen. This parameter is not specific to the auto model.

```bash
# Auto route only to reasoning models
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
-H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "auto",
"messages": [{"role": "user", "content": "Hello!"}],
"reasoning_effort": "medium"
}'
```

### Exclude Reasoning Models

When using auto routing, you can exclude reasoning models from selection by setting the `no_reasoning` parameter to `true`. This is useful when you want faster responses or need to avoid the additional cost and latency of reasoning models:

```bash
# Auto route excluding reasoning models
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
-H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "auto",
"messages": [{"role": "user", "content": "Hello!"}],
"no_reasoning": true
}'
```

<Callout type="info">
The `no_reasoning` parameter only works with auto routing (`"model": "auto"`).
If no non-reasoning models are available that meet your request requirements,
the API will return an error.
</Callout>

<Callout type="success">
Auto routing analyzes your payload and automatically chooses between
cost-effective models for simple requests and more powerful models for complex
Expand Down
26 changes: 26 additions & 0 deletions apps/gateway/src/chat/chat.ts
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,11 @@ const completionsRequestSchema = z.object({
"When used with auto routing, only route to free models (models with zero input and output pricing)",
example: false,
}),
no_reasoning: z.boolean().optional().default(false).openapi({
description:
"When used with auto routing, exclude reasoning models from selection",
example: false,
}),
});

const completions = createRoute({
Expand Down Expand Up @@ -387,6 +392,7 @@ chat.openapi(completions, async (c) => {
tools,
tool_choice,
free_models_only,
no_reasoning,
} = validationResult.data;

// Extract reasoning_effort as mutable variable for auto-routing modification
Expand Down Expand Up @@ -903,6 +909,14 @@ chat.openapi(completions, async (c) => {
const modelContextSize = provider.contextSize ?? 8192;
const contextSizeMet = modelContextSize >= requiredContextSize;

// If no_reasoning is true, exclude reasoning models
if (
no_reasoning &&
(provider as ProviderModelMapping).reasoning === true
) {
return false;
}

// If reasoning_effort is specified, only include providers that support reasoning
if (reasoning_effort !== undefined) {
return (
Expand Down Expand Up @@ -965,6 +979,12 @@ chat.openapi(completions, async (c) => {
message:
"No free models are available for auto routing. Remove free_models_only parameter or use a specific model.",
});
} else if (no_reasoning) {
// If no non-reasoning models are available, return error
throw new HTTPException(400, {
message:
"No non-reasoning models are available for auto routing. Remove no_reasoning parameter or use a specific model.",
});
}
} else {
if (free_models_only) {
Expand All @@ -973,6 +993,12 @@ chat.openapi(completions, async (c) => {
message:
"No free models are available for auto routing. Remove free_models_only parameter or use a specific model.",
});
} else if (no_reasoning) {
// If no_reasoning is true but no suitable model found, return error
throw new HTTPException(400, {
message:
"No non-reasoning models are available for auto routing. Remove no_reasoning parameter or use a specific model.",
});
}
// Default fallback if no suitable model is found - use cheapest allowed model
usedModel = "gpt-5-nano";
Expand Down
Loading