feat(auto-model): add reasoning support - #768
Merged
Merged
Conversation
![terragon-labs[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
- Expanded allowedAutoModels to include reasoning-capable models when reasoning_effort is specified - Updated provider filtering to consider reasoning capability along with context size - Ensures models/providers supporting reasoning are prioritized when reasoning_effort is used Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
Contributor
WalkthroughUpdates chat request handling to adjust reasoning_effort validation and auto-routing filtering. Validation now applies only to non-auto/custom models. Auto-routing filters providers by context size and, when reasoning_effort is present, by reasoning capability. No exported signatures changed. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant C as Client
participant G as Chat Gateway
participant R as Provider Registry
participant P as Provider
Note over C,G: Request includes model and optional reasoning_effort
alt model is non-auto/custom
G->>R: Check if any provider supports reasoning (when reasoning_effort defined)
alt no reasoning-capable provider
G-->>C: 400 Bad Request (reasoning unsupported)
else provider(s) exist
G->>P: Forward request to selected provider
P-->>G: Response
G-->>C: Response
end
else model is auto/custom
G->>R: Get candidate providers
alt reasoning_effort defined
rect rgba(200,240,255,0.3)
Note over G,R: Filter by context size AND reasoning==true
end
else no reasoning_effort
Note over G,R: Filter by context size only
end
G->>P: Route to chosen provider
P-->>G: Response
G-->>C: Response
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Pre-merge checks (3 passed)✅ Passed checks (3 passed)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
changed the title
…oning-in-auto-model
Eliminated reasoning-capable models from the auto selection list when `reasoning_effort` is specified. This ensures the list only contains hardcoded models for consistent behavior.
Skip reasoning support check for models set to "auto" or "custom" since they are resolved dynamically. This prevents incorrect validation failures when reasoning_effort is specified for these model types. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
steebchen
marked this pull request as ready for review
September 10, 2025 00:23
steebchen
added this pull request to the merge queue
Sep 10, 2025
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/gateway/src/chat/chat.ts (1)
2214-2244: Validate reasoning support against the explicitly requested provider (not just “any” provider).Current check passes if any provider for the model supports reasoning, which allows
reasoning_effortwith a provider that doesn’t. This will later misroute or cause upstream errors. When a provider is explicitly requested (e.g.,openai/...), validate that provider’s mapping hasreasoning === true; otherwise, keep the “any provider” check.Apply:
-// Skip this check for "auto" and "custom" models as they will be resolved dynamically -if ( - reasoning_effort !== undefined && - requestedModel !== "auto" && - requestedModel !== "custom" -) { - // Check if any provider for this model supports reasoning - const supportsReasoning = modelInfo.providers.some( - (provider) => (provider as ProviderModelMapping).reasoning === true, - ); - - if (!supportsReasoning) { +// Skip this check for "auto" and "custom" models as they will be resolved dynamically +if ( + reasoning_effort !== undefined && + requestedModel !== "auto" && + requestedModel !== "custom" +) { + // If a provider was explicitly requested, require that exact mapping to support reasoning. + // Otherwise, ensure at least one provider supports reasoning. + const supportsReasoningForRequested = + requestedProvider + ? modelInfo.providers.some( + (p) => + p.providerId === requestedProvider && + (p as ProviderModelMapping).reasoning === true, + ) + : modelInfo.providers.some( + (p) => (p as ProviderModelMapping).reasoning === true, + ); + + if (!supportsReasoningForRequested) { logger.error( `Reasoning effort specified for non-reasoning model: ${requestedModel}`, { requestedModel, requestedProvider, reasoning_effort, modelProviders: modelInfo.providers.map((p) => ({ providerId: p.providerId, reasoning: (p as ProviderModelMapping).reasoning, })), }, ); throw new HTTPException(400, { message: `Model ${requestedModel} does not support reasoning. Remove the reasoning_effort parameter or use a reasoning-capable model.`, }); } }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/gateway/src/chat/chat.ts(2 hunks)
🧰 Additional context used
📓 Path-based instructions (5)
**/*.{js,jsx,ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use localStorage instead of cookies for client-side data persistence
Files:
apps/gateway/src/chat/chat.ts
**/*.{js,ts}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{js,ts}: Use drizzle with the latest object syntax for database operations
For read queries, always usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/chat.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
Never use
as anyor: anyin TypeScript files.
Files:
apps/gateway/src/chat/chat.ts
apps/{api,gateway}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
.findMany() or db().query.
apps/{api,gateway}/**/*.{ts,tsx}: Use Drizzle with the latest object syntax for database operations
For read queries, use db().query..findFirst()
apps/{api,gateway}/**/*.{ts,tsx}: Use the Hono framework for backend HTTP services in apps/api and apps/gateway
Use Zod for request/response validation in backend routes and handlers
Maintain OpenAPI/Swagger documentation for backend APIsFiles:
apps/gateway/src/chat/chat.tsapps/gateway/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
In apps/gateway (Hono), always use Hono + Zod + OpenAPI for validation and typesafety
Files:
apps/gateway/src/chat/chat.ts🧬 Code graph analysis (1)
apps/gateway/src/chat/chat.ts (1)
packages/models/src/models.ts (1)
ProviderModelMapping(23-100)
Comment on lines
+2456
to
2471
| // Filter by context size requirement and reasoning capability if needed | ||
| const suitableProviders = availableModelProviders.filter((provider) => { | ||
| // Use the provider's context size, defaulting to a reasonable value if not specified | ||
| const modelContextSize = provider.contextSize ?? 8192; | ||
| return modelContextSize >= requiredContextSize; | ||
| const contextSizeMet = modelContextSize >= requiredContextSize; | ||
|
|
||
| // If reasoning_effort is specified, only include providers that support reasoning | ||
| if (reasoning_effort !== undefined) { | ||
| return ( | ||
| contextSizeMet && | ||
| (provider as ProviderModelMapping).reasoning === true | ||
| ); | ||
| } | ||
|
|
||
| return contextSizeMet; | ||
| }); |
Contributor
There was a problem hiding this comment.
Auto-routing ignores reasoning_effort because allowedAutoModels isn’t expanded; fallback silently drops the requirement.
- You correctly filter providers by
reasoningwhenreasoning_effortis set, but the candidate model list stays fixed to nanos. If none of those nanos are reasoning-capable, selection fails and we fall back to a non-reasoning model, violating the request/PR objective. - Expand
allowedAutoModelsto include all reasoning-capable models whenreasoning_effortis provided, and avoid defaulting to non-reasoning models when none are available.
Add after defining allowedAutoModels (and after the free_models_only branch):
// If reasoning is requested, consider all reasoning-capable models for auto routing
if (reasoning_effort !== undefined) {
const reasoningModelIds = Array.from(
new Set(
models
.filter(
(m) =>
m.id !== "auto" &&
m.id !== "custom" &&
m.providers?.some((p) => (p as ProviderModelMapping).reasoning === true),
)
.map((m) => m.id),
),
);
allowedAutoModels = Array.from(new Set([...allowedAutoModels, ...reasoningModelIds]));
}Also, enforce the requirement in the fallback block so we don’t silently choose a non-reasoning model:
// Before default fallback
if (!selectedModel) {
if (reasoning_effort !== undefined) {
throw new HTTPException(400, {
message:
"No reasoning-capable providers available for auto routing. Remove reasoning_effort or choose a specific reasoning model.",
});
}
// existing non-reasoning fallback
usedModel = "gpt-5-nano";
usedProvider = "openai";
}🤖 Prompt for AI Agents
In apps/gateway/src/chat/chat.ts around lines 2456 to 2471, the auto-routing
logic filters providers by reasoning but never expands allowedAutoModels to
include non-nano reasoning-capable models (so the candidate set can lack
reasoning providers) and the fallback silently picks a non-reasoning model; fix
it by, immediately after allowedAutoModels is defined (and after the
free_models_only branch), append all reasoning-capable model ids to
allowedAutoModels when reasoning_effort is set (dedupe via a Set), and in the
fallback where selectedModel is null, if reasoning_effort is provided throw an
HTTPException(400) with a clear message about no reasoning-capable providers
available instead of falling back to a non-reasoning model.
This was referenced Oct 21, 2025
This was referenced Nov 1, 2025
This was referenced Nov 17, 2025
This was referenced Jan 30, 2026
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
reasoning_effortis specifiedChanges
Core Functionality
reasoning_effortis providedreasoning_effortis specifiedTest plan
reasoning_effortis setreasoning_effortis specifiedreasoning_effortis not provided🌿 Generated by Terry
ℹ️ Tag @terragon-labs to ask questions and address PR feedback
📎 Task: https://www.terragonlabs.com/task/2be69044-f5c3-4659-8752-af969fc9c0e5
Summary by CodeRabbit