feat(api): add no_reasoning flag - #1080
Conversation
![terragon-labs[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
…om auto routing - Introduced `no_reasoning` boolean parameter in chat completions API. - When set to true with `model: auto`, excludes reasoning models from selection. - Returns an error if no suitable non-reasoning models are available. - Updated documentation with usage example and parameter description. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
❌ Preview Environment deleted from BunnyshellAvailable commands (reply to this comment):
|
WalkthroughAdded a Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant API as API Handler
participant Router as Auto-Router
participant Provider as Provider Selector
Client->>API: POST /completions\n{ no_reasoning: true, reasoning_effort: ... }
API->>API: Validate request (includes no_reasoning)
API->>Router: Invoke auto-routing with flags
rect rgb(240,248,255)
note over Router: Filter providers\n- apply free_models_only\n- apply reasoning_effort\n- if no_reasoning -> exclude reasoning-enabled
Router->>Router: Filter candidate providers
end
alt non-reasoning providers found
Router->>Provider: Select best non-reasoning model
Provider-->>Client: Stream/Return completion
else none available
Router-->>Client: 400 Error — No suitable non-reasoning models
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
apps/gateway/src/chat/chat.ts (2)
976-987: Consider improving error message clarity when both flags are set.When both
free_models_onlyandno_reasoningare true and no models are found, only thefree_models_onlyerror (lines 976-981) will be shown due to the else-if chain. While technically correct, this could be confusing if the actual issue is that no free non-reasoning models exist.Consider consolidating or enhancing the error message to account for both flags:
- } else if (free_models_only) { + } else if (free_models_only && no_reasoning) { + // If no free models are available, return error + throw new HTTPException(400, { + message: + "No free non-reasoning models are available for auto routing. Remove free_models_only or no_reasoning parameters or use a specific model.", + }); + } else if (free_models_only) { // If no free models are available, return error throw new HTTPException(400, {
990-1001: Same error message clarity issue applies here.This error handling block has the same issue as lines 976-987. When both
free_models_onlyandno_reasoningare true, only thefree_models_onlyerror will be shown. The same consolidation approach suggested in the previous comment would improve clarity here as well.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
apps/docs/content/features/routing.mdx(1 hunks)apps/gateway/src/chat/chat.ts(5 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level import; never use require() or dynamic import()
Files:
apps/gateway/src/chat/chat.ts
apps/{gateway,api}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/**/*.ts: Use Hono for HTTP routing in Gateway and API services
Use Zod schemas for request/response validation in server routes
Files:
apps/gateway/src/chat/chat.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyin this TypeScript project unless absolutely necessary
Always use top-levelimport; do not userequireor dynamicimport()
Files:
apps/gateway/src/chat/chat.ts
{apps/{api,gateway}/**/*.ts,packages/db/**/*.ts}
📄 CodeRabbit inference engine (CLAUDE.md)
For read operations, use
db().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/chat.ts
🧬 Code graph analysis (1)
apps/gateway/src/chat/chat.ts (1)
packages/models/src/models.ts (1)
ProviderModelMapping(23-120)
🔇 Additional comments (4)
apps/gateway/src/chat/chat.ts (3)
228-232: LGTM! Schema definition follows best practices.The
no_reasoningparameter definition is consistent with thefree_models_onlyparameter above it, using appropriate types and defaults.
395-395: LGTM! Parameter extraction is correct.The
no_reasoningparameter is properly extracted from the validation result alongside other parameters.
912-918: LGTM! Filtering logic correctly excludes reasoning models.The implementation properly filters out reasoning-capable providers when
no_reasoningis true, and the placement within thesuitableProvidersfilter ensures this happens before price calculations.apps/docs/content/features/routing.mdx (1)
123-143: LGTM! Documentation is clear and well-structured.The new documentation section effectively explains the
no_reasoningparameter, provides a practical curl example, and appropriately warns users about its limitations. The structure and style are consistent with the existing documentation.
Clarified the explanation of optimized auto routing and added details on specifying reasoning models and excluding reasoning models.
1 participant
Summary
Changes
API / Completions
Auto-routing logic
Documentation
Documentation Example (Snippet)
Test plan
Notes
🌿 Generated by Terry
ℹ️ Tag @terragon-labs to ask questions and address PR feedback
📎 Task: https://www.terragonlabs.com/task/ab95c290-3e3a-4246-ad31-c971ae8c5862
Summary by CodeRabbit
New Features
no_reasoningoption for completions to exclude reasoning-enabled models during auto-routing; surfaces clear errors when no suitable models remain.Documentation
reasoning_effortandno_reasoning, and updated guidance (includes large-context request behavior).