feat: add vLLM channel type with native dual-format routing - #5912
Open
mlzy wants to merge 2 commits into
Open
Conversation
Add a dedicated vLLM channel type (59) that routes requests based on
the relay format without protocol conversion:
- Claude format (Anthropic API) → {baseURL}/v1/messages
- OpenAI format → {baseURL}/v1/chat/completions (or embeddings/rerank)
This preserves Anthropic prompt cache headers end-to-end when clients
send Claude-format requests to a vLLM instance, which was broken when
using the generic OpenAI channel type (forces Claude→OpenAI conversion).
Unlike the Moonshot channel's SpecialBases mechanism, vLLM uses the
actual configured base URL directly — no magic keyword required.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
✅ Files skipped from review due to trivial changes (1)
WalkthroughThis PR adds vLLM as a supported channel type. It introduces backend constants and routing, implements a new relay adaptor, and adds matching web channel metadata for the new type. ChangesvLLM Channel Integration
Estimated code review effort: 2 (Simple) | ~15 minutes Sequence Diagram(s)sequenceDiagram
participant RelayAdaptor
participant VLLMAdaptor
participant ClaudeAdaptor
participant OpenAIAdaptor
RelayAdaptor->>VLLMAdaptor: GetAdaptor(APITypeVLLM)
VLLMAdaptor->>VLLMAdaptor: GetRequestURL(info)
VLLMAdaptor->>VLLMAdaptor: SetupRequestHeader(...)
alt Claude format
VLLMAdaptor->>ClaudeAdaptor: ConvertClaudeRequest(...)
ClaudeAdaptor-->>VLLMAdaptor: converted request
else image request
VLLMAdaptor->>OpenAIAdaptor: ConvertImageRequest(...)
OpenAIAdaptor-->>VLLMAdaptor: converted request
end
alt Claude response format
VLLMAdaptor->>ClaudeAdaptor: DoResponse(...)
else
VLLMAdaptor->>OpenAIAdaptor: DoResponse(...)
end
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@relay/channel/vllm/adaptor.go`:
- Around line 24-40: The GetRequestURL routing in Adaptor currently misses
RelayModeImagesGenerations and incorrectly falls back to chat completions.
Update the RelayInfo-based switch in GetRequestURL to add a
RelayModeImagesGenerations branch that returns the /v1/images/generations
endpoint, alongside the existing embeddings, rerank, and completions cases, so
ConvertImageRequest calls are sent to the correct upstream path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9c10235e-1eff-48e9-9996-48dfc2489d99
📒 Files selected for processing (6)
common/api_type.goconstant/api_type.goconstant/channel.gorelay/channel/vllm/adaptor.gorelay/channel/vllm/constants.gorelay/relay_adaptor.go
Comment on lines
+24
to
+40
| func (a *Adaptor) GetRequestURL(info *relaycommon.RelayInfo) (string, error) { | ||
| switch info.RelayFormat { | ||
| case types.RelayFormatClaude: | ||
| return fmt.Sprintf("%s/v1/messages", info.ChannelBaseUrl), nil | ||
| default: | ||
| switch info.RelayMode { | ||
| case relayconstant.RelayModeEmbeddings: | ||
| return fmt.Sprintf("%s/v1/embeddings", info.ChannelBaseUrl), nil | ||
| case relayconstant.RelayModeRerank: | ||
| return fmt.Sprintf("%s/v1/rerank", info.ChannelBaseUrl), nil | ||
| case relayconstant.RelayModeCompletions: | ||
| return fmt.Sprintf("%s/v1/completions", info.ChannelBaseUrl), nil | ||
| default: | ||
| return fmt.Sprintf("%s/v1/chat/completions", info.ChannelBaseUrl), nil | ||
| } | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the relevant files and nearby adaptors for routing/image support.
git ls-files 'relay/channel/vllm/*' 'relay/channel/*/adaptor.go' | sed -n '1,120p'
echo '--- vllm adaptor outline ---'
ast-grep outline relay/channel/vllm/adaptor.go --view expanded || true
echo '--- vllm adaptor excerpt ---'
cat -n relay/channel/vllm/adaptor.go | sed -n '1,220p'
echo '--- search for RelayModeImagesGenerations routing in adaptors ---'
rg -n "RelayModeImagesGenerations|images/generations|ConvertImageRequest" relay/channel -g 'adaptor.go' -g '*.go'Repository: QuantumNous/new-api
Length of output: 16659
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- openai adaptor URL routing and image conversion ---'
ast-grep outline relay/channel/openai/adaptor.go --view expanded || true
echo '--- openai adaptor excerpt around URL routing ---'
cat -n relay/channel/openai/adaptor.go | sed -n '1,220p'
echo '--- openai adaptor excerpt around image conversion / response routing ---'
cat -n relay/channel/openai/adaptor.go | sed -n '400,720p'
echo '--- shared request flow ---'
ast-grep outline relay/channel/adapter.go --view expanded || true
cat -n relay/channel/adapter.go | sed -n '1,220p'
echo '--- search for GetRequestURL usage ---'
rg -n "GetRequestURL\\(" relay/channel -g '*.go'Repository: QuantumNous/new-api
Length of output: 35779
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- vllm constants ---'
cat -n relay/channel/vllm/constants.go | sed -n '1,220p'
echo '--- search for vllm channel registration / supported relay modes ---'
rg -n "vllm|RelayModeImagesGenerations|RelayModeImagesEdits" relay -g '*.go'Repository: QuantumNous/new-api
Length of output: 4206
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- api request dispatch around conversion selection ---'
cat -n relay/channel/api_request.go | sed -n '250,430p'
echo '--- search for ConvertImageRequest dispatch ---'
rg -n "ConvertImageRequest\\(|RelayModeImagesGenerations|RelayModeImagesEdits" relay/channel/api_request.go relay/helper relay -g '*.go'Repository: QuantumNous/new-api
Length of output: 17151
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- image handler ---'
cat -n relay/image_handler.go | sed -n '1,220p'
echo '--- image request validation ---'
cat -n relay/helper/valid_request.go | sed -n '120,190p'
echo '--- openai image request test ---'
cat -n relay/channel/openai/image_edit_test.go | sed -n '1,140p'Repository: QuantumNous/new-api
Length of output: 13515
Route image generation to the image endpoint RelayModeImagesGenerations reaches ConvertImageRequest, but GetRequestURL still falls through to /v1/chat/completions. Add a branch here for /v1/images/generations so image-generation requests are sent to the correct upstream path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@relay/channel/vllm/adaptor.go` around lines 24 - 40, The GetRequestURL
routing in Adaptor currently misses RelayModeImagesGenerations and incorrectly
falls back to chat completions. Update the RelayInfo-based switch in
GetRequestURL to add a RelayModeImagesGenerations branch that returns the
/v1/images/generations endpoint, alongside the existing embeddings, rerank, and
completions cases, so ConvertImageRequest calls are sent to the correct upstream
path.
Calcium-Ion
force-pushed
the
main
branch
2 times, most recently
from
August 30, 2026 15:03
51fdfc5 to
2b6f1df
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
{baseURL}/v1/messages(passthrough, preserves prompt cache headers){baseURL}/v1/chat/completions(or/v1/embeddings,/v1/rerankby relay mode)Motivation
vLLM natively supports both the Anthropic Messages API (
/v1/messages) and the OpenAI Chat Completions API (/v1/chat/completions). When using the generic Custom (OpenAI) channel type to point at a local vLLM instance, New API forces all Claude-format requests throughClaudeToOpenAIRequest, losing Anthropic prompt cache headers and causing intermittent cache misses.The existing workaround — using the Moonshot channel type — routes Claude requests to
{baseURL}/anthropic/v1/messages, which is the Moonshot/Kimi path layout and does not match vLLM's standard/v1/messagesendpoint.Design
The vLLM adaptor mirrors the Moonshot dual-format pattern but uses standard OpenAI-compatible paths:
RelayFormatClaude{baseURL}/v1/messagesRelayFormatOpenAI(chat){baseURL}/v1/chat/completions{baseURL}/v1/embeddings{baseURL}/v1/rerankConvertClaudeRequestdelegates toclaude.Adaptor(passthrough, no conversion).DoResponsedelegates toclaude.Adaptororopenai.Adaptorbased on relay format. No model list is shipped — vLLM serves whatever models are loaded at runtime; users configure model names via channel model mapping.Files changed
relay/channel/vllm/adaptor.go— new adaptorrelay/channel/vllm/constants.go— channel name, empty model listconstant/channel.go—ChannelTypeVLLM = 59, base URL entry, name entryconstant/api_type.go—APITypeVLLMcommon/api_type.go—ChannelTypeVLLM → APITypeVLLMmappingrelay/relay_adaptor.go— registerAPITypeVLLM → &vllm.Adaptor{}Summary by CodeRabbit