feat: claude count token - #2384
Conversation
fix: handle JSON parsing for thinking content in ollama stream
豆包语音2.0音色支持情感,情绪,音量
Comment out the debug log for MiniMax TTS Request.
增加MiniMax语音合成TTS支持
…options Ali channel support stream options
feat: openai tts support streaming realtime audio
feat: doubao tts support streaming realtime audio
multipart/form-data; boundary
feat(gemini): implement markdown image handling in text processing
chore: update openapi files
chore: update the relay openapi file
- Introduced new OpenAI text models in `common/model.go`. - Added `IsOpenAITextModel` function to check for OpenAI text models. - Refactored token estimation methods across various channels to use estimated prompt tokens instead of direct prompt token counts. - Updated related functions and structures to accommodate the new token estimation approach, enhancing overall token management.
fix: try resolve the high concurrency issue to a single host
feat: refactor token estimation logic
feat: gemini 3 thinking level gemini-3-pro-preview-high
WalkthroughAdds a Claude "count tokens" relay mode: new RelayFormat and HTTP route, validation and relay-info wiring, adaptor support for /v1/messages/count_tokens, controller changes to bypass quota/pricing and disable channel autoban for the count-tokens path while preserving error/reporting behavior. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Client
participant HTTP_Router as Router
participant Controller
participant Distributor
participant Channel_Adaptor as Adaptor
participant Claude_Service as Claude
participant Quota_System as Quota
Client->>Router: POST /v1/messages/count_tokens
Router->>Controller: Relay(request, RelayFormatClaudeCountTokens)
Controller->>Distributor: validate/select channel (path = count_tokens)
Distributor-->>Controller: allow missing model (skip validation)
Controller->>Adaptor: select channel (skip_channel_autoban=true)
Adaptor-->>Controller: Channel (AutoBan=0)
Controller->>Claude: POST /v1/messages/count_tokens (non-stream)
Claude-->>Controller: token-count response
Controller--xQuota: skip pre-consume (no quota consumed)
Controller->>Client: return token-count response
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
relay/channel/claude/adaptor.go (1)
142-142: Unreachable code:returnstatement after complete if-else branches.The
returnon line 142 is unreachable since both theif(line 137-138) andelse(line 139-141) branches return values.} else { return ClaudeHandler(c, resp, info, a.RequestMode) } - return }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
controller/relay.go(7 hunks)relay/channel/claude/adaptor.go(4 hunks)relay/claude_handler.go(1 hunks)relay/common/relay_info.go(1 hunks)relay/helper/valid_request.go(1 hunks)router/relay-router.go(1 hunks)types/relay_format.go(1 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: 9Ninety
Repo: QuantumNous/new-api PR: 1273
File: relay/channel/gemini/relay-gemini.go:97-116
Timestamp: 2025-06-21T03:37:41.726Z
Learning: In relay/channel/gemini/relay-gemini.go, the thinking budget calculation logic (including the MaxOutputTokens multiplication) was introduced in PR #1247. PR #1273 focused specifically on decoupling the thoughts summary feature from thinking budget settings and did not modify the existing thinking budget behavior.
📚 Learning: 2025-06-21T03:37:41.726Z
Learnt from: 9Ninety
Repo: QuantumNous/new-api PR: 1273
File: relay/channel/gemini/relay-gemini.go:97-116
Timestamp: 2025-06-21T03:37:41.726Z
Learning: In relay/channel/gemini/relay-gemini.go, the thinking budget calculation logic (including the MaxOutputTokens multiplication) was introduced in PR #1247. PR #1273 focused specifically on decoupling the thoughts summary feature from thinking budget settings and did not modify the existing thinking budget behavior.
Applied to files:
controller/relay.go
📚 Learning: 2025-08-05T17:14:17.246Z
Learnt from: neotf
Repo: QuantumNous/new-api PR: 1511
File: setting/ratio_setting/model_ratio.go:118-123
Timestamp: 2025-08-05T17:14:17.246Z
Learning: Claude models handle "-thinking" variants differently from Gemini models. For Claude models, only the base model (without "-thinking") gets an entry in defaultModelRatio map. The "-thinking" variants rely on the Claude relay handler stripping the suffix using strings.TrimSuffix(textRequest.Model, "-thinking") before looking up the ratio, so they automatically use the base model's ratio.
Applied to files:
controller/relay.gorelay/claude_handler.go
🧬 Code graph analysis (5)
relay/helper/valid_request.go (1)
types/relay_format.go (1)
RelayFormatClaudeCountTokens(8-8)
relay/common/relay_info.go (1)
types/relay_format.go (1)
RelayFormatClaudeCountTokens(8-8)
relay/channel/claude/adaptor.go (3)
types/relay_format.go (2)
RelayFormat(3-3)RelayFormatClaudeCountTokens(8-8)service/http.go (2)
CloseResponseBodyGracefully(15-23)IOCopyBytesGracefully(25-60)types/error.go (1)
ErrorCodeBadResponseBody(72-72)
router/relay-router.go (2)
controller/relay.go (1)
Relay(64-211)types/relay_format.go (1)
RelayFormatClaudeCountTokens(8-8)
relay/claude_handler.go (1)
types/relay_format.go (2)
RelayFormat(3-3)RelayFormatClaudeCountTokens(8-8)
🔇 Additional comments (15)
relay/claude_handler.go (1)
164-166: LGTM! Early return correctly bypasses quota consumption for count_tokens.The implementation correctly allows
DoResponseto handle the response (which in the adaptor copies the response body to the client) before returning early to skipPostClaudeConsumeQuota. This is the expected behavior for a token counting endpoint that shouldn't consume quota.router/relay-router.go (1)
80-82: LGTM! Route correctly placed before the more general/messagesendpoint.The route ordering ensures
/messages/count_tokensis matched before the catch-all/messagesroute. The route correctly delegates tocontroller.Relaywith theClaudeCountTokensformat.types/relay_format.go (1)
6-15: LGTM! New constant follows existing conventions.The
RelayFormatClaudeCountTokensconstant is properly defined and follows the established naming pattern. The explicit type annotation on the first constant is idiomatic Go.relay/channel/claude/adaptor.go (3)
128-136: LGTM! CountTokens response handling correctly streams raw response to client.The implementation properly reads the upstream response body and copies it to the client without processing, which is appropriate for a token count response. The graceful close and copy helpers ensure proper resource cleanup.
50-53: LGTM! Early return correctly prevents model-based mode selection for CountTokens.The guard clause ensures
RequestModeCountTokensis set before the model prefix checks, preventing incorrect mode assignment.
63-70: LGTM! URL routing correctly maps CountTokens to the appropriate endpoint.The switch statement cleanly routes each mode to its corresponding API path.
controller/relay.go (7)
69-69: LGTM! Clear boolean flag improves readability.Using
isClaudeCountTokensas a named flag makes the subsequent conditional checks more readable than repeating the format comparison.
118-122: LGTM! Disabling auto-ban for count_tokens prevents channel penalties for utility requests.This correctly prevents channels from being penalized or banned due to failures on the count_tokens endpoint, which is a utility operation that shouldn't affect channel reliability metrics.
146-170: LGTM! Quota and pricing logic correctly bypassed for count_tokens.The entire pre-consume and refund logic is wrapped in the
!isClaudeCountTokenscheck, ensuring token counting requests don't affect user quotas. The defer block for quota refund is also correctly scoped within this conditional.
187-188: LGTM! ClaudeCountTokens correctly routes to ClaudeHelper.Both
RelayFormatClaudeandRelayFormatClaudeCountTokensare handled byClaudeHelper, which internally differentiates behavior based on the format.
227-258: LGTM! Channel selection respects skip_channel_autoban flag.The
getChannelfunction correctly handles theskip_channel_autobanflag for both initial channel selection (lines 234-236) and retry scenarios (lines 255-258), ensuring channels aren't banned due to count_tokens operations.
93-93: LGTM! Error response format correctly includes ClaudeCountTokens.Both
RelayFormatClaudeandRelayFormatClaudeCountTokensreturn Claude-style error responses, maintaining API consistency.
135-142: Verify token estimation is intentionally skipped for count_tokens.Setting
tokens = 0and skippingEstimateRequestTokenmeansrelayInfo.SetEstimatePromptTokens(0)will be called. Confirm this is intentional and identify which downstream logic depends on estimated tokens being non-zero for this code path.relay/common/relay_info.go (1)
451-452: Potential bug:RelayFormatwill be overwritten toRelayFormatClaude.
GenRelayInfoClaudesetsinfo.RelayFormat = types.RelayFormatClaudeon line 288, which will overwrite theClaudeCountTokensformat. This could cause issues downstream where the code checks forRelayFormatClaudeCountTokens.Consider either:
- Setting
info.RelayFormat = relayFormatafter callingGenRelayInfoClaude, or- Creating a dedicated
GenRelayInfoClaudeCountTokensfunction, or- Passing the format as a parameter to
GenRelayInfoClaude.relay/helper/valid_request.go (1)
35-36: The count_tokens validation is correct.
GetAndValidateClaudeRequestappropriately validates the requiredMessagesandModelfields—both are mandatory for Claude's/v1/messages/count_tokensAPI endpoint.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
relay/common/relay_info.go (1)
286-305: Shared Claude base helper looks correct; consider documenting its dual useRefactoring
GenRelayInfoClaudethroughgenRelayInfoClaudeBasekeeps the previous behavior (same flags, same beta handling) while cleanly enablingGenRelayInfoClaudeCountTokensvia a differentRelayFormat, which is good.Since this helper now underpins both
/v1/messagesand/v1/messages/count_tokensflows, consider a brief comment to make that contract explicit so future changes (e.g., pricing/usage behavior) don’t accidentally diverge:-func genRelayInfoClaudeBase(c *gin.Context, request dto.Request, relayFormat types.RelayFormat) *RelayInfo { +// genRelayInfoClaudeBase sets up common RelayInfo for Claude-style requests. +// It is used by both /v1/messages and /v1/messages/count_tokens paths. +func genRelayInfoClaudeBase(c *gin.Context, request dto.Request, relayFormat types.RelayFormat) *RelayInfo {
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
relay/common/relay_info.go(2 hunks)
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: 9Ninety
Repo: QuantumNous/new-api PR: 1273
File: relay/channel/gemini/relay-gemini.go:97-116
Timestamp: 2025-06-21T03:37:41.726Z
Learning: In relay/channel/gemini/relay-gemini.go, the thinking budget calculation logic (including the MaxOutputTokens multiplication) was introduced in PR #1247. PR #1273 focused specifically on decoupling the thoughts summary feature from thinking budget settings and did not modify the existing thinking budget behavior.
🧬 Code graph analysis (1)
relay/common/relay_info.go (2)
types/relay_format.go (3)
RelayFormatClaude(7-7)RelayFormatClaudeCountTokens(8-8)RelayFormat(3-3)dto/request_common.go (1)
Request(8-12)
🔇 Additional comments (1)
relay/common/relay_info.go (1)
459-460: GenRelayInfo wiring for ClaudeCountTokens is consistentThe new
RelayFormatClaudeCountTokensbranch cleanly delegates toGenRelayInfoClaudeCountTokensand matches the pattern of other simple formats; no issues noted here.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
middleware/distributor.go(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: 9Ninety
Repo: QuantumNous/new-api PR: 1273
File: relay/channel/gemini/relay-gemini.go:97-116
Timestamp: 2025-06-21T03:37:41.726Z
Learning: In relay/channel/gemini/relay-gemini.go, the thinking budget calculation logic (including the MaxOutputTokens multiplication) was introduced in PR #1247. PR #1273 focused specifically on decoupling the thoughts summary feature from thinking budget settings and did not modify the existing thinking budget behavior.
🧬 Code graph analysis (1)
middleware/distributor.go (2)
dto/request_common.go (1)
Request(8-12)model/model_meta.go (1)
Model(23-44)
🔇 Additional comments (1)
middleware/distributor.go (1)
79-82: Model validation bypass is correctly implemented.The conditional logic properly skips model name validation for count_tokens requests while preserving it for all other endpoints. The change aligns with the feature requirements described in the AI summary.
| // count_tokens 接口跳过模型名称校验 | ||
| isCountTokens := strings.HasPrefix(c.Request.URL.Path, "/v1/messages/count_tokens") |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Search for CacheGetRandomSatisfiedChannel implementation and its handling of model parameter
rg -n "CacheGetRandomSatisfiedChannel" --type=go -B2 -A10 | head -100Repository: QuantumNous/new-api
Length of output: 3835
🏁 Script executed:
# Check if count_tokens is handled elsewhere with shouldSelectChannel logic
rg -n "count_tokens" --type=go -B3 -A3Repository: QuantumNous/new-api
Length of output: 3266
🏁 Script executed:
# Look for how shouldSelectChannel is set to false in the file
rg -n "shouldSelectChannel\s*=" middleware/distributor.go -B2 -A2Repository: QuantumNous/new-api
Length of output: 1093
🏁 Script executed:
# Check the full context around line 102 where CacheGetRandomSatisfiedChannel is called
sed -n '95,110p' middleware/distributor.goRepository: QuantumNous/new-api
Length of output: 705
🏁 Script executed:
# Get full implementation of CacheGetRandomSatisfiedChannel
sed -n '14,80p' service/channel_select.goRepository: QuantumNous/new-api
Length of output: 1018
🏁 Script executed:
# Check what happens with modelName inside the function - look for where it's used
rg -A 50 "func CacheGetRandomSatisfiedChannel" service/channel_select.go | head -70Repository: QuantumNous/new-api
Length of output: 1018
🏁 Script executed:
# Search for GetRandomSatisfiedChannel implementation in model package
rg -n "func GetRandomSatisfiedChannel" --type=go -A 30Repository: QuantumNous/new-api
Length of output: 1832
🏁 Script executed:
# Also check if there's any special handling for count_tokens in controller/relay.go
sed -n '110,130p' controller/relay.goRepository: QuantumNous/new-api
Length of output: 751
🏁 Script executed:
# Check the full context around line 102-125 to see how nil channel is handled
sed -n '95,125p' middleware/distributor.goRepository: QuantumNous/new-api
Length of output: 1322
🏁 Script executed:
# Check if count_tokens requests even need a valid channel selected
rg -n "isCountTokens\|RelayFormatClaudeCountTokens" controller/relay.go -B5 -A10 | head -80Repository: QuantumNous/new-api
Length of output: 45
🏁 Script executed:
# Double-check by searching for how count_tokens is routed - does it even reach distributor middleware?
rg -n "count_tokens" router/relay-router.go -B3 -A5Repository: QuantumNous/new-api
Length of output: 397
🏁 Script executed:
# Check middleware chain to understand if count_tokens goes through Distribute() middleware
sed -n '70,90p' router/relay-router.goRepository: QuantumNous/new-api
Length of output: 680
Set shouldSelectChannel = false for count_tokens requests to avoid unnecessary channel selection failures.
The current logic skips model validation when isCountTokens is true (line 79), but then requires a valid channel at line 114, causing count_tokens requests with empty models to fail with a 503 "no available channel" error. Since count_tokens should bypass normal channel selection like other similar operations (Midjourney fetch, Suno fetch, Video fetch), add shouldSelectChannel = false when isCountTokens is true, similar to the pattern used at lines 153, 169, 179, etc.
🤖 Prompt for AI Agents
In middleware/distributor.go around lines 76-77, the isCountTokens flag is set
but the code still proceeds to channel selection causing 503 errors when model
is empty; update the logic so that when isCountTokens is true you also set
shouldSelectChannel = false (same pattern used for other bypass paths like
Midjourney/Suno/Video fetch) so count_tokens requests skip channel selection and
won't fail due to missing channel.
Implements Anthropic's POST /v1/messages/count_tokens endpoint (https://docs.claude.com/en/api/messages-count-tokens) by estimating input_tokens locally instead of forwarding to any upstream channel. Why --- The Anthropic JS SDK and Claude CLI poll this endpoint before each chat to size the context window. When new-api returns 404 (which it does today, since the route is unregistered), the SDK falls back to sending fake max_tokens=1 messages with the entire tool schema attached. We observed bursts of 250 RPM of these probes from a single Claude Code Desktop session, exhausting the upstream provider's RPM quota and starving real traffic on the same channel for ~2 minutes. Returning a fast 200 here from new-api itself fixes the failure mode at the source: the SDK is satisfied, no upstream RPM is consumed, no billing entry is created. Closes QuantumNous#1694 Closes QuantumNous#2847 Closes QuantumNous#1979 Approach -------- - Mounted on relayV1Router directly, NOT under httpRouter — the former gives us TokenAuth + RouteTag + SystemPerformanceCheck + ModelRequestRateLimit; the latter would also pull in middleware.Distribute() which selects a channel and starts the PreConsume flow. Anthropic defines count_tokens as token-counting only with no quota impact, so the bypass is intentional. - Estimation reuses the project's existing pieces: * ClaudeRequest.GetTokenCountMeta() for the canonical flattening of system / messages (text, tool_use, tool_result) / tools * EstimateTokenByModel() for the Claude-tuned tokenizer that /v1/messages itself bills against This keeps the count consistent with what /v1/messages would report on the same body, so callers can reason about both numbers together. - One small helper, normalizeRequestTools(), converts raw map[string]any tool entries (what json.Unmarshal produces when the field is `any`) into the typed *dto.Tool / *dto.ClaudeWebSearchTool values that dto.ProcessTools requires. Without this every tool entry on a count_tokens request would be silently dropped on the `default: continue` arm of ProcessTools (dto/claude.go:439-442) and the tools text — usually 80%+ of a CLI probe body — would not be counted. - Body parsing uses common.UnmarshalBodyReusable so the request body remains available to any logging/observability that runs after this handler. - Image tokens are intentionally not added: getImageToken() needs a RelayInfo + http.Request context, which this route doesn't have. The CLI probe (the failure mode this PR exists to mitigate) never carries images. Documented in the estimator's godoc. How vs. PR QuantumNous#2384 ---------------- QuantumNous#2384 took the deeper path: a new RelayFormat enum value + adaptor changes + middleware.distributor changes + autoban bypass. That PR was closed without merge. This PR keeps the surface area small — one new route, one controller, one estimator, no enum/adaptor changes, no impact on any code path that doesn't hit the new route. Files ----- - router/relay-router.go + 9 lines (one route registration) - controller/claude_count_tokens.go new (~35 lines incl. doc) - service/claude_token_estimator.go new (~90 lines) - service/claude_token_estimator_test.go new (12 cases) Tests ----- $ go test ./service/ -run 'TestEstimateClaude|TestNormalizeRequestTools' -v === RUN TestEstimateClaudeInputTokens --- PASS (10 cases: empty / short user / system+user / CLI probe / system as array / CJK / tool_use / tool_result / web-search / pre-typed tools) === RUN TestEstimateClaudeInputTokens_NilSafety --- PASS (2 cases: nil request / zero-value request) === RUN TestNormalizeRequestTools --- PASS (3 cases: nil / non-array / malformed dropped) Manual verification of the failure mode (sample CLI probe body — trivial message + Bash tool schema, ~200 bytes): EstimateClaudeInputTokens = 43 input tokens Endpoint returns: 200 OK {"input_tokens": 43} Claude CLI accepts and stops sending fake max_tokens=1 fallback.
Implements Anthropic's POST /v1/messages/count_tokens endpoint (https://docs.claude.com/en/api/messages-count-tokens) by estimating input_tokens locally instead of forwarding to any upstream channel. Why --- The Anthropic JS SDK and Claude CLI poll this endpoint before each chat to size the context window. When new-api returns 404 (which it does today, since the route is unregistered), the SDK falls back to sending fake max_tokens=1 messages with the entire tool schema attached. We observed bursts of 250 RPM of these probes from a single Claude Code Desktop session, exhausting the upstream provider's RPM quota and starving real traffic on the same channel for ~2 minutes. Returning a fast 200 here from new-api itself fixes the failure mode at the source: the SDK is satisfied, no upstream RPM is consumed, no billing entry is created. Closes QuantumNous#1694 Closes QuantumNous#2847 Closes QuantumNous#1979 Approach -------- - Mounted on relayV1Router directly, NOT under httpRouter — the former gives us TokenAuth + RouteTag + SystemPerformanceCheck + ModelRequestRateLimit; the latter would also pull in middleware.Distribute() which selects a channel and starts the PreConsume flow. Anthropic defines count_tokens as token-counting only with no quota impact, so the bypass is intentional. - Estimation reuses the project's existing pieces: * ClaudeRequest.GetTokenCountMeta() for the canonical flattening of system / messages (text, tool_use, tool_result) / tools * EstimateTokenByModel() for the Claude-tuned tokenizer that /v1/messages itself bills against This keeps the count consistent with what /v1/messages would report on the same body, so callers can reason about both numbers together. - One small helper, normalizeRequestTools(), converts raw map[string]any tool entries (what json.Unmarshal produces when the field is `any`) into the typed *dto.Tool / *dto.ClaudeWebSearchTool values that dto.ProcessTools requires. Without this every tool entry on a count_tokens request would be silently dropped on the `default: continue` arm of ProcessTools (dto/claude.go:439-442) and the tools text — usually 80%+ of a CLI probe body — would not be counted. - Body parsing uses common.UnmarshalBodyReusable so the request body remains available to any logging/observability that runs after this handler. - Image tokens are intentionally not added: getImageToken() needs a RelayInfo + http.Request context, which this route doesn't have. The CLI probe (the failure mode this PR exists to mitigate) never carries images. Documented in the estimator's godoc. How vs. PR QuantumNous#2384 ---------------- QuantumNous#2384 took the deeper path: a new RelayFormat enum value + adaptor changes + middleware.distributor changes + autoban bypass. That PR was closed without merge. This PR keeps the surface area small — one new route, one controller, one estimator, no enum/adaptor changes, no impact on any code path that doesn't hit the new route. Files ----- - router/relay-router.go + 9 lines (one route registration) - controller/claude_count_tokens.go new (~35 lines incl. doc) - service/claude_token_estimator.go new (~90 lines) - service/claude_token_estimator_test.go new (12 cases) Tests ----- $ go test ./service/ -run 'TestEstimateClaude|TestNormalizeRequestTools' -v === RUN TestEstimateClaudeInputTokens --- PASS (10 cases: empty / short user / system+user / CLI probe / system as array / CJK / tool_use / tool_result / web-search / pre-typed tools) === RUN TestEstimateClaudeInputTokens_NilSafety --- PASS (2 cases: nil request / zero-value request) === RUN TestNormalizeRequestTools --- PASS (3 cases: nil / non-array / malformed dropped) Manual verification of the failure mode (sample CLI probe body — trivial message + Bash tool schema, ~200 bytes): EstimateClaudeInputTokens = 43 input tokens Endpoint returns: 200 OK {"input_tokens": 43} Claude CLI accepts and stops sending fake max_tokens=1 fallback.
Implements Anthropic's POST /v1/messages/count_tokens endpoint (https://docs.claude.com/en/api/messages-count-tokens) by estimating input_tokens locally instead of forwarding to any upstream channel. Why --- The Anthropic JS SDK and Claude CLI poll this endpoint before each chat to size the context window. When new-api returns 404 (which it does today, since the route is unregistered), the SDK falls back to sending fake max_tokens=1 messages with the entire tool schema attached. We observed bursts of 250 RPM of these probes from a single Claude Code Desktop session, exhausting the upstream provider's RPM quota and starving real traffic on the same channel for ~2 minutes. Returning a fast 200 here from new-api itself fixes the failure mode at the source: the SDK is satisfied, no upstream RPM is consumed, no billing entry is created. Closes QuantumNous#1694 Closes QuantumNous#2847 Closes QuantumNous#1979 Approach -------- - Mounted on relayV1Router directly, NOT under httpRouter — the former gives us TokenAuth + RouteTag + SystemPerformanceCheck + ModelRequestRateLimit; the latter would also pull in middleware.Distribute() which selects a channel and starts the PreConsume flow. Anthropic defines count_tokens as token-counting only with no quota impact, so the bypass is intentional. - Estimation reuses the project's existing pieces: * ClaudeRequest.GetTokenCountMeta() for the canonical flattening of system / messages (text, tool_use, tool_result) / tools * EstimateTokenByModel() for the Claude-tuned tokenizer that /v1/messages itself bills against This keeps the count consistent with what /v1/messages would report on the same body, so callers can reason about both numbers together. - One small helper, normalizeRequestTools(), converts raw map[string]any tool entries (what json.Unmarshal produces when the field is `any`) into the typed *dto.Tool / *dto.ClaudeWebSearchTool values that dto.ProcessTools requires. Without this every tool entry on a count_tokens request would be silently dropped on the `default: continue` arm of ProcessTools (dto/claude.go:439-442) and the tools text — usually 80%+ of a CLI probe body — would not be counted. - Body parsing uses common.UnmarshalBodyReusable so the request body remains available to any logging/observability that runs after this handler. - Image tokens are intentionally not added: getImageToken() needs a RelayInfo + http.Request context, which this route doesn't have. The CLI probe (the failure mode this PR exists to mitigate) never carries images. Documented in the estimator's godoc. How vs. PR QuantumNous#2384 ---------------- QuantumNous#2384 took the deeper path: a new RelayFormat enum value + adaptor changes + middleware.distributor changes + autoban bypass. That PR was closed without merge. This PR keeps the surface area small — one new route, one controller, one estimator, no enum/adaptor changes, no impact on any code path that doesn't hit the new route. Files ----- - router/relay-router.go + 9 lines (one route registration) - controller/claude_count_tokens.go new (~35 lines incl. doc) - service/claude_token_estimator.go new (~90 lines) - service/claude_token_estimator_test.go new (12 cases) Tests ----- $ go test ./service/ -run 'TestEstimateClaude|TestNormalizeRequestTools' -v === RUN TestEstimateClaudeInputTokens --- PASS (10 cases: empty / short user / system+user / CLI probe / system as array / CJK / tool_use / tool_result / web-search / pre-typed tools) === RUN TestEstimateClaudeInputTokens_NilSafety --- PASS (2 cases: nil request / zero-value request) === RUN TestNormalizeRequestTools --- PASS (3 cases: nil / non-array / malformed dropped) Manual verification of the failure mode (sample CLI probe body — trivial message + Bash tool schema, ~200 bytes): EstimateClaudeInputTokens = 43 input tokens Endpoint returns: 200 OK {"input_tokens": 43} Claude CLI accepts and stops sending fake max_tokens=1 fallback.
20 participants
Summary by CodeRabbit
New Features
Behavior Changes
Bug Fixes
✏️ Tip: You can customize this high-level summary in your review settings.