Skip to content

feat(providers): add CoralBricks as a native provider - #44700

Merged
mateo-berri merged 16 commits into
mainfrom
litellm_coralbricks_provider
Oct 12, 2026
Merged

feat(providers): add CoralBricks as a native provider#44700
mateo-berri merged 16 commits into
mainfrom
litellm_coralbricks_provider

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 5, 2026 •
edited by mateo-berri
Loading

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • coralbricks/<model> is not a known provider, so every call 400s
  • No CoralBricks pricing, so spend and cache-write cost cannot be tracked
  • No CoralBricks option in the Admin UI Add Model flow

How it solves it:

  • Registers coralbricks as a JSON OpenAI-compatible provider at https://inference.coralbricks.ai/v1
  • Declares native /v1/chat/completions, /v1/responses and /v1/messages, so nothing is translated
  • Adds cost-map rows for glm-5.3-fast and deepseek-v4.1-flash-fast with cache-write pricing and free cached reads, rates from the live catalog (GET https://www.coralbricks.ai/api/public/models, read 2026-10-12; the pricing page at https://www.coralbricks.ai/pricing renders client side)
  • Keeps rows for the deprecated glm-5.3, glm-5.3-fp4, deepseek-v4.1-flash, and deepseek-v4.1-flash-fast-fp4 slugs CoralBricks still serves, at the same rates, so a call on an old name is billed rather than logged at $0
  • Drops the issue's glm-5.3-flash-fp4, which CoralBricks retired on 2026-10-07 (every call answers 404 model_retired)
  • Lists those rows in the provider model sets, so get_valid_models knows them and a coralbricks/* wildcard deployment expands to them on /v1/models
  • Adds the Add Model provider entry (optional API base, required API key) and the dashboard provider mapping so its models show in the picker

User Flow

Before: a proxy admin who configures a CoralBricks model gets a 400 on every endpoint and cannot list or pick its models

  1. Admin adds model: coralbricks/glm-5.3-fast with CORALBRICKS_API_KEY to the proxy config
  2. Developer sends POST http://localhost:4000/v1/chat/completions (or /v1/responses, /v1/messages) with that model
  3. Proxy returns 400 There are no healthy deployments for this model
  4. Admin sends POST http://localhost:4000/model/new with coralbricks/*; it answers 500 not live in this pod's router after the reload, and GET http://localhost:4000/v1/models lists no coralbricks/ model
  5. On http://localhost:3000/models-and-endpoints, Add Model, typing Coral in Provider shows No providers found

Before: Add Model provider search for Coral finds nothing

After: the same calls go straight to CoralBricks, are billed at its rates, and the catalog shows up everywhere

  1. Admin adds model: coralbricks/glm-5.3-fast with CORALBRICKS_API_KEY to the proxy config
  2. Developer sends POST http://localhost:4000/v1/chat/completions (or /v1/responses, /v1/messages) with that model
  3. Proxy returns 200 with CoralBricks' answer, x-litellm-response-cost matches the cost CoralBricks reports, and a repeated prompt comes back with cached_tokens that add $0 to spend
  4. The same POST http://localhost:4000/model/new with coralbricks/* answers 200 with a deployment id, and GET http://localhost:4000/v1/models lists every CoralBricks catalog model
  5. On http://localhost:3000/models-and-endpoints, Add Model, typing Coral in Provider offers CoralBricks, which lists its models and shows https://inference.coralbricks.ai/v1 as the API base placeholder (left empty, the provider's base applies)

After: Add Model provider search lists CoralBricks

Relevant issues

Fixes #44678

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

/qa at d5bbab5 against 29f48e6, the merge base before main was merged in as 93c5048

Last updated: d5bbab5

Two proxies, one per leg, each booted with --num_workers 2 from its own worktree on a random port with its own Postgres database, STORE_MODEL_IN_DB=True, LITELLM_LOCAL_MODEL_COST_MAP=True, the same config (coralbricks/glm-5.3-fast, coralbricks/deepseek-v4.1-flash-fast, and the deprecated coralbricks/glm-5.3-fp4, each with api_key: os.environ/CORALBRICKS_API_KEY, plus an openai/gpt-5.4-mini control, drop_params: true), and the real CoralBricks key. The same script ran against both; $KEY is the proxy master key. Run on 2026-10-12, the before leg at 17:50Z and the after leg at 18:56Z on the staged tree 113351dece, every product file identical to d5bbab5 (committed at 19:27Z)

Before (29f48e6): the router drops every CoralBricks deployment at boot, so each call is a 400 and the wildcard create is a 500

Proxy log at boot, once per deployment and worker:

LiteLLM Router:ERROR: router.py:9371 - Error creating deployment: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=coralbricks/deepseek-v4.1-flash-fast
LiteLLM Router:ERROR: router.py:9371 - Error creating deployment: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=coralbricks/glm-5.3-fast
LiteLLM Router:ERROR: router.py:9371 - Error creating deployment: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=coralbricks/glm-5.3-fp4

### 1. chat completions, glm-5.3-fast, non-stream
$ curl -s -D - $B/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model":"coralbricks/glm-5.3-fast","messages":[{"role":"user","content":"Reply with exactly: coral ok"}],"max_tokens":300}'
HTTP/1.1 400 Bad Request
x-litellm-call-id: 2ed48b51-79a3-48a9-99ff-cdce936593d5
x-litellm-response-cost: 0
{"error":{"message":"litellm.BadRequestError: You passed in model=coralbricks/glm-5.3-fast. There are no healthy deployments for this model\n\nLiteLLM: model group 'coralbricks/glm-5.3-fast' failed with the error above. No fallback was attempted.","type":"invalid_request_error","param":null,"code":"400"}}

### 2. chat completions, deepseek-v4.1-flash-fast, stream with usage
$ curl -s -N $B/v1/chat/completions ... -d '{"model":"coralbricks/deepseek-v4.1-flash-fast","messages":[{"role":"user","content":"Count from 1 to 5, digits only"}],"stream":true,"stream_options":{"include_usage":true},"max_tokens":30}' | tail -n 4
{"error":{"message":"litellm.BadRequestError: You passed in model=coralbricks/deepseek-v4.1-flash-fast. There are no healthy deployments for this model\n\nLiteLLM: model group 'coralbricks/deepseek-v4.1-flash-fast' failed with the error above. No fallback was attempted.","type":"invalid_request_error","param":null,"code":"400"}}

### 3. prompt caching: the same 1,000-word deepseek chat twice, expect cached_tokens on the second
$ (first) curl -s -D - http://127.0.0.1:23090/v1/chat/completions ... -d '{"model":"coralbricks/deepseek-v4.1-flash-fast","messages":[{"role":"user","content":"<1,000-word prompt>"}],"max_tokens":40}'
HTTP/1.1 400 Bad Request
x-litellm-call-id: 6569c202-e0c2-4856-a7cc-b5c90bee2741
x-litellm-response-cost: 0
{"error":{"message":"litellm.BadRequestError: You passed in model=coralbricks/deepseek-v4.1-flash-fast. There are no healthy deployments for this model\n\nLiteLLM: model group 'coralbricks/deepseek-v4.1-flash-fast' failed with the error above. No fallback was attempted.","type":"invalid_request_error","param":null,"code":"400"}}
$ (second) curl -s -D - http://127.0.0.1:23090/v1/chat/completions ... -d '{"model":"coralbricks/deepseek-v4.1-flash-fast","messages":[{"role":"user","content":"<1,000-word prompt>"}],"max_tokens":40}'
HTTP/1.1 400 Bad Request
x-litellm-call-id: 72ca5058-a1f4-4822-b9a7-d0389b103e2b
x-litellm-response-cost: 0
{"error":{"message":"litellm.BadRequestError: You passed in model=coralbricks/deepseek-v4.1-flash-fast. There are no healthy deployments for this model\n\nLiteLLM: model group 'coralbricks/deepseek-v4.1-flash-fast' failed with the error above. No fallback was attempted.","type":"invalid_request_error","param":null,"code":"400"}}

### 4. responses API, glm-5.3-fast
$ curl -s -D - $B/v1/responses ... -d '{"model":"coralbricks/glm-5.3-fast","input":"Reply with exactly: coral ok","max_output_tokens":300}'
HTTP/1.1 400 Bad Request
x-litellm-call-id: 76476c4a-41a8-4c5b-932e-a2352bdecd85
x-litellm-response-cost: 0
{"error":{"message":"litellm.BadRequestError: You passed in model=coralbricks/glm-5.3-fast. There are no healthy deployments for this model\n\nLiteLLM: model group 'coralbricks/glm-5.3-fast' failed with the error above. No fallback was attempted.","type":"invalid_request_error","param":null,"code":"400"}}

### 5. messages API, deepseek-v4.1-flash-fast
$ curl -s -D - $B/v1/messages -H "anthropic-version: 2023-06-01" ... -d '{"model":"coralbricks/deepseek-v4.1-flash-fast","max_tokens":50,"messages":[{"role":"user","content":"Reply with exactly: coral ok"}]}'
HTTP/1.1 400 Bad Request
x-litellm-call-id: 89e9e698-75ee-4f76-b4a8-52e1d26f6b1b
x-litellm-response-cost: 0
{"type":"error","error":{"type":"invalid_request_error","message":"litellm.BadRequestError: You passed in model=coralbricks/deepseek-v4.1-flash-fast. There are no healthy deployments for this model\n\nLiteLLM: model group 'coralbricks/deepseek-v4.1-flash-fast' failed with the error above. No fallback was attempted."}}

### 6. deprecated slug glm-5.3-fp4, chat, non-stream
$ curl -s -D - $B/v1/chat/completions ... -d '{"model":"coralbricks/glm-5.3-fp4","messages":[{"role":"user","content":"Reply with exactly: coral ok"}],"max_tokens":300}'
HTTP/1.1 400 Bad Request
x-litellm-call-id: 80346a19-f3a7-4693-a068-bbf20ea18dde
x-litellm-response-cost: 0
{"error":{"message":"litellm.BadRequestError: You passed in model=coralbricks/glm-5.3-fp4. There are no healthy deployments for this model\n\nLiteLLM: model group 'coralbricks/glm-5.3-fp4' failed with the error above. No fallback was attempted.","type":"invalid_request_error","param":null,"code":"400"}}

### 7. control: openai/gpt-5.4-mini chat
$ curl -s -D - $B/v1/chat/completions ... -d '{"model":"openai/gpt-5.4-mini","messages":[{"role":"user","content":"Reply with exactly: control ok"}],"max_tokens":20}'
HTTP/1.1 200 OK
x-litellm-call-id: aaaebd8a-4f4a-4185-83c9-736288699e41
x-litellm-response-cost: 3.15e-05
x-litellm-response-cost-original: 3.15e-05
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 9e-06
x-litellm-response-cost-output: 2.25e-05
x-litellm-response-cost-tool-usage: 0.0
{"id":"chatcmpl-EYERT4yPR9m9HZTzmDL3alBrdWjdU","model":"openai/gpt-5.4-mini","content":"control ok"}

### 8. model info rates for the CoralBricks deployments
$ curl -s $B/v1/model/info -H "Authorization: Bearer $KEY" | jq -c '.data[] | select(.model_name | startswith("coralbricks/")) | {model_name, litellm_provider: .model_info.litellm_provider, input: .model_info.input_cost_per_token, output: .model_info.output_cost_per_token, cache_write: .model_info.cache_creation_input_token_cost, cache_read: .model_info.cache_read_input_token_cost}'
(rows printed: 0)

### 9. spend log row for case 1
(case 1 returned no request id, so there is no spend row to read)

### 10. wildcard deployment coralbricks/* lists the catalog on /v1/models
$ curl -s -X POST $B/model/new -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model_name":"coralbricks/*","litellm_params":{"model":"coralbricks/*","api_key":"os.environ/CORALBRICKS_API_KEY"}}' | jq -c '{model_name, id: .model_info.id} // .'
{"error":{"message":"Model create was saved to the database, but the model id(s) ['5c960339-79aa-4601-b250-a508b8af30e8'] are not live in this pod's router after the reload and are not being served by this pod. Other pods reload on their own interval. Check server logs for 'Error upserting deployment' or 'Error creating deployment' for the cause.","type":"internal_server_error","param":null,"code":"500"}}
$ curl -s $B/v1/models -H "Authorization: Bearer $KEY" | jq -c '[.data[].id | select(startswith("coralbricks/"))] | sort'
[]

On http://localhost:3000/models-and-endpoints, Add Model, typing Coral into Provider:

Before: Add Model provider search for Coral finds nothing

After (d5bbab5): chat, responses, and messages answer 200 on both models, billed at the cost-map rates, coralbricks/* lists the catalog, and the deprecated slugs bill at their current names' rates

### 1. chat completions, glm-5.3-fast, non-stream
$ curl -s -D - $B/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model":"coralbricks/glm-5.3-fast","messages":[{"role":"user","content":"Reply with exactly: coral ok"}],"max_tokens":300}'
HTTP/1.1 200 OK
x-litellm-call-id: 9b704ba8-9d3c-406c-835d-4a1f3db6628f
x-litellm-response-cost: 0.00015344000000000002
x-litellm-response-cost-original: 0.00015344000000000002
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.0
x-litellm-response-cost-output: 0.0001232
x-litellm-response-cost-cache-creation: 3.0240000000000002e-05
x-litellm-response-cost-reasoning: 0.0001056
x-litellm-response-cost-tool-usage: 0.0
{"id":"req_fbe4ee05b378465f9fd8c9dc1ed0aacb","model":"coralbricks/glm-5.3-fast","content":"coral ok","usage":{"completion_tokens":28,"prompt_tokens":18,"total_tokens":46,"completion_tokens_details":{"reasoning_tokens":24},"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":18,"cache_creation_tokens":18,"billable_cache_write_tokens":18,"cache_write_blocks":3},"cost":0.00015344,"cost_details":{"currency":"USD","input":0.0,"cached_input":0.0,"cache_write":0.00003024,"output":0.0001232}}}

### 2. chat completions, deepseek-v4.1-flash-fast, stream with usage
$ curl -s -N $B/v1/chat/completions ... -d '{"model":"coralbricks/deepseek-v4.1-flash-fast","messages":[{"role":"user","content":"Count from 1 to 5, digits only"}],"stream":true,"stream_options":{"include_usage":true},"max_tokens":30}' | tail -n 4
data: {"id":"req_e0ca0d9b1e954757bdb92dde2aa9ba96","object":"chat.completion.chunk","created":1791831401,"model":"coralbricks/deepseek-v4.1-flash-fast","choices":[{"index":0,"delta":{"content":"5"}}]}
data: {"id":"req_e0ca0d9b1e954757bdb92dde2aa9ba96","object":"chat.completion.chunk","created":1791831401,"model":"coralbricks/deepseek-v4.1-flash-fast","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: {"id":"req_e0ca0d9b1e954757bdb92dde2aa9ba96","created":1791831401,"model":"coralbricks/deepseek-v4.1-flash-fast","object":"chat.completion.chunk","choices":[{"index":0,"delta":{}}],"usage":{"completion_tokens":10,"prompt_tokens":14,"total_tokens":24,"completion_tokens_details":{"reasoning_tokens":0},"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":14,"cache_creation_tokens":14,"billable_cache_write_tokens":14,"cache_write_blocks":3},"cost":0.00001326}}
data: [DONE]

### 3. prompt caching: the same 1,000-word deepseek chat twice, expect cached_tokens on the second
$ (first) curl -s -D - http://127.0.0.1:38554/v1/chat/completions ... -d '{"model":"coralbricks/deepseek-v4.1-flash-fast","messages":[{"role":"user","content":"<1,000-word prompt>"}],"max_tokens":40}'
HTTP/1.1 200 OK
x-litellm-call-id: 9f7aa395-87d3-4237-9438-ddcf72d87fef
x-litellm-response-cost: 0.00019298999999999998
x-litellm-response-cost-original: 0.00019298999999999998
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.0
x-litellm-response-cost-output: 1.2e-05
x-litellm-response-cost-cache-creation: 0.00018099
x-litellm-response-cost-tool-usage: 0.0
{"id":"req_784994a0b3394d7e8ef07e73994a30b0","prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":2011,"cache_creation_tokens":2011,"billable_cache_write_tokens":2011,"cache_write_blocks":3},"vendor_cost":0.00019299,"vendor_cost_details":{"currency":"USD","input":0.0,"cached_input":0.0,"cache_write":0.00018099,"output":0.000012}}
$ (second) curl -s -D - http://127.0.0.1:38554/v1/chat/completions ... -d '{"model":"coralbricks/deepseek-v4.1-flash-fast","messages":[{"role":"user","content":"<1,000-word prompt>"}],"max_tokens":40}'
HTTP/1.1 200 OK
x-litellm-call-id: d48ed7b9-91c4-4914-b8e2-d4ab39ecf3fb
x-litellm-response-cost: 3.171e-05
x-litellm-response-cost-original: 3.171e-05
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.0
x-litellm-response-cost-output: 1.2e-05
x-litellm-response-cost-cache-creation: 1.971e-05
x-litellm-response-cost-tool-usage: 0.0
{"id":"req_7df8b26764384343bb31b3557f9b553d","prompt_tokens_details":{"cached_tokens":1792,"cache_write_tokens":219,"cache_creation_tokens":219,"billable_cache_write_tokens":219,"cache_write_blocks":3},"vendor_cost":0.00003171,"vendor_cost_details":{"currency":"USD","input":0.0,"cached_input":0.0,"cache_write":0.00001971,"output":0.000012}}

### 4. responses API, glm-5.3-fast
$ curl -s -D - $B/v1/responses ... -d '{"model":"coralbricks/glm-5.3-fast","input":"Reply with exactly: coral ok","max_output_tokens":300}'
HTTP/1.1 200 OK
x-litellm-call-id: ca513cda-e1f3-4cf4-9c3a-2b9eb9d78207
x-litellm-response-cost: 0.00021064
x-litellm-response-cost-original: 0.00021064
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.0
x-litellm-response-cost-output: 0.0001804
x-litellm-response-cost-cache-creation: 3.0240000000000002e-05
x-litellm-response-cost-reasoning: 0.0001628
x-litellm-response-cost-tool-usage: 0.0
{"id":"resp_UV9Q0i4-Xsd6UoZau_F07G1ZuQb2eRV4MR2TnHY4WW6sl714iKfPnJvTncBvvPmC5C4IeGZ0aXt4TlB1npcO_QSd7uC38-hZHwanrmoqaUWYsfwNOU_VewpDbhKOjpc0N5pBAEbbx6lp5LkhFlusn_3DJYDfbYp-HTxbpjeokD56JT3geEA-vlp6miVu2wcwTrOSkVSLwTdgnr-1g7psl8gXlgh1a12k1E9w0HCsDZsiATbYORA7aMaZbsz_W-2uxxf-pLuzvOkeeO9cXkPpx8Pwv6G6JfOo6EJ0lZ4veY8h7_BtNb35yhpT7_mdD4Z7FNld6Y81w7v_Vzd0Yy0EYsK7jxHOZV-ScXB5j569id371Rmuuv5NOv5VmERPE14eFn-Ko4x8ffuudp0ZYpjFqN5wSc5vVDcYfm-dnwm77jUOzIq3pX_4uv0xhDZl1xTc2Tc=","model":"coralbricks/glm-5.3-fast","status":"completed","text":"coral ok","usage":{"input_tokens":18,"input_tokens_details":{"audio_tokens":null,"cached_tokens":0,"cached_tokens_details":null,"image_tokens":null,"text_tokens":null,"video_tokens":null,"cache_write_tokens":18,"billable_cache_write_tokens":18},"output_tokens":41,"output_tokens_details":{"audio_tokens":null,"reasoning_tokens":37,"text_tokens":null},"total_tokens":59,"cost":0.00021064,"reasoning_output_tokens":37,"cost_details":{"currency":"USD","input":0.0,"cached_input":0.0,"cache_write":0.00003024,"output":0.0001804}}}

### 5. messages API, deepseek-v4.1-flash-fast
$ curl -s -D - $B/v1/messages -H "anthropic-version: 2023-06-01" ... -d '{"model":"coralbricks/deepseek-v4.1-flash-fast","max_tokens":50,"messages":[{"role":"user","content":"Reply with exactly: coral ok"}]}'
HTTP/1.1 200 OK
x-litellm-call-id: 328eb1cb-b643-403d-b998-25ced39377a9
x-litellm-response-cost: 5.7e-06
x-litellm-response-cost-original: 5.7e-06
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.0
x-litellm-response-cost-output: 4.8e-06
x-litellm-response-cost-cache-creation: 9e-07
x-litellm-response-cost-tool-usage: 0.0
{"id":"msg_3c92b4158e49499aba1259ca","model":"coralbricks/deepseek-v4.1-flash-fast","stop_reason":"end_turn","text":"coral ok","usage":{"input_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":10,"output_tokens":4}}

### 6. deprecated slug glm-5.3-fp4, chat, non-stream
$ curl -s -D - $B/v1/chat/completions ... -d '{"model":"coralbricks/glm-5.3-fp4","messages":[{"role":"user","content":"Reply with exactly: coral ok"}],"max_tokens":300}'
HTTP/1.1 200 OK
x-litellm-call-id: 8be70ae1-0d8d-4cef-95bd-da97bc2719fe
x-litellm-response-cost: 0.00036464
x-litellm-response-cost-original: 0.00036464
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.0
x-litellm-response-cost-output: 0.0003344
x-litellm-response-cost-cache-creation: 3.0240000000000002e-05
x-litellm-response-cost-reasoning: 0.0003168
x-litellm-response-cost-tool-usage: 0.0
{"id":"req_be61f45c11ad445a962bdf4a715b4ab9","model":"coralbricks/glm-5.3-fp4","content":"coral ok","usage":{"completion_tokens":76,"prompt_tokens":18,"total_tokens":94,"completion_tokens_details":{"reasoning_tokens":72},"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":18,"cache_creation_tokens":18,"billable_cache_write_tokens":18,"cache_write_blocks":3},"cost":0.00036464,"cost_details":{"currency":"USD","input":0.0,"cached_input":0.0,"cache_write":0.00003024,"output":0.0003344}}}

### 7. control: openai/gpt-5.4-mini chat
$ curl -s -D - $B/v1/chat/completions ... -d '{"model":"openai/gpt-5.4-mini","messages":[{"role":"user","content":"Reply with exactly: control ok"}],"max_tokens":20}'
HTTP/1.1 200 OK
x-litellm-call-id: 2e4837af-b0bb-4110-9266-260d50ae039f
x-litellm-response-cost: 3.15e-05
x-litellm-response-cost-original: 3.15e-05
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 9e-06
x-litellm-response-cost-output: 2.25e-05
x-litellm-response-cost-tool-usage: 0.0
{"id":"chatcmpl-EYFUWfe2jNod04vzPS21jW2UYEpws","model":"openai/gpt-5.4-mini","content":"control ok"}

### 8. model info rates for the CoralBricks deployments
$ curl -s $B/v1/model/info -H "Authorization: Bearer $KEY" | jq -c '.data[] | select(.model_name | startswith("coralbricks/")) | {model_name, litellm_provider: .model_info.litellm_provider, input: .model_info.input_cost_per_token, output: .model_info.output_cost_per_token, cache_write: .model_info.cache_creation_input_token_cost, cache_read: .model_info.cache_read_input_token_cost}'
{"model_name":"coralbricks/glm-5.3-fast","litellm_provider":"coralbricks","input":0.00000112,"output":0.0000044,"cache_write":0.00000168,"cache_read":0.0}
{"model_name":"coralbricks/deepseek-v4.1-flash-fast","litellm_provider":"coralbricks","input":1E-8,"output":0.0000012,"cache_write":9E-8,"cache_read":0.0}
{"model_name":"coralbricks/glm-5.3-fp4","litellm_provider":"coralbricks","input":0.00000112,"output":0.0000044,"cache_write":0.00000168,"cache_read":0.0}
(rows printed: 3)

### 9. spend log row for case 1
$ curl -s "http://127.0.0.1:38554/spend/logs?request_id=req_fbe4ee05b378465f9fd8c9dc1ed0aacb" -H "Authorization: Bearer $KEY" | jq -c '.[0] | {model, custom_llm_provider, prompt_tokens, completion_tokens, spend}'
{"model":"coralbricks/glm-5.3-fast","custom_llm_provider":"coralbricks","prompt_tokens":18,"completion_tokens":28,"spend":0.00015344}

### 10. wildcard deployment coralbricks/* lists the catalog on /v1/models
$ curl -s -X POST $B/model/new -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model_name":"coralbricks/*","litellm_params":{"model":"coralbricks/*","api_key":"os.environ/CORALBRICKS_API_KEY"}}' | jq -c '{model_name, id: .model_info.id} // .'
{"model_name":"coralbricks/*","id":"1f9f672f-dda7-4740-978a-275545026991"}
$ curl -s $B/v1/models -H "Authorization: Bearer $KEY" | jq -c '[.data[].id | select(startswith("coralbricks/"))] | sort'
["coralbricks/deepseek-v4.1-flash","coralbricks/deepseek-v4.1-flash-fast","coralbricks/deepseek-v4.1-flash-fast","coralbricks/deepseek-v4.1-flash-fast-fp4","coralbricks/glm-5.3","coralbricks/glm-5.3-fast","coralbricks/glm-5.3-fast","coralbricks/glm-5.3-fp4","coralbricks/glm-5.3-fp4"]

### 11. new deprecated slugs glm-5.3 and deepseek-v4.1-flash, chat non-stream, each added as a deployment
$ curl -s -X POST http://127.0.0.1:38554/model/new -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model_name":"coralbricks/glm-5.3","litellm_params":{"model":"coralbricks/glm-5.3","api_key":"os.environ/CORALBRICKS_API_KEY"}}' | jq -c '{model_name, id: .model_info.id}'
{"model_name":"coralbricks/glm-5.3","id":"e5c38a8d-14fd-413d-b5b6-3b2ef502a40a"}
$ curl -s -D - http://127.0.0.1:38554/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model":"coralbricks/glm-5.3","messages":[{"role":"user","content":"Reply with exactly: coral ok"}],"max_tokens":300}'
HTTP/1.1 200 OK
x-litellm-call-id: 9a8e96f2-dc83-4ec6-83e3-4c2a00460836
x-litellm-response-cost: 0.00036464
x-litellm-response-cost-original: 0.00036464
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.0
x-litellm-response-cost-output: 0.0003344
x-litellm-response-cost-cache-creation: 3.0240000000000002e-05
x-litellm-response-cost-reasoning: 0.0003168
x-litellm-response-cost-tool-usage: 0.0
{"id":"req_06c5c7d979974bf3815e53cdd833475f","model":"coralbricks/glm-5.3","content":"coral ok","usage":{"completion_tokens":76,"prompt_tokens":18,"total_tokens":94,"completion_tokens_details":{"reasoning_tokens":72},"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":18,"cache_creation_tokens":18,"billable_cache_write_tokens":18,"cache_write_blocks":3},"cost":0.00036464,"cost_details":{"currency":"USD","input":0.0,"cached_input":0.0,"cache_write":0.00003024,"output":0.0003344}}}
$ curl -s "http://127.0.0.1:38554/spend/logs?request_id=req_06c5c7d979974bf3815e53cdd833475f" -H "Authorization: Bearer $KEY" | jq -c '.[0] | {model, custom_llm_provider, prompt_tokens, completion_tokens, spend}'
{"model":null,"custom_llm_provider":null,"prompt_tokens":null,"completion_tokens":null,"spend":null}

$ curl -s -X POST http://127.0.0.1:38554/model/new -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model_name":"coralbricks/deepseek-v4.1-flash","litellm_params":{"model":"coralbricks/deepseek-v4.1-flash","api_key":"os.environ/CORALBRICKS_API_KEY"}}' | jq -c '{model_name, id: .model_info.id}'
{"model_name":"coralbricks/deepseek-v4.1-flash","id":"c0455cb4-90c4-4708-b747-2fa053850876"}
$ curl -s -D - http://127.0.0.1:38554/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model":"coralbricks/deepseek-v4.1-flash","messages":[{"role":"user","content":"Reply with exactly: coral ok"}],"max_tokens":300}'
HTTP/1.1 200 OK
x-litellm-call-id: dd5de478-fb6d-447b-aba2-76c4c247ab81
x-litellm-response-cost: 5.7e-06
x-litellm-response-cost-original: 5.7e-06
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.0
x-litellm-response-cost-output: 4.8e-06
x-litellm-response-cost-cache-creation: 9e-07
x-litellm-response-cost-tool-usage: 0.0
{"id":"req_93c2e5c5b80f4dd3aeac49886db87f72","model":"coralbricks/deepseek-v4.1-flash","content":"coral ok","usage":{"completion_tokens":4,"prompt_tokens":10,"total_tokens":14,"completion_tokens_details":{"reasoning_tokens":0},"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":10,"cache_creation_tokens":10,"billable_cache_write_tokens":10,"cache_write_blocks":3},"cost":0.0000057,"cost_details":{"currency":"USD","input":0.0,"cached_input":0.0,"cache_write":9E-7,"output":0.0000048}}}
$ curl -s "http://127.0.0.1:38554/spend/logs?request_id=req_93c2e5c5b80f4dd3aeac49886db87f72" -H "Authorization: Bearer $KEY" | jq -c '.[0] | {model, custom_llm_provider, prompt_tokens, completion_tokens, spend}'
{"model":null,"custom_llm_provider":null,"prompt_tokens":null,"completion_tokens":null,"spend":null}


### 11b. spend rows for case 11 re-read after the batched spend write (the first read in case 11 ran before the write)
$ curl -s "http://127.0.0.1:38554/spend/logs?request_id=req_06c5c7d979974bf3815e53cdd833475f" -H "Authorization: Bearer $KEY" | jq -c '.[0] | {model, custom_llm_provider, prompt_tokens, completion_tokens, spend}'
{"model":"coralbricks/glm-5.3","custom_llm_provider":"coralbricks","prompt_tokens":18,"completion_tokens":76,"spend":0.00036464}
$ curl -s "http://127.0.0.1:38554/spend/logs?request_id=req_93c2e5c5b80f4dd3aeac49886db87f72" -H "Authorization: Bearer $KEY" | jq -c '.[0] | {model, custom_llm_provider, prompt_tokens, completion_tokens, spend}'
{"model":"coralbricks/deepseek-v4.1-flash","custom_llm_provider":"coralbricks","prompt_tokens":10,"completion_tokens":4,"spend":0.0000057}

The same Add Model flow: the Provider search offers CoralBricks, the picker lists its catalog, and API Base shows https://inference.coralbricks.ai/v1 as its placeholder

After: Add Model provider search lists CoralBricks

After: the Select models picker lists the CoralBricks catalog and the API Base placeholder

Observed along the way, none changed here:

  • Every uncached prompt token bills as a cache write, input cost 0.0
  • /v1/messages reports input_tokens 0, all under cache_creation_input_tokens
  • cache_read_input_token_cost is 0.0 on every CoralBricks row
  • DeepSeek's cache-write rate 9e-8 is nine times its input rate
  • A repeated prompt still writes a few hundred tokens to cache
  • CoralBricks' usage.cost equals x-litellm-response-cost on every case
  • /v1/models repeats a name both configured and wildcard-expanded (shared concat)
  • On the base the wildcard create is a 500, the chat calls 400
  • On the base a streaming error arrives as plain JSON, not SSE

/live-pr-risk at a4fb718, then merged with main as 9f355f016a on the rig (the branch's own merge of main is 93c5048)

Verdict: pass. The diff since 18ffd76 is additive: two cost-map rows (coralbricks/glm-5.3-fast, coralbricks/deepseek-v4.1-flash-fast), a repriced row (coralbricks/deepseek-v4.1-flash-fast-fp4 input 3e-07 to 1e-08), two placeholder strings, and coralbricks_models joining the provider model sets in litellm/__init__.py (_populate_provider_model_sets's parameter is now typed Mapping[str, Mapping[str, object]], an annotation with no runtime change). The readers of the new set are get_valid_models (explicit provider, and env-inferred through CORALBRICKS_API_KEY), get_provider_models behind wildcard expansion on /v1/models, /model/info, and key model lists, pick_cheapest_chat_models_from_llm_provider behind the wildcard health probe, and litellm.model_list, so this run re-checked every row of the 18ffd76 table at the tip and added the rows those readers reach. Both sides ran 2 uvicorn workers on their own Postgres, no Redis, STORE_MODEL_IN_DB=True, LITELLM_LOCAL_MODEL_COST_MAP=True, on the GCE lane: the merge base 29f48e6 on 127.0.0.1:33079 and the tip on 127.0.0.1:45539, each with a forwarding recorder in front of https://inference.coralbricks.ai so the vendor wire could be diffed. origin/main (7af7919) is 272 commits past the base and moves pyproject.toml, uv.lock, and the Rust tree, so the tip was merged with it (clean, 9f355f016a), the merged tree resynced, its Rust bridge rebuilt, and the proxy restarted on the same port, where every cell below answered the same as the tip (/public/providers 169 and /public/providers/fields 132 there, main having added two providers of its own), with one main-side difference: an openai/*-scoped key no longer lists the coralbricks catalog on /v1/models (0 rows across 4 tries after both workers reloaded) where the tip listed it, while the coralbricks/*-scoped key and the master key still list the 4 rows, so a wildcard-scoped key now expands only its own provider, a change main made between the base and 7af7919 and not this PR's

Breaking: none. Backward incompatible: none. Every pre-existing path answers identically on both sides, and the public registries differ only by the added entry: /public/providers 166 to 167, /public/providers/fields 129 to 130 with the CoralBricks placeholder coralbricks/glm-5.3-fast, /model/settings 166 to 167, /public/endpoints lists coralbricks under chat_completions, messages, and responses

Untested dependents driven live, the same request on both sides (a base cell answers 400 invalid model because no coralbricks deployment can go live there):

Dependent path Entrypoint Base 29f48e6 Head a4fb718
api_base inference /model/new, bare glm-5.3-fast + api_base, then chat 500 not live, chat 400 200, chat 200
stored credential /credentials + /model/new by name, then chat 500 not live, chat 400 200, chat 200
openai-compatible params chat with response_format json_object 400 invalid model 200 {"ok": true}
openai-compatible params chat with an unknown param, drop_params on 400 invalid model 200
streaming chat stream: true with include_usage 400, 0 chunks 200, 20 chunks, usage, spend row
speech dispatch /v1/audio/speech 400 invalid model 404, vendor Not Found
embeddings dispatch /v1/embeddings 400 invalid model 400 unmapped provider
health check /health?model=<deployment> 0 healthy, 0 unhealthy 1 healthy
wildcard listing /model/new coralbricks/*, then /v1/models 500 not live, no catalog names 200, the 4 catalog rows listed
wildcard listing /model/info through the wildcard no coralbricks rows 4 rows at the cost-map prices
wildcard listing /v1/model/info, /model_group/info no rows, no group 4 names, no group
scoped key key models: ["coralbricks/*"], then /v1/models every proxy model every proxy model plus the 4 rows (on the merged tree an openai/* key no longer gets them, see above)
deprecated slug chat coralbricks/deepseek-v4.1-flash-fast-fp4 through the wildcard 400 invalid model 200, spend 0.0000057
non-catalog name chat coralbricks/not-a-model through the wildcard 400 invalid model 404, vendor does not exist
messages, responses /v1/messages deepseek, /v1/responses glm through the wildcard 400 invalid model 200, 200
wildcard health probe full /health with the wildcard present 1 healthy, 1 unhealthy 5 healthy, 1 unhealthy, the wildcard probed one catalog row
env-inferred models get_valid_models() with CORALBRICKS_API_KEY set no coralbricks rows the 4 rows
hosted map upgrade the base proxy rebooted with LITELLM_MODEL_COST_MAP_URL at the tip's map ready, fetched the map 5 times, /model/info and a control chat bill at the map's openai rates (0.000036), get_model_info prices the 4 coralbricks rows, but coralbricks is still not in provider_list, get_valid_models is empty with the pre-existing not a valid LlmProviders warning, /model/new coralbricks still 500 not live, and a coralbricks/* row a new pod wrote logs Error upserting deployment ... LLM Provider NOT provided on every reload of the old pod, a warning only n/a

The api_base-inferred chat's spend row names provider coralbricks and model glm-5.3-fast with spend 0.00033824 (18 cache-write tokens at 1.68e-06 and 70 output tokens at 4.4e-06); the credential-backed deepseek chat's row is 0.0000057 (10 cache-write tokens at 9e-08 and 4 output tokens at 1.2e-06), the deprecated fp4 slug through the wildcard bills the same 0.0000057 for the same usage at the new rate, and the streaming glm chat's row is 0.00030096 (22 cache-write, 60 output). The recorder in front of the vendor logged 17 requests on the head and 0 on the base, one vendor request per proxy request with no retry behind either 404, the same header set per client (the OpenAI SDK headers on chat and speech, plain httpx on /v1/responses, anthropic-version on /v1/messages). On the head tree with the local cost map, get_valid_models(custom_llm_provider="coralbricks") returns the four rows (empty at 18ffd76), litellm.models_by_provider["coralbricks"] and litellm.model_list carry them, pick_cheapest_chat_models_from_llm_provider("coralbricks", 3) picks the fp4 slug, deepseek-v4.1-flash-fast, and glm-5.3-fast, and get_model_info on each row matches the cost map (glm 1.12e-06 in, 4.4e-06 out, 1.68e-06 cache write, vision false; both deepseek slugs 1e-08 in, 1.2e-06 out, 9e-08 cache write, vision true; cache read 0 everywhere)

Dependents already tested through the change and not re-driven here: chat, responses, and messages sync, async, streaming, and raw (the wire suite and the /qa leg above), 4xx, 5xx, and unreachable error mapping (the chaos suite), key scoping, fallback, the add-model fields and the dashboard picker (the browser cell and the /qa screenshots), config versus env credential precedence (the env wire suite), the catalog behind a wildcard (test_coralbricks_rows_are_listed_for_wildcard_expansion and the wire test test_wildcard_deployment_lists_every_coralbricks_catalog_row), the deprecated slug's price (test_coralbricks_deprecated_slug_bills_like_its_current_slug), and every cost-map provider having a registry entry (test_every_cost_map_provider_is_registered). The 13 unit files that read models_by_provider, model_list, or the cost map without naming coralbricks (test_utils, test_model_checks, test_health_check_helpers, test_get_llm_provider_logic, test_get_model_cost_map, test_constants, test_cost_calculator, test_router_model_cost_isolation, test_model_prices_schema, the two MCP sampling files, the soniox registration file, and this PR's unit file) ran on the merged tree: 1373 passed, 1 skipped, none pins a provider or model count. Legacy suites: no hit in tests/llm_translation or tests/local_testing; the tests/integration hits are this PR's own three files, run under the audit rounds

Not verified, each the generic path every JSON provider already takes: /v1/images/generations and the assistants routes with a coralbricks model (same dispatch class as speech, the vendor serves neither), websearch interception with coralbricks as an enabled provider, /config/cost_discount_config or /config/cost_margin_config naming coralbricks, and the rendered add-model form with the new placeholder (its API row, /public/providers/fields, is in the table)

Observations, each reproduced on the base and so pre-existing, none for this PR to fix: a key scoped to a provider wildcard (anthropic/*, openai/*, or coralbricks/*) lists every proxy model on /v1/models and can call any configured deployment (an anthropic/* key called control-openai and openai/gpt-5.4-mini through an openai/* deployment on both sides, and coralbricks/glm-5.3-fast on the head), worth its own ticket; a vendor 404 puts a single deployment into cooldown for the minute (an openai control answered one 404 then 429 on both sides, per worker without Redis), which is why the first head pass saw embeddings and streaming answer 429 after the speech cell and the table reports the re-run on a fresh deployment ordered before speech; a chat with a just-deleted key answered 200 once on both sides (the other worker's key cache); and an api_base-inferred deployment shows null provider and prices in /model/info while its spend row bills at the cost-map rate, as at 18ffd76. No vendor 429 or quota error; every 429 above was the proxy's own cooldown

/live-pr-risk at d5bbab5, then merged with main as 5e2e004dca on the rig

Verdict: pass. The delta since a4fb718 is two cost-map rows, coralbricks/glm-5.3 (a copy of coralbricks/glm-5.3-fp4) and coralbricks/deepseek-v4.1-flash (a copy of coralbricks/deepseek-v4.1-flash-fast-fp4), in model_prices_and_context_window.json and the backup map, plus test-only edits; the merge commit 93c5048 brings main f516668 in and touches no symbol the PR owns (its 8 commits are decisions, supports_video_input, other providers' rows, one endpoint flag). Every consumer of the map that the two rows reach was driven on a real proxy against the real CoralBricks API, base f516668 against the tip, two uvicorn workers each with Postgres and Redis, a forwarding recorder in front of the vendor on every leg

Breaking: none. Backward incompatible: none. The public registries differ only by the rows: /public/litellm_model_cost_map 4115 entries with no coralbricks rows on base, 4129 with the 6 coralbricks rows on the tip; /public/providers 168 to 169 and /public/providers/fields 131 to 132 with the placeholder still coralbricks/glm-5.3-fast, both unchanged by this delta

Untested dependents of the two rows driven live, the same request on both sides (a base cell answers 400 invalid model because no coralbricks deployment can go live there):

Dependent path Entrypoint Base f516668 Tip d5bbab5
picker data /public/litellm_model_cost_map, catalog and full no coralbricks rows 6 rows, the 2 new at the copied prices
dashboard picker /ui Add Model, provider CoralBricks, model list provider absent 8 options: Custom, Wildcard, the 6 rows (screenshot in the QA proof)
wildcard listing /model/new coralbricks/*, then /v1/models x4 500 not live, [] the 6 rows on every call, both workers
wildcard listing /model/info, /v1/model/info [] 6 rows at the map prices
wildcard group /model_group/info?model_group=coralbricks/* no group no group
valid models get_valid_models, model_list, models_by_provider in-process, 3 hash seeds empty, not a valid LlmProviders warning the 6 rows on every seed
cheapest rows pick_cheapest_chat_models_from_llm_provider("coralbricks", 3), 6 seeds [] the 3 deepseek rows, seed-dependent order
wildcard health /health?model=coralbricks/* x4 503, 0 healthy, 0 unhealthy 200, 1 healthy each; probes went to deepseek-v4.1-flash-fast-fp4 x3 and deepseek-v4.1-flash x1
chat on the new slugs coralbricks/glm-5.3, coralbricks/deepseek-v4.1-flash through the wildcard 400 invalid model 200, spend 0.00038448 and 0.0000057
streaming deepseek stream: true with include_usage 400, 0 chunks 200, usage 14/10, spend 0.00001326
messages, responses /v1/messages deepseek, /v1/responses glm 400 invalid model 200 spend 0.0000057, 200 spend 0.00012872
key budget key max_budget 0.0001 on coralbricks/*, glm chat as that key, then 2 more 400, key spend 0 200, key spend 0.00020984 within 1s, then 422 x2
model budget key model_max_budget deepseek 1e-06 per day, chat as that key, then 2 more 400 200 spend 0.0000066, then 422 x2
scoped key key models: ["coralbricks/*"], /v1/models x2 and /model/info [] the 6 rows on each
single-slug key key models: ["coralbricks/glm-5.3"]: glm chat, deepseek chat, /v1/models 400, 403, the slug 200 spend 0.00024064, 403, the slug
vendor wire recorder in front of the vendor, every leg 0 requests 21 requests, one per vendor-bound proxy call, all 200 with auth

The spend rows reproduce the copied prices: the key-budget glm chat billed 28 prompt tokens as cache-write at 1.68e-06 plus 37 output tokens at 4.4e-06 (0.00020984), the model-budget deepseek chat 20 cache-write tokens at 9e-08 plus 4 output tokens at 1.2e-06 (0.0000066), and the first glm chat 17 cached tokens at the row's cache-read price 0.0, 1 cache-write token, and 87 output tokens (0.00038448). The vendor serves both deprecated slugs directly at the same rates, and its /v1/models lists only the two -fast names. The 422 and 403 cells produced no vendor request, and no cell produced more than one

Merge-ref check: the ctl leg ran the clean merge 5e2e004dca (d5bbab5 with main a86ad9e, the map +333/-2 with zero coralbricks lines) through the same cells and answered as the tip on every row above, with the health probes landing on deepseek-v4.1-flash-fast x2 and deepseek-v4.1-flash x2 (the three deepseek rows tie at the cheapest price and the set order is per process), the key budget tripping at 0.00065048, and 13 vendor requests. Main has since moved 3 commits (to 932a0d6: MCP OAuth scopes, Admin UI session failure rows, an e2e rename) that touch no file the PR owns and carry no coralbricks line

Dependents already proved at a4fb718 and untouched by this delta (no provider code moved): api_base inference, stored credentials, response_format and drop_params, speech and embeddings dispatch, the deprecated fp4 slug, a non-catalog name, env-inferred credentials, and the hosted-map upgrade of a base proxy, all in the section above

Not verified: the hosted-map upgrade re-pointed at the tip's map (the a4fb718 cell proved the mechanism and the delta only adds rows to it), /v1/images/generations and the assistants routes with a coralbricks model, websearch interception with coralbricks enabled, and the cost discount and margin config naming coralbricks, each the generic path every JSON provider already takes

Observations, none for this delta to fix: a request carrying two Authorization headers is authenticated as the first one (the rig's helper sent the master key ahead of the virtual key at first, so the budget and scoped-key cells above are the re-runs with one header, and the first attempt billed the master key); the Redis response cache answers an identical chat with spend 0.0 and no budget consumption, so the budget cells carry a nonce; with no vendor key resolvable (the ctl leg's first boot), /v1/chat/completions fails before the wire with a 500 AuthenticationError while /v1/messages and /v1/responses send the request upstream without an Authorization header and relay the vendor's 401, which is the provider's handling from before this delta and the same on every leg that lacks the key

Audit round 3 of 3 (at d5bbab5)

/audit run at the tip d5bbab5 (the branch's merge of main 93c5048 plus the cost-map rows coralbricks/glm-5.3 and coralbricks/deepseek-v4.1-flash, two unit cases, and the test files below) against the merge base f516668. The 53 deterministic cells, the controls, and the browser cell ran on the synced tree 113351dece (every product file identical to the tip); the driver then rewrote test_coralbricks_provider_chaos.py for the TQ010 gate (tests only, tree 17b0284724, every product file and every other test file unchanged), so its 4 cells re-ran on both legs at that tree, twice on head, and every other row keeps its result, per the audit skill's rerun rule. One proxy per leg from tests/integration/proxy_config.yaml with 2 uvicorn workers (--num_workers 2), its own Postgres database, Redis, STORE_MODEL_IN_DB=True, LITELLM_LOCAL_MODEL_COST_MAP=True, and LITELLM_UI_PATH at the leg's own ui/litellm-dashboard/out build, on a GCE lane (the merge base on 127.0.0.1:48325, the tip on 127.0.0.1:58825 through an ssh tunnel). Every deterministic cell's provider is a real HTTP peer the test owns (tests/integration/_support/wire.py), so each cell asserts what the caller received, what the provider received, and the Postgres spend row by response id. The live rows reach the real CoralBricks API (https://inference.coralbricks.ai, real spend) through the same two proxies booted with CORALBRICKS_API_KEY from .env, the master key in $KEY

Cells: 53 node ids under tests/integration/providers/ (test_coralbricks_provider_wire.py 38, test_coralbricks_provider_env_wire.py 11, test_coralbricks_provider_chaos.py 4, sharing _coralbricks.py) plus the browser cell tests/e2e/ui/tests/integrationCritical/addModelCoralBricks.spec.ts, registered in expected.json. New this round: test_wildcard_deployment_lists_every_coralbricks_catalog_row (E9), which derives the catalog from the leg's own cost map and asserts every coralbricks/ row on /v1/models and /model/info. Leg command, the same 53 ids on both legs:

uv run --no-sync python tests/integration/run.py providers <53 node ids> --seed 141 --order-seed 141 --results <dir>

Verdict: PASS. Every inventory row (H1 to H15, S1 to S9, E1 to E9, M1 to M4 with the secrecy cell, C1 to C4, U1, L1 to L4) is VERIFIED: red on the merge base for the reason its base cell names, green on the tip twice with the same seeds, the controls green on both legs, the live rows reaching the real vendor on the tip only

Before (f516668)

/v1/chat/completions

  1. Cells H1 to H5 (sync on both models, async, stream sync, stream async, raw httpx) register a coralbricks/deepseek-v4.1-flash-fast or coralbricks/glm-5.3-fast deployment through POST /model/new
  2. Observed: 500 "Model create was saved to the database, but the model id(s) [...] are not live in this pod's router", so every chat cell is red at setup
  3. Live: POST /model/new for a3-rec-glmd5b5 (coralbricks/glm-5.3-fast, real key) and a3-rec-dsd5b5 (coralbricks/deepseek-v4.1-flash-fast) answers 500 the same way (live/d5bbab56a5/base/new_a3-rec-glmd5b5.body), and POST /v1/chat/completions on them 400 "Invalid model name passed in", the stream too, 0 chunks (L1_chat.body, L1_stream.body)

/v1/responses

  1. Cells H6 to H10 register the same deployment
  2. Observed: the same 500 at /model/new; POST /v1/responses on the live deployment 400 "Invalid model name passed in" (L1_responses.body)

/v1/messages

  1. Cells H11 to H15 register the same deployment
  2. Observed: the same 500 at /model/new; POST /v1/messages on the live deployment 400 the same way (L1_messages.body)

Credentials and provider inference

  1. Cells M1 to M4 boot an owned proxy with CORALBRICKS_API_KEY and CORALBRICKS_API_BASE set and a config model coralbricks-env-config (coralbricks/deepseek-v4.1-flash-fast) with no credentials
  2. Observed: the config model never loads ("no healthy deployments" on all three endpoints), /model/info has no row for it, and the /model/new cells (M2 to M4) 500 at setup
  3. Live L2 (POST /model/new with only the env key, catalog base) and L3 (unprefixed glm-5.3-fast with api_base: https://inference.coralbricks.ai/v1): both 500 at /model/new (new_a3-envcred-glmd5b5.body, new_a3-inferred-glmd5b5.body), the chats on them 400 "Invalid model name passed in" (L2_chat.body, L3_chat.body)
  4. Live L4 (bogus key at the catalog base): /model/new 500, the chat 400 before any vendor call (L4_chat.body)

Sad paths and chaos

  1. Cells S1 to S8 (provider 401, 404, 429, 500, refused port, missing or null or empty key) and C1 to C4 (provider outage mid burst, slow streams, worker SIGKILL, proxy SIGTERM and reboot) register a coralbricks/ deployment first
  2. Observed: all red at setup (/model/new 500; C3 and C4 "State did not converge: 0", the burst never reached the wire); the chaos rerun at tree 17b0284724 is red the same way (C1 and C2 /model/new 500 in under a second, C3 and C4 "State did not converge: 0" after 129 s each, chaos-base 4 collected, 0 passed, pytest 212 s)
  3. Cells S9 (model field as an int, a list, a 5 KB string): 4xx each and /health/liveliness 200, green on this leg too (controls)

Listing

  1. E2, E3 and E9 ask /model/info, /v1/models, /model_group/info for the coralbricks rows
  2. Observed: E2 and E3 KeyError on coralbricks/deepseek-v4.1-flash-fast (no such row in the merge base cost map), E5 finds no CORALBRICKS entry on /public/providers/fields, E9 500 at /model/new for the coralbricks/* deployment

Admin UI Add Model

  1. The browser cell signs in with the master key, opens /ui/?page=add-model, types CoralBricks into the Provider combobox
  2. Observed: the option never appears (locator.click times out after 15 s waiting for getByRole('option', { name: /CoralBricks$/ }) with CoralBricks typed into the expanded Provider combobox, the picker shows "No providers found"), the cell fails at the provider step (phase12/d5bbab56a5/ui-base)

Full matrix

  1. The Base column of the After matrix: 50 red at setup for the reasons above (46 kept rows plus the 4 chaos rows re-run), 3 green controls

After (d5bbab5)

/v1/chat/completions

  1. Cells H1 to H5 against the tip: /model/new 200, each chat 200 with the scripted id, the provider received POST /v1/chat/completions with Authorization: Bearer <deployment key> and the model prefix stripped, one spend row per response id with custom_llm_provider = coralbricks, 4357 prompt (4352 cached, 5 cache write) and 20 completion tokens, spend 2.445e-05 on deepseek-v4.1-flash-fast and 9.64e-05 on glm-5.3-fast (the usage frame of the stream bills the same)
  2. Live: /model/new 200 for a3-rec-glm20f6 (coralbricks/glm-5.3-fast) and a3-rec-ds20f6 (coralbricks/deepseek-v4.1-flash-fast), POST /v1/chat/completions on the glm deployment 200 (req_9d10dcc9...), 20 prompt and 30 completion tokens, x-litellm-response-cost 0.0001656 (30 x 4.4e-06 output plus the 20 prompt tokens the vendor reports as cache creation at 1.68e-06), the spend row custom_llm_provider coralbricks, api_base https://inference.coralbricks.ai/v1/, spend 0.0001656 (live/d5bbab56a5/head/L1_chat.{hdr,body,spend}); the streamed chat on the deepseek deployment 200, 5 chunks ending in [DONE], its row 12 prompt and 7 completion tokens, spend 9.48e-06 (L1_stream.*)

/v1/responses

  1. Cells H6 to H10: 200, the provider received POST /v1/responses natively (no chat bridge) with the input intact, the stream's created, delta and completed events all forwarded, one row per response id (call_type aresponses)
  2. Live: POST /v1/responses 200 (resp_zeZA..., status incomplete at max_output_tokens 30, the vendor's reasoning used the budget), cost 0.0001876, the row call_type aresponses, api_base .../v1/responses (L1_responses.*)

/v1/messages

  1. Cells H11 to H15: 200 with a msg_ id, the provider received POST /v1/messages with the bearer key and anthropic-version, the full Anthropic SSE forwarded, one row (call_type anthropic_messages)
  2. Live: POST /v1/messages 200 (msg_ebf6e949...), cost 0.0001656, the row call_type anthropic_messages, api_base .../v1/messages (L1_messages.*)

Credentials and provider inference

  1. Cells M1 to M4 (11 node ids): the config model with no credentials sends the env key to the env base on all three endpoints, explicit api_base and api_key beat the env pair, a missing or null or empty api_key or api_base falls through to the env value, and /model/info never echoes the env key
  2. Live L2: /model/new with only CORALBRICKS_API_KEY set and no api_base 200, chat 200, spend row api_base https://inference.coralbricks.ai/v1/, cost 0.0001744 (L2_chat.*)
  3. Live L3: unprefixed glm-5.3-fast with the catalog api_base 200, spend row custom_llm_provider coralbricks, model glm-5.3-fast, cost 0.00017 (L3_chat.*)
  4. Live L4: a bogus key at the catalog base reaches the vendor and the caller gets its 401 with the vendor's "invalid api key" message, x-litellm-response-cost 0 (L4_chat.body)

Sad paths and chaos

  1. S1 to S6: the provider's 401 and 404 reach the caller after exactly one attempt on chat, responses and messages; 429 and 500 after 3 attempts (1 + num_retries 2) with the provider message in the body
  2. S7: a refused api_base answers 5xx with the connection error in the body and the proxy serves a control request after; S8: no key anywhere fails before the provider, naming the missing key, 0 provider requests
  3. C1 to C4 at tree 17b0284724 (the TQ010 rewrite, owned.replacing_worker and owned.begin_shutdown): C1: 30 concurrent requests across the three endpoints while the provider is down, /health?model= unhealthy during the outage and healthy after the restart on the same port, every x-litellm-call-id lands exactly once; C2: 20 paused streams complete with one row each; C3: a SIGKILLed worker's sibling keeps serving; C4: the SIGTERMed proxy's 200s land once, the rebooted proxy serves the deploy; green twice at that tree with the same seeds (chaos-head-r1, chaos-head-r2), collected and passed selections identical

Listing

  1. E2: /model/info reports the cost-map rates and limits of the deployment; E3: /v1/models and /model_group/info carry it; E5: /public/providers/fields names the CoralBricks credential fields; E7: a coralbricks/* deployment routes the requested model; E8: a refused primary falls back onto coralbricks
  2. E9 (new): a coralbricks/* deployment lists all 6 catalog rows (glm-5.3-fast, glm-5.3-fp4, glm-5.3, deepseek-v4.1-flash-fast, deepseek-v4.1-flash-fast-fp4, deepseek-v4.1-flash) on /v1/models, and /model/info expands it into one entry per row with litellm_params.model equal to the row and litellm_provider coralbricks, the catalog read from the leg's own cost map rather than a literal count

Admin UI Add Model

  1. The browser cell picks CoralBricks in the Provider combobox, fills the custom model name, the public name, the API base and the API key, submits
  2. Observed: /model/new receives model: deepseek-v4.1-flash-fast, custom_llm_provider: coralbricks, the api_base and the api_key; a chat on the new deployment 200; its spend row has custom_llm_provider coralbricks, 20 prompt and 20 completion tokens, spend 2.42e-05 (20 x 1e-08 + 20 x 1.2e-06, the rates read back from /model/info); green twice against the tip's own dashboard build (phase12/d5bbab56a5/ui-head-r1 14.2 s, ui-head-r2 23.7 s)

Full matrix

  1. Head run 1 (tree 113351dece): 53 passed, 0 skipped in 169 s (execution.json: hypothesis_seed 141, order_seed 141, complete true)
  2. Head run 2, same seeds: 53 passed in 167 s, collected and passed selections identical to run 1, no retries
  3. Chaos rerun at tree 17b0284724 (d5bbab5), same seeds: run 1 4 passed, 0 skipped, pytest 186 s (complete true), run 2 4 passed, pytest 245 s, collected and passed selections identical to run 1; the base leg at the same tree 4 collected, 0 passed, pytest 212 s; the control lane (3 S9 ids on the ctl leg): 3 passed in 43 s
  4. Per cell (base cell = why the merge base is red; "Last ran at" names the tree each row's result comes from):
Row Node id under tests/integration/providers/ Base f516668 Head d5bbab5 run 1 Head run 2 Last ran at Verdict
H1 test_coralbricks_provider_wire.py::test_openai_sdk_chat_reaches_coralbricks_and_bills_the_cost_map_row[deepseek-v4.1-flash-fast] red: /model/new 500: deployment not live in the router (0s) passed (3s) passed (3s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H1 test_coralbricks_provider_wire.py::test_openai_sdk_chat_reaches_coralbricks_and_bills_the_cost_map_row[glm-5.3-fast] red: /model/new 500: deployment not live in the router (0s) passed (5s) passed (3s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H2 test_coralbricks_provider_wire.py::test_async_openai_sdk_chat_reaches_coralbricks red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (3s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H3 test_coralbricks_provider_wire.py::test_openai_sdk_chat_stream_is_forwarded_and_billed_from_the_usage_frame red: /model/new 500: deployment not live in the router (0s) passed (4s) passed (2s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H4 test_coralbricks_provider_wire.py::test_async_openai_sdk_chat_stream_is_forwarded_and_billed red: /model/new 500: deployment not live in the router (0s) passed (4s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H5 test_coralbricks_provider_wire.py::test_raw_chat_request_carries_the_call_id_and_lands_once red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (3s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H6 test_coralbricks_provider_wire.py::test_openai_sdk_responses_go_natively_to_coralbricks_responses red: /model/new 500: deployment not live in the router (0s) passed (2s) passed (2s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H7 test_coralbricks_provider_wire.py::test_async_openai_sdk_responses_go_natively_to_coralbricks_responses red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H8 test_coralbricks_provider_wire.py::test_openai_sdk_responses_stream_is_forwarded_natively_and_billed_once red: /model/new 500: deployment not live in the router (0s) passed (7s) passed (7s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H9 test_coralbricks_provider_wire.py::test_async_openai_sdk_responses_stream_is_forwarded_natively_and_billed_once red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H10 test_coralbricks_provider_wire.py::test_raw_responses_request_lands_once red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H11 test_coralbricks_provider_wire.py::test_anthropic_sdk_messages_go_natively_to_coralbricks_messages red: /model/new 500: deployment not live in the router (0s) passed (5s) passed (3s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H12 test_coralbricks_provider_wire.py::test_async_anthropic_sdk_messages_go_natively_to_coralbricks_messages red: /model/new 500: deployment not live in the router (0s) passed (3s) passed (2s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H13 test_coralbricks_provider_wire.py::test_anthropic_sdk_messages_stream_is_forwarded_natively_and_billed_once red: /model/new 500: deployment not live in the router (0s) passed (3s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H14 test_coralbricks_provider_wire.py::test_async_anthropic_sdk_messages_stream_is_forwarded_natively_and_billed_once red: /model/new 500: deployment not live in the router (0s) passed (4s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
H15 test_coralbricks_provider_wire.py::test_raw_messages_request_lands_once red: /model/new 500: deployment not live in the router (0s) passed (6s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S1 test_coralbricks_provider_wire.py::test_provider_client_error_reaches_the_caller_after_one_attempt[/v1/chat/completions-401] red: /model/new 500: deployment not live in the router (0s) passed (5s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S2 test_coralbricks_provider_wire.py::test_provider_client_error_reaches_the_caller_after_one_attempt[/v1/chat/completions-404] red: /model/new 500: deployment not live in the router (0s) passed (2s) passed (3s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S5 test_coralbricks_provider_wire.py::test_provider_client_error_reaches_the_caller_after_one_attempt[/v1/responses-401] red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S6 test_coralbricks_provider_wire.py::test_provider_client_error_reaches_the_caller_after_one_attempt[/v1/messages-401] red: /model/new 500: deployment not live in the router (0s) passed (3s) passed (2s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S3 test_coralbricks_provider_wire.py::test_provider_server_error_is_retried_then_reaches_the_caller[429] red: /model/new 500: deployment not live in the router (0s) passed (11s) passed (10s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S4 test_coralbricks_provider_wire.py::test_provider_server_error_is_retried_then_reaches_the_caller[500] red: /model/new 500: deployment not live in the router (0s) passed (10s) passed (11s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S7 test_coralbricks_provider_wire.py::test_unreachable_api_base_reports_the_connection_error_and_leaves_the_proxy_serving red: /model/new 500: deployment not live in the router (4s) passed (3s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S8 test_coralbricks_provider_wire.py::test_deployment_without_a_key_and_no_env_key_fails_before_the_provider[missing] red: /model/new 500: deployment not live in the router (0s) passed (4s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S8 test_coralbricks_provider_wire.py::test_deployment_without_a_key_and_no_env_key_fails_before_the_provider[null] red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (2s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S8 test_coralbricks_provider_wire.py::test_deployment_without_a_key_and_no_env_key_fails_before_the_provider[empty] red: /model/new 500: deployment not live in the router (0s) passed (4s) passed (6s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S9 (control) test_coralbricks_provider_wire.py::test_hostile_model_field_is_rejected_without_crashing[int] PASS (0s) passed (0s) passed (0s), ctl lane passed (0s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S9 (control) test_coralbricks_provider_wire.py::test_hostile_model_field_is_rejected_without_crashing[list] PASS (0s) passed (0s) passed (0s), ctl lane passed (0s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
S9 (control) test_coralbricks_provider_wire.py::test_hostile_model_field_is_rejected_without_crashing[5kb] PASS (0s) passed (0s) passed (0s), ctl lane passed (0s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
E1 test_coralbricks_provider_wire.py::test_identical_uncached_chats_are_each_forwarded_and_billed red: /model/new 500: deployment not live in the router (0s) passed (4s) passed (3s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
E2 test_coralbricks_provider_wire.py::test_model_info_reports_the_coralbricks_cost_map_row red: no coralbricks row in the merge base cost map (0s) passed (4s) passed (4s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
E3 test_coralbricks_provider_wire.py::test_models_listing_and_model_group_info_carry_the_coralbricks_deployment red: no coralbricks row in the merge base cost map (0s) passed (0s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
E9 test_coralbricks_provider_wire.py::test_wildcard_deployment_lists_every_coralbricks_catalog_row red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
E4 test_coralbricks_provider_wire.py::test_health_probe_reaches_coralbricks_with_the_deployment_key red: /model/new 500: deployment not live in the router (0s) passed (0s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
E5 test_coralbricks_provider_wire.py::test_add_model_form_fields_name_the_coralbricks_credentials red: /public/providers/fields has no CORALBRICKS entry (0s) passed (0s) passed (0s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
E6 test_coralbricks_provider_wire.py::test_key_scoped_to_the_coralbricks_deployment_is_charged_its_spend red: /model/new 500: deployment not live in the router (0s) passed (3s) passed (2s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
E7 test_coralbricks_provider_wire.py::test_wildcard_coralbricks_deployment_routes_the_requested_model red: /model/new 500: deployment not live in the router (0s) passed (2s) passed (2s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
E8 test_coralbricks_provider_wire.py::test_fallback_from_a_refused_primary_lands_on_coralbricks red: /model/new 500: deployment not live in the router (0s) passed (4s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M1 test_coralbricks_provider_env_wire.py::test_config_model_without_credentials_uses_the_env_key_and_base[/v1/chat/completions-acompletion] red: config model never loaded: no healthy deployments (0s) passed (1s) passed (2s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M1 test_coralbricks_provider_env_wire.py::test_config_model_without_credentials_uses_the_env_key_and_base[/v1/responses-aresponses] red: config model never loaded: no healthy deployments (0s) passed (0s) passed (0s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M1 test_coralbricks_provider_env_wire.py::test_config_model_without_credentials_uses_the_env_key_and_base[/v1/messages-anthropic_messages] red: config model never loaded: no healthy deployments (0s) passed (1s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M2 test_coralbricks_provider_env_wire.py::test_explicit_credentials_beat_the_env_credentials red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (4s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M3 test_coralbricks_provider_env_wire.py::test_deployment_without_a_key_uses_the_env_key[missing] red: /model/new 500: deployment not live in the router (44s) passed (45s) passed (49s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M3 test_coralbricks_provider_env_wire.py::test_deployment_without_a_key_uses_the_env_key[null] red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (1s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M3 test_coralbricks_provider_env_wire.py::test_deployment_without_a_key_uses_the_env_key[empty] red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (4s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M4 test_coralbricks_provider_env_wire.py::test_deployment_without_a_base_uses_the_env_base[missing] red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (2s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M4 test_coralbricks_provider_env_wire.py::test_deployment_without_a_base_uses_the_env_base[null] red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (3s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M4 test_coralbricks_provider_env_wire.py::test_deployment_without_a_base_uses_the_env_base[empty] red: /model/new 500: deployment not live in the router (0s) passed (1s) passed (3s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
M1 (secrecy) test_coralbricks_provider_env_wire.py::test_env_credentials_are_not_echoed_by_model_info red: config model absent from /model/info (5s) passed (5s) passed (5s) tree 113351dece (pre-commit sync, product files identical to d5bbab5) PASS
C1 test_coralbricks_provider_chaos.py::test_provider_outage_mid_burst_fails_fast_reports_unhealthy_and_recovers red: /model/new 500: deployment not live in the router (1s) passed (11s) passed (23s) d5bbab5 (tree 17b0284724) PASS
C2 test_coralbricks_provider_chaos.py::test_slow_provider_streams_are_forwarded_once_under_concurrency red: /model/new 500: deployment not live in the router (1s) passed (9s) passed (16s) d5bbab5 (tree 17b0284724) PASS
C3 test_coralbricks_provider_chaos.py::test_worker_sigkill_mid_burst_leaves_the_sibling_serving_coralbricks red: burst never reached the wire: the owned proxy serves no coralbricks deployment (129s) passed (100s) passed (127s) d5bbab5 (tree 17b0284724) PASS
C4 test_coralbricks_provider_chaos.py::test_proxy_sigterm_mid_burst_then_reboot_serves_coralbricks_again red: burst never reached the wire: the owned proxy serves no coralbricks deployment (128s) passed (127s) passed (151s) d5bbab5 (tree 17b0284724) PASS
U1 tests/e2e/ui/tests/integrationCritical/addModelCoralBricks.spec.ts::the Add Model form offers CoralBricks and the deployment it creates serves and bills a chat (browser, each leg's own dashboard build) red: the CoralBricks option never appears in the provider picker (22.9s) passed (14.2s) passed (23.7s) d5bbab5 (tree 17b0284724) PASS

Type

🆕 New Feature

Caveats (if any)

Medium

  • A litellm.completion(stream=True) call with no stream_options logs usage 0/0 and response_cost 0.0 for CoralBricks in the SDK; the proxy is unaffected

    • CoralBricks sends a usage chunk only when the request carries stream_options: {"include_usage": true}, litellm injects that only for OpenAI-hosted api_bases (get_stream_options, litellm/llms/openai/openai.py:1233-1241), and CustomStreamWrapper stamps a zeroed Usage into the last chunk's hidden params when the vendor sent none, so the chunk builder's token-count fallback (calculate_usage counts only when the per-chunk counts are None) never runs. The same mocked usage-less stream through hosted_vllm/ with a custom base logs the same zeros, so this is the shared streaming wrapper's handling of every OpenAI-compatible provider whose vendor omits usage, not changed here and tracked on its own ticket
    • The proxy injects include_usage upstream for every stream (litellm/proxy/common_request_processing.py:542-554): on the tip a stream: true chat with no stream_options logged 33 prompt and 20 completion tokens and spend 0.00002697 (req_7146e491..., the vendor's usage with 33 cache-write tokens), so key, team, and budget spend are right. SDK callers pass stream_options: {"include_usage": true} to get the vendor's usage and cost

  • A reply whose reasoning_tokens exceed completion_tokens is overbilled by the difference

    • CoralBricks reports that on a max_tokens cut (60 completion, 61 reasoning, finish_reason length): the vendor charged 0.00029256 and litellm 0.00029696, one output token more; an earlier finished non-stream call showed the same two tokens over
    • The rule is the shared text_tokens_without_nested_reasoning, not changed here, so it is bounded at (reasoning minus completion) tokens per reply; a clamp there changes reasoning billing for every provider that nests reasoning, so the shared calculator fix is tracked on its own ticket

  • The four deprecated alias rows are listed as models of their own

    • A coralbricks/* deployment lists glm-5.3, glm-5.3-fp4, deepseek-v4.1-flash, and deepseek-v4.1-flash-fast-fp4 next to the two current names on /v1/models, /model/info, get_valid_models, and the Add Model picker (six entries for two models), and the wildcard health probe picks the fp4 slug as the cheapest row, which serves at the same price
    • No alias row carries a deprecation_date, so /model/deprecations cannot warn before CoralBricks drops an alias. The catalog publishes deprecatedSlugs with no end date (GET https://www.coralbricks.ai/api/public/models, 2026-10-12) and /model/deprecations counts the days until a real date, so none can be set truthfully; litellm lists every priced row of a provider (litellm/__init__.py:1016), so keeping the aliases billable (the P1 review thread) means listing them. Add the date once the vendor announces one
    • The alternative, pricing the aliases without listing them, needs a listing mechanism the cost map does not have (litellm/__init__.py:1016 builds coralbricks_models from every coralbricks row), so the rows list until the vendor stops serving the aliases; an operator who wants only the two current names configures them by name

Low

  • The /qa before leg ran at 29f48e6, the merge base when it ran, 273 commits behind the current merge base f516668; the audit's base rows L1 to L4 at f516668 show the same 400s
  • The openai/<model> plus api_base workaround from the earlier PR stays unpriced: get_model_info("glm-5.3-fp4", custom_llm_provider="openai") raises ModelNotMappedError on the tip because the rows are keyed coralbricks/, so those operators switch the prefix to coralbricks/ to get billing
  • The packaged dashboard build (litellm/proxy/_experimental/out) is not rebuilt here, so a pip install serves an Add Model picker without CoralBricks until the periodic "chore: update Next.js build artifacts" rebuild lands after the merge
    • The API side (/public/providers/fields, /model/new with custom_llm_provider: coralbricks) works on the tip; the browser cell runs against ui/litellm-dashboard/out built from the branch, the layout CI uses
  • input_cost_per_token is a rate no request pays: the vendor bills every uncached prompt token as a cache write (cost_details.input 0.0, cache_write_tokens = prompt_tokens on every call), so pre-call estimates and pick_cheapest_chat_models_from_llm_provider, which rank on input plus output cost, read 1.5x (glm) to 9x (deepseek) below what the spend log records; spend rows match the vendor's own cost
  • The rows' cache-write rates do not match the catalog field they cite one to one
    • GET /api/public/models reports cacheWritePerM 0.56 and 0.08 as a surcharge on top of input, so the rows' 1.68e-06 and 9e-08 are input plus surcharge; billed calls confirm the rows, and the next rate update should not "fix" them
  • The deprecated glm-5.3, glm-5.3-fp4, deepseek-v4.1-flash, and deepseek-v4.1-flash-fast-fp4 rows stay priced at the current names' rates because CoralBricks still serves all four (a chat on each answered 200 on 2026-10-12, billed at those rates by the vendor's own cost field); they can go once the vendor stops serving them
  • The issue also named glm-5.3-flash-fp4, but CoralBricks retired it on 2026-10-07 (every call now 404s and it is gone from /v1/models), so 18ffd76 drops its row and the tests use the current deepseek-v4.1-flash-fast
  • No CoralBricks logo, the dashboard shows the letter fallback
    • A logo is a separate asset change; the letter fallback is what the dashboard gives every provider without one
  • With neither an api_key on the deployment nor CORALBRICKS_API_KEY set, a request falls through to OPENAI_API_KEY, the way every OpenAI-compatible provider does (speech() resolves it explicitly, chat through the SDK's env read); not changed here
  • The Rust port's OPENAI_COMPATIBLE_PROVIDERS exception-mapping list (litellm-rust/crates/core-utils/src/exception_mapping_utils/original.rs) does not name coralbricks, nor cortecs, prism, sail, or edenai; it only picks the exception-mapping branch there, left alone
  • max_tokens is set to the 1,048,576 context length because the catalog publishes no output limit (max_output_tokens: null on 2026-10-06), so litellm.get_max_tokens quotes 1M output for these models and a request asking for max_tokens: 1048576 gets the vendor's 400
  • The reasoning-replay wire suite (tests/integration/providers/test_responses_minted_reasoning_replay_wire.py) lists nine JSON providers for native /v1/responses and not coralbricks; the PR's own wire suite covers chat, responses, and messages, so this is coverage parity only
  • /v1/model/info shows litellm_provider: null and no prices for a deployment that names the bare model with the CoralBricks api_base (provider inferred from the base), while chat through it bills at the cost-map rate
    • The cognition control behaves the same on both sides, so it is the shared info route's handling of every base-inferred deployment, not changed here
  • A stored credential whose api_key is the os.environ/CORALBRICKS_API_KEY literal is sent to CoralBricks verbatim and answers 401
    • Credentials store values, not references, for every provider; the same credential with the key itself works on the tip (the /live-pr-risk table above), not changed here

QA runbook

  • tests/e2e/ui/tests/integrationCritical/addModelCoralBricks.spec.ts::the Add Model form offers CoralBricks and the deployment it creates serves and bills a chat - the Add Model picker lists CoralBricks, the form submits a coralbricks deployment, and a chat on it is served and billed from the cost map
    • Boot the proxy with STORE_MODEL_IN_DB=True, a Postgres DATABASE_URL, and the integration upstream on INTEGRATION_UPSTREAM_URL (default http://127.0.0.1:8190), then sign in at http://localhost:4000/ui/login with admin and the master key
    • Open Models and Endpoints, click the Add Model tab, type CoralBricks into Provider and pick it, choose Custom Model Name (Enter below) under Select models, and type deepseek-v4.1-flash-fast
    • Set the public name to a fresh alias, fill API Base with <upstream>/v1 and API Key with cb_ui_integration_key, and click Add Model; expect the POST /model/new body to carry model_name = the alias, litellm_params.model = deepseek-v4.1-flash-fast, custom_llm_provider = coralbricks, that api_base and that api_key, and a "created successfully" toast
    • POST /v1/chat/completions with the alias and the master key until it answers 200, read its id, and GET /v2/model/info for the alias's input_cost_per_token and output_cost_per_token (both finite and above 0)
    • GET /spend/logs?request_id=<id> and expect custom_llm_provider = coralbricks, model = deepseek-v4.1-flash-fast, model_group = the alias, 20 prompt and 20 completion tokens (the upstream's scripted usage), and spend = 20 x input rate + 20 x output rate
    • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/cc554ddbbcad40cd834ab4a99655d5bb
Open in Devin Desktop: https://app.devin.ai/desktop/session/cc554ddbbcad40cd834ab4a99655d5bb?variant=devin
Requested by: @mubashir1osmani

Note

Low Risk
Changes are additive registry, pricing, and UI metadata with no modifications to existing provider behavior; risk is mainly misconfigured pricing or routing for the new provider only.

Overview
Adds CoralBricks (coralbricks) as a first-class OpenAI-compatible provider at https://inference.coralbricks.ai/v1, with env vars CORALBRICKS_API_KEY / CORALBRICKS_API_BASE and native support for /v1/chat/completions, /v1/responses, and /v1/messages.

Registers the provider across LiteLLM and proxy surfaces: openai_compatible_providers, endpoint support matrices, README provider table, proxy Add Model credential fields, and dashboard provider enum/placeholder (logoless fallback).

Adds cost-map rows for coralbricks/glm-5.3-fast and coralbricks/deepseek-v4.1-flash-fast (including cache-write pricing and free cached reads) so routing, x-litellm-response-cost, and spend logs can bill CoralBricks usage.

Ships broad coverage: unit tests for resolution and costing, integration wire/env/chaos suites, and a Playwright Add Model flow that creates a deployment and verifies chat + spend.

Reviewed by Cursor Bugbot for commit cb1a23d. Bugbot is set up for automated code reviews on this repo. Configure here.

Mubashir Osmani and others added 2 commits October 5, 2026 22:44
Mubashir Osmani @devin-ai-integration
feat(providers): add CoralBricks as a native provider
5763b4a 
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Mubashir Osmani @devin-ai-integration
test(coralbricks): harden cost map consistency tests
88814a9 
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot requested a review from a team October 5, 2026 22:49
@devin-ai-integration

devin-ai-integration Bot commented Oct 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

CLAassistant commented Oct 5, 2026 •
edited
Loading

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ Mubashir Osmani

Mubashir Osmani seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Oct 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the supported aliases have pricing again, and no new blocking issue was found.

Summary

Adds CoralBricks as an OpenAI-compatible provider with native chat, Responses, and Messages support.

  • CoralBricks models route through LiteLLM’s native provider paths.
  • CoralBricks model entries include token and cache prices.
  • Admins can choose CoralBricks in the Add Model form.

The latest change addresses the previous alias-pricing finding. No new actionable issues were found.

Reviews (11) · Last reviewed commit: "fix(coralbricks): keep pricing for the d..." · Reviewed by Greptile

greptile-apps[bot]

This comment was marked as resolved.

Sign in to view

@codspeed

codspeed Bot commented Oct 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

⚠️ 1 benchmark spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

✅ 31 untouched benchmarks

Comparing litellm_coralbricks_provider (d5bbab5) with main (249add6)

Open in CodSpeed

@codecov

codecov Bot commented Oct 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Mubashir Osmani and others added 2 commits October 6, 2026 00:22
Mubashir Osmani @devin-ai-integration
test(coralbricks): type cost helpers, read cost maps from disk, cover…
5f5db59 
… streaming spend

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Mubashir Osmani @devin-ai-integration
test(coralbricks): drop Final on loop-rebound names
f8568e3 
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

devin-ai-integration Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai

@mateo-berri

mateo-berri commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

bugbot run

cursor[bot]
cursor Bot reviewed Oct 6, 2026
View reviewed changes

@cursor cursor Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@mateo-berri mateo-berri mentioned this pull request Oct 6, 2026
Merged
@mateo-berri

mateo-berri commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

mateo-berri commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

bugbot run

cursor[bot]
cursor Bot reviewed Oct 6, 2026
View reviewed changes

@cursor cursor Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

greptile-apps[bot]

This comment was marked as resolved.

Sign in to view

@devin-ai-integration devin-ai-integration Bot mentioned this pull request Oct 8, 2026
Merged
Mubashir Osmani and others added 2 commits October 8, 2026 20:14
Mubashir Osmani
Merge remote-tracking branch 'origin/main' into litellm_coralbricks_p…
07dc9e8 
…rovider_wt

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

# Conflicts:
#	tests/e2e/ui/tests/integrationCritical/expected.json
#	ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx
Mubashir Osmani @devin-ai-integration
test(coralbricks): derive UI spend from model info, type browser rows…
430c736 
…, always release held upstream

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@mateo-berri

mateo-berri commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

@greptileai

1 similar comment
@mateo-berri

mateo-berri commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

@greptileai

Mubashir Osmani @devin-ai-integration
test(coralbricks): number forwarded chats with itertools.count instea…
35f9b6f 
…d of a mutable list

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@mateo-berri

mateo-berri commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

mateo-berri commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

bugbot run

cursor[bot]
cursor Bot reviewed Oct 8, 2026
View reviewed changes

@cursor cursor Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Mubashir Osmani @devin-ai-integration
fix(coralbricks): drop retired glm-5.3-flash-fp4 and test against dee…
18ffd76 
…pseek-v4.1-flash-fast-fp4

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@mateo-berri

mateo-berri commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

mateo-berri commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

bugbot run

cursor[bot]
cursor Bot reviewed Oct 8, 2026
View reviewed changes

@cursor cursor Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@mateo-berri
test(coralbricks): pin retries, await the respawned worker, read the …
1dfd035 
…UI rate from the catalog

The retry cells now set num_retries on the deployment, since the CI rig posts
router_settings.num_retries 0 after readiness and the router default of 2 no
longer applies there. The SIGKILL chaos cell waits for the respawned worker's
startup before the owned proxy is stopped and derives its timeout from the
harness budget. The Add Model spec waits for the catalog the form loads, reads
the expected rate from it, and finds the deployment through /v1/model/info,
which does not paginate
@mateo-berri

mateo-berri commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

@greptileai

greptile-apps[bot]

This comment was marked as resolved.

Sign in to view

@monotykamary

monotykamary commented Oct 9, 2026

Copy link
Copy Markdown

Thanks for picking this up. Two catalog changes since the rows were read on 2026-10-06, for whenever you next touch it:

  • Price change, 2026-10-08: DeepSeek V4.1 Flash input dropped from $0.30 to $0.01 per 1M. Everything else is unchanged.
  • Slugs: the current names are glm-5.3-fast and deepseek-v4.1-flash-fast. glm-5.3-fp4 and deepseek-v4.1-flash-fast-fp4 still resolve as deprecated aliases, so the PR works as is.
model input_cost_per_token cache_creation_input_token_cost cache_read_input_token_cost output_cost_per_token
coralbricks/glm-5.3-fast 1.12e-06 1.68e-06 0 4.4e-06
coralbricks/deepseek-v4.1-flash-fast 1e-08 9e-08 0 1.2e-06

Machine-readable source: https://www.coralbricks.ai/api/public/models. Its cacheWritePerM is the amount on top of input, so the full written-token price above is input + cacheWritePerM (1.12 + 0.56, 0.01 + 0.08).

@mateo-berri mateo-berri mentioned this pull request Oct 10, 2026
Merged
@mateo-berri
fix(coralbricks): price the current model names and the new DeepSeek …
cb1a23d 
…input rate

CoralBricks renamed glm-5.3-fp4 to glm-5.3-fast and deepseek-v4.1-flash-fast-fp4
to deepseek-v4.1-flash-fast (the fp4 names stay as deprecated aliases), and cut
DeepSeek V4.1 Flash input from $0.30 to $0.01 per 1M on 2026-10-08. Rates read
from https://www.coralbricks.ai/api/public/models on 2026-10-10; a live cache
write billed 6014 tokens at 9e-08 each, so the cache-write rate stays input
plus cacheWritePerM.

The test config builders now build new dicts instead of editing the parsed
YAML, and the cost-map test locals carry Final.
@mateo-berri

mateo-berri commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

@monotykamary thanks, cb1a23d switches to glm-5.3-fast and deepseek-v4.1-flash-fast with the $0.01 input rate, checked against the live catalog and a billed call

@mateo-berri

mateo-berri commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

mateo-berri commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

bugbot run

cursor[bot]
cursor Bot reviewed Oct 10, 2026
View reviewed changes

@cursor cursor Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

greptile-apps[bot]

This comment was marked as resolved.

Sign in to view

@mateo-berri

mateo-berri commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

@greptileai

devin-ai-integration[bot]
devin-ai-integration Bot commented Oct 12, 2026
View reviewed changes

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment thread model_prices_and_context_window.json
"max_input_tokens": 1048576,
"max_tokens": 1048576,
"mode": "chat",
"output_cost_per_token": 4.4e-06,

@devin-ai-integration devin-ai-integration Bot Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Reasoning-heavy calls inflate CoralBricks spend

When reasoning tokens exceed completion tokens, coralbricks bills every reasoning token at the new output rate. generic_cost_per_token adds that count without limiting it to reported output. Non-streaming spend and budget deductions exceed the vendor's cost

Learn more

CoralBricks can return a chat usage block where completion_tokens_details.reasoning_tokens exceeds completion_tokens. The new pricing rows send that usage through generic_cost_per_token, which bills the whole reasoning count separately at the output rate even when it exceeds the output total. The same calculation feeds the response-cost header and spend logs. Streaming usage follows a different normalization path, so identical requests can have different bills.

Example: A response reporting 10 completion tokens and 12 reasoning tokens bills 12 output tokens, rather than the vendor's 10, even before any text tokens are counted

Recommended fix: Normalize CoralBricks reasoning details against completion_tokens before pricing non-stream responses, without changing other providers' accounting. Add a regression with reasoning tokens greater than completion tokens and compare the billed amount to the vendor cost

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@mateo-berri mateo-berri Oct 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CoralBricks nests reasoning inside its output count (35 of 39 tokens in the proof). The arithmetic is the shared calculator every OpenAI-style provider uses

Comment thread litellm/types/utils.py
Comment thread ui/litellm-dashboard/src/components/provider_info_helpers.tsx
COHERE_CHAT: "cohere_chat",
COMETAPI: "cometapi",
COMPACTIFAI: "compactifai",
CORALBRICKS: "coralbricks",

@devin-ai-integration devin-ai-integration Bot Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Packaged Add Model picker lacks CoralBricks

Default proxy installs serve the tracked packaged UI, not the updated dashboard source. Its unrebuilt assets lack CORALBRICKS, so those installs still cannot pick CoralBricks in Add Model

Learn more

The dashboard's provider mapping is compiled into static assets. The default proxy serves packaged UI assets from litellm/proxy/_experimental/out. This PR changes the dashboard source but does not update those tracked assets. Consequently the updated backend provider metadata reaches an older Add Model UI that lacks this provider's mapping, and the UI does not offer the new provider in the default packaged installation. The browser test uses a freshly built dashboard instead of the packaged copy.

Example: Install the wheel built from this tree and open Add Model without setting LITELLM_UI_PATH; the bundled picker lacks CoralBricks even though /public/providers/fields lists it

Recommended fix: Include the regenerated packaged dashboard artifacts in the same release or ensure the release build regenerates them before publishing. Test the packaged UI path as well as the source-built dashboard

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@mateo-berri mateo-berri Oct 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The packaged dashboard is rebuilt by the weekly "update Next.js build artifacts" commit before each release. No provider PR commits it (tencent, Sail, TypeSafe)

@mateo-berri

mateo-berri commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

bugbot run

cursor[bot]
cursor Bot reviewed Oct 12, 2026
View reviewed changes

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit d5bbab5. Configure here.

mateo-berri
mateo-berri approved these changes Oct 12, 2026
View reviewed changes

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mateo-berri
mateo-berri merged commit ca4169d into main Oct 12, 2026
110 checks passed
@mateo-berri
mateo-berri deleted the litellm_coralbricks_provider branch October 12, 2026 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments
@cursor cursor[bot] cursor[bot] left review comments
@mateo-berri mateo-berri mateo-berri approved these changes
@ryan-crabbe-berri ryan-crabbe-berri Awaiting requested review from ryan-crabbe-berri ryan-crabbe-berri is a code owner
@kerrylu-berri kerrylu-berri Awaiting requested review from kerrylu-berri kerrylu-berri is a code owner

Assignees

@mubashir1osmani mubashir1osmani

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Add CoralBricks as a native provider

4 participants

@CLAassistant @mateo-berri @monotykamary @mubashir1osmani