Repository navigation
fix(bedrock_mantle): send max_completion_tokens for gpt-5.6 and gpt-6 chat completions - #43982
Conversation
… chat completions
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Merging this PR will not alter performance
|
…e_gpt_max_completion_tokens_s2
GitHub Actions
Error
|
bugbot run |
GitHub Actions
Error
|
bugbot run |
GitHub Actions
Error
GitHub Actions
Error
|
bugbot run |
GitHub Actions
Error
GitHub Actions
Waiting
|
bugbot run |
…e_gpt_max_completion_tokens # Conflicts: # tests/e2e/models.py # tests/integration/_support/process.py
…e_gpt_max_completion_tokens
GitHub Actions
Error
| result = client.proxy.transport.send( | ||
| "/chat/completions", | ||
| headers=client.proxy.transport.bearer(resources.key()), | ||
| json=ChatBody(model=model, messages=_prompt(), max_tokens=OUTPUT_LIMIT), |
There was a problem hiding this comment.
All three new chat tests use transport.send or chat_stream instead of the suite's sdk fixture. The repository requires LLM calls in llm_translation/ to use the real provider SDKs so tests check the responses and exceptions customers receive. Switch all three calls to sdk before merging to meet this requirement.
File Used: tests/e2e/AGENTS.md (source)
|
|
||
|
|
||
| class TestBedrockMantleChatOutputLimit: | ||
| def test_client_max_tokens_returns_completion(self, client: PassthroughClient, resources: ResourceManager) -> None: |
There was a problem hiding this comment.
None of the three new tests declares Subject. The repository requires every test under tests/e2e/, except the load suite, to declare at least its domain. Add @meta(Subject(...)) to each test before merging so their results identify what they test and meet this requirement.
File Used: tests/e2e/AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
… fixture and tag their subject The three live tests in tests/e2e/llm_translation/test_bedrock_mantle_output_limit_e2e.py called the proxy through a hand-built client; they now use the suite's sdk fixture (SdkClients.openai with NO_PROXY_CACHE) and declare @meta(Subject(...)) for the bedrock_mantle provider, the chat completions route, and the stream and non-stream modes, the way tests/e2e/AGENTS.md asks. The streamed tool-call assertion reads the tool name and arguments from the deltas and validates the arguments as JSON
GitHub Actions
Error
…e_gpt_max_completion_tokens
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 1e8767c. Configure here.
This branch was successfully deployed
1 participant
TLDR
Problem this solves:
max_tokensfor gpt-5.6 and gpt-6 with a 400bedrock_mantlegpt-5.6 deployment sentmax_tokensfor any output limitmax_completion_tokenswas rewritten tomax_tokensand failed the same wayHow it solves it:
max_completion_tokensmax_tokensis carried over asmax_completion_tokens, value unchangedmax_tokensas beforeDecisions: the rule is keyed on the model name (the same GPT reasoning-series check the native OpenAI config uses), not on the
openai/v1base path that #43581 keys its Mantle override on, because the customer's deployment runs on a/v1base and Mantle acceptsmax_completion_tokensfor every GPT model tested while gpt-5.4, gpt-oss and gemma-4 accept both names. When a request carries bothmax_tokensandmax_completion_tokens, the explicitmax_completion_tokenswins andmax_tokensis dropped (the alternative, sending the smaller of the two, was not taken because the request already named the parameter this family accepts). Values are never scaled or clampedUser Flow
Before: a platform engineer whose apps call a Bedrock Mantle gpt-5.6 deployment through chat completions gets a 400 the moment any output limit is set, on the deployment or on the request
bedrock_mantle/openai.gpt-5.6-terra,custom_llm_provider: bedrock_mantle,model_info.mode: chat, andmax_tokens: 1000in its paramsstream: true,stream_options: {"include_usage": true}, and a function tool, no limit of its ownUnsupported parameter: 'max_tokens' is not supported with this model.instead of a stream, and the same 400 comes back when the deployment has no limit and the app sendsmax_tokens: 1000ormax_completion_tokens: 1000withstream: falseAfter: the same deployment and the same requests return completions, with the limit still applied
max_tokens: 1000in its paramsstream: true,stream_options: {"include_usage": true}, and a function toolmax_tokens: 1000ormax_completion_tokens: 1000from the app withstream: falsereturns HTTP 200 with a completion whose usage stays under 1000 output tokensRelevant issues
Related: #44307 (merged) gave the Bedrock runtime chat path the same
max_tokenstomax_completion_tokensrename for gpt-5.6+; this PR is the standalone fix for the Mantle chat path. #43581 carried both inside a larger stacked change and was closed unmergedLinear ticket
Resolves LIT-9089
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Last updated: 1e8767c. This tip merges main (705bd84) into the branch so the lint workflow's OpenAPI docs step, which checks out the PR head, finds
scripts/openapi_docs_gate.py(added to main today in #45843); the merge is clean and changes nothing underlitellm/llms/bedrock_mantle, and 008996b before it changed only the e2e test file (the three tests now drive the proxy through the suite's OpenAI SDK fixture and declare their Subject metadata, closing the two Greptile threads). The product behavior is still the one 62a88f7 shipped. QA ran on d4d2694 (two 2-worker proxies, the Before leg at the merge base 988cc7d) and carries; the e2e step below was re-run from this merged tree against the same head proxy, and the 63 unit tests pass at this tip. /live-pr-risk ran at d4d2694 against 988cc7d and the audit matrix (round 2 of 3) at d4d2694 against 988cc7d, both below; /live-pr-risk also drove its 71 cells on the head merged with main at ff60d27, and of the 11 main commits since then only c1e3d07 toucheslitellm/llms/bedrock, in the guardrail, not the Mantle chat path. d4d2694 merged main (988cc7d) into the branch, resolving the conflicts intests/e2e/models.py(both sides' deployment fields kept) andtests/integration/_support/process.py(main's version taken whole, so the wire test's chaos budget now reads main'sgraceful_stop_seconds()), and typed the override's signature and its one parent call so the lint gates stay at main's counts. Greptile: 4/5 at d4d2694 (Reviews (6), 2026-10-10 22:04Z), with its two P2 threads on the e2e test closed by 008996b (outdated, left for the bot to resolve). Bare@greptileaiat 008996b (2026-10-10 23:20Z) and twice at 1e8767c (00:12Z and 17:57Z on 10-11) drew no answer, and no PR in this repo has had a Greptile review since that 22:04Z edit (new PRs opened 16:39Z on 10-11 have none, eight triggers by five people today are unanswered), so the silence is repo-wide and is recorded here, never counted as a review of the two later commits (tests-only and a main merge)Shared setup: each leg is a LiteLLM proxy booted from the named commit with
--num_workers 2, Postgres withSTORE_MODEL_IN_DB=True, and a real Bedrock Mantle bearer token for us-east-1 (real calls, real spend). Both legs register the same deployments through the proxy, then run the same requests in the same ordergpt-5.6-terra-chat-deploymaxis the same deployment with"max_tokens": 1000added tolitellm_params(the customer's shape). The controlsgpt-5.4-chat(bedrock_mantle/openai.gpt-5.4) andgpt-oss-120b-chat(bedrock_mantle/openai.gpt-oss-120b) are registered the same wayBefore (988cc7d)
OpenAI SDK: deployment max_tokens, stream + stream_options + tools (the customer's shape)
OpenAI SDK: client max_tokens, then client max_completion_tokens
curl /v1/chat/completions: deployment limit, stream
curl /v1/messages: max_tokens 1000
curl /v1/responses: max_output_tokens 1000
Controls: gpt-5.4 and gpt-oss-120b with max_tokens 50 and max_completion_tokens 50
https://bedrock-mantle.us-east-1.api.aws/openai/v1/chat/completionswith the same bearer token,model: openai.gpt-oss-120b,max_tokens: 50, no proxy in between, answered the identical body{"error":{"code":"validation_error","message":"model \openai.gpt-oss-120b` isn't supported on this route", ...}}` with HTTP 400, so the gpt-oss control is refused by Mantle itself today)e2e test file against this proxy
The test file does not exist on this tree, so the head tree's
tests/e2e/llm_translation/test_bedrock_mantle_output_limit_e2e.pywas copied into this scratch directory only (nothing copied into the tree) and run with the tree's.venvand the tree's own harness (tests/e2e/conftest.pyandllm_translation/conftest.pyregistered with-p,LITELLM_PROXY_URLandLITELLM_MASTER_KEYpointed at this proxy). The PR also editstests/e2e/models.py(adds"chat"toModelMode,supported_endpointstoModelInfoBody, andmax_tokens,max_completion_tokens,custom_llm_provider,reasoning_efforttoLiteLLMParamsBody), and this tree's harness lacks those, so every test fails inside the harness's pydantic validation ofModelInfoBody(mode="chat", ...)before a single request reaches the proxy. This step therefore measures the harness gap, not the proxyAfter (d4d2694)
OpenAI SDK: deployment max_tokens, stream + stream_options + tools (the customer's shape)
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:52737/v1", api_key=LITELLM_KEY)
stream = client.chat.completions.create(
model="gpt-5.6-terra-chat-deploymax",
messages=[{"role": "user", "content": "What is the weather in Paris?"}],
stream=True, stream_options={"include_usage": True},
tools=[{"type": "function", "function": {"name": "get_weather",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}])
run as: cd /private/tmp/claude-501/-Users-mateo-Development-litellm/89961715-811c-4684-82f6-82c7d78ce259/scratchpad/lit-9089/wt && .venv/bin/python -P /private/tmp/claude-501/-Users-mateo-Development-litellm/89961715-811c-4684-82f6-82c7d78ce259/scratchpad/lit-9089/qa-after/sdk_steps.py 1 (LITELLM_KEY read from $LITELLM_MASTER_KEY)
OpenAI SDK: client max_tokens, then client max_completion_tokens
client.chat.completions.create(model="gpt-5.6-terra-chat", stream=False, max_tokens=1000,
messages=[{"role": "user", "content": "Say hello in one short sentence."}])
client.chat.completions.create(model="gpt-5.6-terra-chat", stream=False, max_completion_tokens=1000,
messages=[{"role": "user", "content": "Say hello in one short sentence."}])
run as: cd /private/tmp/claude-501/-Users-mateo-Development-litellm/89961715-811c-4684-82f6-82c7d78ce259/scratchpad/lit-9089/wt && .venv/bin/python -P /private/tmp/claude-501/-Users-mateo-Development-litellm/89961715-811c-4684-82f6-82c7d78ce259/scratchpad/lit-9089/qa-after/sdk_steps.py 2
curl /v1/chat/completions: deployment limit, stream
curl -s -N -w "\nHTTP %{http_code}\n" http://127.0.0.1:52737/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json"
-d '{"model":"gpt-5.6-terra-chat-deploymax","messages":[{"role":"user","content":"What is the weather in Paris?"}],"stream":true,"stream_options":{"include_usage":true},"tools":[{"type":"function","function":{"name":"get_weather","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]}'
curl /v1/messages: max_tokens 1000
curl -s -w "\nHTTP %{http_code}\n" http://127.0.0.1:52737/v1/messages -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json"
-d '{"model":"gpt-5.6-terra-chat","max_tokens":1000,"messages":[{"role":"user","content":"Say hello in one short sentence."}]}'
curl /v1/responses: max_output_tokens 1000
curl -s -w "\nHTTP %{http_code}\n" http://127.0.0.1:52737/v1/responses -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json"
-d '{"model":"gpt-5.6-terra-chat","input":"Say hello in one short sentence.","max_output_tokens":1000}'
(from the JSON above: status=completed, max_output_tokens=1000, usage.output_tokens=6, output text='Hello!')
Controls: gpt-5.4 and gpt-oss-120b with max_tokens 50 and max_completion_tokens 50
for m in gpt-5.4-chat gpt-oss-120b-chat; do for p in max_tokens max_completion_tokens; do
curl -s -w "\nHTTP %{http_code}\n" http://127.0.0.1:52737/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json"
-d "{"model":"$m","messages":[{"role":"user","content":"Say hello in one short sentence."}],"stream":false,"$p":50}"
done; done
Diagnostic, outside the brief's steps, to tell a vendor change from a proxy one (same bearer token, no proxy; the token is read from the rig env and never shown):
3. ```bash
for p in max_tokens max_completion_tokens; do
curl -s -w "\nHTTP %{http_code}\n" https://bedrock-mantle.us-east-1.api.aws/openai/v1/chat/completions -H "Authorization: Bearer $AWS_BEARER_TOKEN_BEDROCK" -H "Content-Type: application/json"
-d "{"model":"openai.gpt-oss-120b","messages":[{"role":"user","content":"Say hello in one short sentence."}],"$p":50}"
done
curl -s -o /dev/null -w "HTTP %{http_code}\n" https://bedrock-mantle.us-east-1.api.aws/openai/v1/chat/completions -H "Authorization: Bearer $AWS_BEARER_TOKEN_BEDROCK" -H "Content-Type: application/json"
-d '{"model":"openai.gpt-5.4","messages":[{"role":"user","content":"Say hello in one short sentence."}],"max_tokens":50}'
The PR's diff against its merge base
988cc7dc94toucheslitellm/llms/bedrock_mantle/chat/transformation.py(+24) and tests only, with no line naming gpt-oss or a routee2e test file against this proxy
cd /private/tmp/claude-501/-Users-mateo-Development-litellm/89961715-811c-4684-82f6-82c7d78ce259/scratchpad/lit-9089/wt && LITELLM_PROXY_URL=http://127.0.0.1:45463 LITELLM_MASTER_KEY=$LITELLM_MASTER_KEY .venv/bin/python -m pytest tests/e2e/llm_translation/test_bedrock_mantle_output_limit_e2e.py -q -rA -p no:cacheprovider
Not verified: the customer's GovCloud
/v1route (the Medium caveat below); gpt-oss-120b, which Mantle refuses on this route today, on both legs and when called directlySeen by the legs, none of it caused by this PR:
/v1/responseswithmax_output_tokensis 200 on both legs/model/newechoes encryptedlitellm_paramsvalues in its 200 body/live-pr-risk: d4d2694 against merge base 988cc7d
Verdict: no dependent path breaks. The only difference on the wire between base and head is
max_tokensbecomingmax_completion_tokenson Mantle chat completions for GPT reasoning-series names (thegpt-5andgpt-6markers), which turns every base-sideUnsupported parameter: 'max_tokens'400 on gpt-5.6 and gpt-6 chat-mode deployments into a 200 and leaves every other request identical in key set, values, header key set, status, and count, plus one narrow bridged edge underallowed_openai_paramsnamed below. The head merged locally withmainat ff60d27b60 (merge commit a5e62b2136, a clean merge with no change underlitellm/llms/bedrock_mantle) answers all 71 cells with the same status as the head;mainhas since gained 906f927 (auth sentinel) and 0126c8d (sagemaker request body), neither touching this pathMethod: an A/B rig on the GCE box (
lit9089ab), the merge base and the head as two proxies on the same 30-deployment config and Postgres layout, each with a forwarding recorder in front of the real us-east-1 Mantle host for its/openai/v1base and another for its/v1base, and one in front of the real Langfuse host, so every outbound request was diffed base vs head on key set, values, header key set, status, and count. Three passes. First 71 cells (chat withmax_tokens,max_completion_tokens, both, none,null, a duplicated raw key, streaming, streaming with tools under a deployment-levelmax_tokens: 64, a region-prefixed model string, a/v1base,allowed_openai_params,additional_drop_params,use_chat_completions_api, a deployment withoutapi_base, gpt-5.4, gpt-6, gpt-5.5 and gpt-oss controls,/v1/messagesstream and non-stream,/v1/responsesnative and emulated over chat, a Langfuse-traced call read back through the Langfuse API, key generate, use, model-not-allowed, info, delete and use-after-delete, the spend row of every call,/health,/health/readiness,/health/liveliness,/v1/models,max_tokens: 50against everybedrock_mantle/openai.gpt*cost-map id, andmax_tokensas a string, a list, 0, -5, 40.0 and 10^9). Then 36 cells for the dependents the graph walk added (every cost-map id driven on the native chat path with an explicitcustom_llm_provider, a bridged gpt-5.5 withallowed_openai_params: ["max_tokens"]and both client keys,/v1/completions, a virtual key withtpm_limit: 400so the v3 rate limiter injects its own cap,/utils/transform_requestwith an uncataloguedopenai.gpt-5.7-preview, Langfuse readback of a call that succeeds on both sides). Then the 71 cells again on the head merged withmain. Credentials were the us-east-1 Mantle bearer token and the Langfuse keys from the main checkout's.env; nothing mocked, no fixtures, every upstream call billedBreaking: none. Nothing went from 200 to a failure, and no status moved on any gpt-oss, gpt-5.4, gpt-5.5, Responses, Messages, key, spend, health, or models cell. The only status moves are 400 to 200 on the gpt-5.6 and gpt-6 chat-mode deployments, each with
max_tokenson the base wire andmax_completion_tokenswith the same value on the head wire: chat withmax_tokens, with both keys, streaming (4 chunks,stop), streaming with tools under the deployment'smax_tokens: 64(6 tool deltas,tool_calls), the region-prefixed id, the deployment with noapi_base, the duplicated raw key,/v1/messagesstream and non-stream,/v1/responsesemulated over chat withmax_output_tokens,/v1/completionswithmax_tokens: 32(anOpenAIException400 on base), the key-scoped call, the Langfuse-traced call, thetpm_limit: 400key's call with no client limit (the limiter's injectedmax_tokens: 100arrives asmax_completion_tokens: 100), and the native sweep of gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra, gpt-6-astra, gpt-6-luna, gpt-6-sol and gpt-6.1-sol. gpt-5.4 takes either name on Mantle (max_tokens: 50200 on base,max_completion_tokens: 50200 on head). The gpt-oss request bodies on/v1are identical multisets on both sides (the per-leg difference is only which random prompt the health check drew), and theirstopvslengthfinish reasons at the same limit are upstream nondeterminism, since gpt-oss spends 27 to 50 tokens reasoning on a one-word prompt. Attempt counts match on every cell (57 and 57 upstream calls in the extra pass, one per request, no retries), the header key set isaccept, authorization, content-type, user-agenton every recorded request on both sides, and the Langfuse export (POST /api/public/otel/v1/traces) answered 200 to every batch on both sides with the same header key setBackward incompatible: four contract changes, none of which fails a caller that works today. First, Mantle's validation error for a malformed limit on a gpt-5.6 chat deployment now quotes
max_completion_tokens(Invalid type for 'max_completion_tokens',integer below minimum value,exceeds model maximum (1050000), the same 400 on both sides; the PR body's Low caveat, shipped as is). Second,additional_drop_params: ["max_completion_tokens"]with a clientmax_tokensno longer removes the limit on gpt-5.x, because the drop runs before the rename: that cell went from the 400 to a 200 with the limit on the wire (Low caveat, shipped as is). Third,allowed_openai_params: ["max_tokens"]re-adds the raw key after mapping, so the head sends bothmax_tokensandmax_completion_tokens. On a native chat deployment that keeps the 400 it had on the base (the client cell, the both-keys cell, and the/healthprobe all 400 on both sides; Low caveat, shipped as is). On a bridged deployment with a client sending both keys the Responses bridge's last-key-wins loop now takes themax_tokensvalue: for gpt-5.5 withmax_tokens: 50andmax_completion_tokens: 200the base sentmax_output_tokens: 200and the head sentmax_output_tokens: 50, both 200 to the client (cellbridged55.allowed.both, confirmed by the time-ordered recorder entries; without the allowlist both sides send 200). This edge needs an operator to allowlist a parameter that was never rejected and a client to send both keys at once, so it ships with the caveat above. Fourth, the operator-visible flips that follow from the fix:/healthnow reports the gpt-5.6 and gpt-6 chat-mode deployments healthy (base 10 healthy and 20 unhealthy on the 30-deployment config, head 26 and 4, the four being theallowed_openai_paramsdeployment, the/v1-base deployment that Mantle rejects on both sides, and the two ids this account cannot call), the spend rows for those calls flip from failure rows (spend 0, 14 total tokens) to success rows (spend above 0, 18 tokens), and/utils/transform_requestfor an uncataloguedbedrock_mantle/openai.gpt-5.7-previewreturnsmax_completion_tokens: 64where it returnedmax_tokens: 64(gpt-oss keepsmax_tokens: 64on both sides)Regression risk, dependents this run could not drive: MCP sampling (
sampling_handler.pysetsmax_tokenson the sampled chat call),modify_paramsinutils.py, anduser_max_tokens(dead code, only ever set toNone) inject or carrymax_tokensthe same way the health check and the v3 rate limiter do, and those two are wire-verified (the probe's 16 and the limiter's 100 arrive renamed), so the untested ones share the class and the direction, a 400 on base becoming a 200. The customer's GovCloud/v1route (our GovCloud SigV4 credentials get 401access_deniedfor gpt-5.6 and 200 for gpt-oss, so the rename on a/v1base is covered only by the us-east-1/v1cell, where Mantle rejects gpt-5.6 with its own validation error on both sides and the body key sets match).openai.gpt-daybreak-blue-5.6-sol, whose name carries neithergpt-5norgpt-6, keepsmax_tokenson the head (recorded on both sides, 404 "does not exist" from this account), which matters only if that model exists elsewhere, rejectsmax_tokens, and runs on the native chat path rather than its cost-mapresponsesmode. Langfuse'smodelParameterskeeps the client's pre-mappingmax_tokens: 40on both sides (vendor-side readback on the base for gpt-oss and on the head for gpt-oss and gpt-5.6), so a dashboard reading that field sees no change and a reader expecting the wire key there finds it on neither sideDependency graph:
BedrockMantleChatConfig.map_openai_paramshas one dispatch, thecustom_llm_provider == "bedrock_mantle"branch ofget_optional_paramsthroughProviderConfigManager.get_bedrock_mantle_config(verified live through/v1/chat/completions,/v1/completions,/v1/messagesvia the completion adapter,/v1/responsesemulated over chat, and/utils/transform_request), no subclass, no other override, no other caller passingreplace_max_completion_tokens_with_max_tokens, andbedrock_mantle_chat_configroutes Claude ids to the Claude config before this code (unchanged). Callers that feed it a limit, with status: the client (verified live), the deployment'slitellm_params.max_tokens(verified, 64 renamed),/health(verified, 16 renamed), the v3 rate limiter cap (verified, 100 renamed),allowed_openai_paramsandadditional_drop_params(verified, above), MCP sampling andmodify_params(untested, same class),user_max_tokens(dead). Paths the override does not reach, verified unchanged on the wire: the native/v1/responsespath (max_output_tokens: 40on both sides), the chat-to-Responses bridge for every cost-map id inresponsesmode (gpt-5.4, gpt-5.5, 5.6-luna, 5.6-terra, 6-astra, 6-luna, 6-sol and 6.1-sol all 200 withmax_output_tokens: 50on both sides, with the oneallowed_openai_paramsedge above), gpt-oss on/v1and/openai/v1(identical bodies), OTel and Langfuse (pre-mapping params, Langfuse verified at the vendor). Claude ids on Mantle are a separate config and were not driven. Tests: the unit cases intests/unit/llms/bedrock_mantle/, the integration wire test intests/integration/providers/, and the e2e test intests/e2e/llm_translation/cover the client path (tested); the proxy-side injectors,/v1/completions, the bridgedallowed_openai_paramsedge, and/utils/transform_requesthave no test naming them and were covered live here (verified-live); MCP sampling andmodify_paramsare untested and unreachable in this rig without an MCP server or amodify_paramshook (unreachable)Not verified: the customer's GovCloud
/v1route for gpt-5.6 (401access_deniedon our GovCloud account with both SigV4 key sets; only gpt-oss is entitled there);openai.gpt-5.6-cyberandopenai.gpt-daybreak-blue-5.6-sol, which us-east-1 answers with 404 "does not exist" on both sides (cyber is renamed on the head, daybreak is not); MCP sampling,modify_params, anduser_max_tokens, reasoned above and not driven; Claude ids on Mantle, which this PR does not touchAudit matrix
Audit round 2 of 3, at the PR tip d4d2694 against the merge base 988cc7d (round 1 ran at 62a88f7 against 0f6a06a and is replaced by this run). The integration cells this PR adds in
tests/integration/providers/test_bedrock_mantle_output_limit_wire.pyran throughtests/integration/run.py providerson a rig shaped like CircleCI's, built on the GCE box byremote_rig.py: an owned Postgres database, an owned Redis and the scripted upstream fromtests/integration/_support/upstream.pyplaying Mantle per leg (it rejectsmax_tokensfor gpt-5.6 and gpt-6 with the vendor's recorded 400 and acceptsmax_completion_tokens; gpt-5.4 and gpt-oss take both), and a two-worker proxy per leg booted from its own worktree, nothing mocked inside it. The product diff against the merge base islitellm/llms/bedrock_mantle/chat/transformation.pyalone, so every cell and the chaos step reran on both legs at the tip. The base run and the two head runs ran side by side as lanes with the same seeds, 1 pytest worker each (CircleCI's setting for the group), and the head runs have the same collected and passed selections, no skips, no retries; the two owned-proxy chaos cells (C4, C5) ran inside the full runs. Cells drive the proxy through the OpenAI SDK, the Anthropic SDK, and raw httpx and assert the caller's response, the exact body the scripted peer received (limit key and value, model, path, attempt count), and the spend row by response idThe 36 cells, base 988cc7d against head d4d2694 twice
test_openai_sdk_max_tokens_reaches_mantle_as_max_completion_tokenstest_async_openai_sdk_stream_keeps_max_completion_tokenstest_stream_with_tools_carries_the_deployment_limit_as_max_completion_tokenstest_anthropic_sdk_messages_max_tokens_reaches_mantle_as_max_completion_tokenstest_async_anthropic_sdk_messages_stream_reaches_mantle_as_max_completion_tokenstest_responses_max_output_tokens_is_forwarded_natively_on_both_legstest_region_prefixed_deployment_on_a_v1_base_sends_the_bare_model_with_max_completion_tokenstest_gpt6_deployment_sends_max_completion_tokenstest_gpt_oss_control_keeps_the_openai_compatible_max_tokens[max_tokens]test_gpt_oss_control_keeps_the_openai_compatible_max_tokens[max_completion_tokens]test_gpt54_control_joins_the_renamed_seriestest_explicit_max_completion_tokens_wins_over_max_tokenstest_request_limit_wins_over_the_deployment_limittest_identical_requests_with_no_cache_are_attempted_and_logged_once_eachtest_hostile_max_tokens_is_rejected_by_mantle_under_its_new_name[string]test_hostile_max_tokens_is_rejected_by_mantle_under_its_new_name[list]test_hostile_max_tokens_is_rejected_by_mantle_under_its_new_name[empty]test_hostile_max_tokens_is_rejected_by_mantle_under_its_new_name[5kb]test_duplicate_max_tokens_keys_in_raw_json_send_the_last_value_renamedtest_out_of_range_max_tokens_is_rejected_by_mantle_under_its_new_name[0]test_out_of_range_max_tokens_is_rejected_by_mantle_under_its_new_name[-5]test_wrong_deployment_bearer_reaches_the_caller_as_401_after_one_attempttest_unknown_mantle_model_reaches_the_caller_as_404_after_one_attempttest_allowed_openai_params_max_tokens_is_sent_verbatim_beside_the_renamed_limittest_additional_drop_params_max_completion_tokens_drops_before_the_renametest_unrelated_key_scoped_to_the_control_deployment_keeps_workingtest_unauthenticated_request_never_reaches_mantletest_null_or_missing_max_tokens_sends_no_limit[null]test_null_or_missing_max_tokens_sends_no_limit[missing]test_identical_requests_without_no_cache_serve_the_second_from_cachetest_model_update_of_the_deployment_limit_stays_renamed_and_convergestest_concurrent_chat_and_messages_burst_is_renamed_and_logged_once_eachtest_peer_503s_reach_their_callers_once_while_the_rest_are_renamed_and_servedtest_slow_mantle_streams_are_forwarded_once_renamedtest_worker_sigkill_mid_burst_leaves_the_sibling_renaming_the_yaml_limittest_proxy_restart_mid_burst_loses_only_in_flight_requests_and_renames_after_restartThe rows only real Mantle can prove ran once per leg as live cells (
live_cells.py, included this time) through a head and a merge-base proxy on the same config, with the request, the proxy response, the vendor response id, and the spend row saved per row in audit-live-artifacts.tar.gz under<leg>/<row>.json, credentials redacted. L1 runs the customer's deployment shape against us-east-1 with the account's Mantle bearer; L2 to L5 run SigV4 deployments against the us-gov-west-1/v1base with the QA account's IAM keysThe 5 live cells, base 988cc7d against head d4d2694
/openai/v1, bearer token/v1, SigV4 (the customer's route)/v1, SigV4/v1, SigV4/v1, SigV4, controlaccess_denied ... contact AWS Saleson both legs, and the GovCloud QA account's Bedrock catalog in us-gov-west-1 and us-gov-east-1 lists only the two gpt-oss models. H7 proves the wire shape of that deployment (bare model on the/v1base,max_completion_tokens), L1 proves real Mantle acceptsmax_completion_tokensfor gpt-5.6-terra, and L5 proves the SigV4 GovCloud route serves on both legsType
🐛 Bug Fix
Caveats (if any)
High
allowed_openai_params: ["max_tokens"], called by a client that sends bothmax_tokens: 50andmax_completion_tokens: 200, now forwardsmax_output_tokens: 50where main forwarded 200 (both HTTP 200). The allow-list re-adds the rawmax_tokensafter the rename moved the explicit value tomax_completion_tokens, and the bridge takes the last key it sees, so the TLDR's "explicitmax_completion_tokenswins" rule flips on this one path. Decision: ship with it. It takes an operator allow-listing a parameter Mantle never rejected plus a client sending two different limits in one request, the limit applied is still one the client sent, and the fix belongs in the bridge (prefermax_completion_tokenswhen both keys arrive), which every provider shares and which would re-run every live gate here for no change on the customer's path, so it is filed as a follow-upMedium
bedrock_mantle/us-gov-west-1/openai.gpt-5.6-terraon the us-gov-west-1/v1base with SigV4, was not observed end to end on either leg: no account we hold is entitled to gpt-5.6-terra in GovCloud (Mantle answers 401access_denied ... contact AWS Sales, and the GovCloud account's Bedrock catalog in us-gov-west-1 and us-gov-east-1 lists only the gpt-oss models), so the audit's rows L2 to L4 stay BLOCKED BY HUMAN. The rename is wire-verified for that exact deployment shape against the scripted peer (H7), real Mantle acceptsmax_completion_tokensfor gpt-5.6-terra in us-east-1 (L1), and the SigV4 GovCloud route itself serves on both legs (L5); the rename does not depend on the base path or the regionLow
max_tokensbefore, now sendmax_completion_tokensas well, because the rename keys on the GPT-5 and GPT-6 series rather than on the names that refusemax_tokens. That is the decision recorded in the TLDR: every gpt-5.x and gpt-6 model this account can call in us-east-1 answered 200 with the new key on the head leg, gpt-5.4 and gpt-5.5 answered 200 with either name on both legs, and a list of the names that refusemax_tokenswould re-break each new GPT release until someone added it. The GovCloud gpt-5.4 rows carry the same credential gap as the Medium above (every gpt-5.x name is 401 there with every key we hold)max_tokens(a string, a list, 0, a negative number) now saysInvalid type for 'max_completion_tokens'orInvalid 'max_completion_tokens', naming the parameter the proxy actually sent. Rewriting a provider's error text back to the caller's spelling would hide what went over the wire, so this stays as isallowed_openai_params: ["max_tokens"]on a gpt-5.x Mantle chat deployment re-adds the rawmax_tokensafter mapping and the upstream 400 stays, exactly as before this PR. That allow-list exists for parameters a provider config does not support, andmax_tokensis supported here, so nobody needs it on these deployments; special-casing it is net-negativeadditional_drop_params: ["max_completion_tokens"]together with a clientmax_tokensno longer strips the limit, because dropping runs before the rename; the request now succeeds with the limit applied where it returned the 400 before. The native OpenAI GPT-5 path has the same order (gpt-5.4with that drop list and a clientmax_tokens: 50also sendsmax_completion_tokens: 50), an operator who wants no limit at all dropsmax_tokens, so re-running the drop after mapping for one provider is net-negativewith_max_completion_tokensdoes for the Bedrock runtime chat path (feat(bedrock): serve gpt-5.6+ chat completions natively by default, with chat_completions/ opt-in for gpt-oss and grok #44307): the same one-line comprehension, keyed by the same series helper. Folding the Mantle override onto that helper changes no behavior and would re-run every live gate at this tip, so it is left for a follow-uplitellm/llms/bedrock_mantleor in the Mantle branch ofget_optional_params, so the proof carries under /qa's no-op rule and the tip itself never served a requestllm.request.max_tokensfrom the mappedmax_tokenskey only, so these gpt-5.x Mantle chat calls log no request limit on the span, the same gap every native OpenAI gpt-5 call has today; readingmax_completion_tokenstoo is a two-integration follow-up, not this PRQA runbook
max_tokenslitellm_paramsmodel: bedrock_mantle/openai.gpt-5.6-terra,custom_llm_provider: bedrock_mantle,api_key: os.environ/AWS_BEARER_TOKEN_BEDROCK,api_base: https://bedrock-mantle.us-east-1.api.aws/openai/v1,reasoning_effort: none;model_infomode: chat,supported_endpoints: ["/v1/chat/completions", "/v1/responses"](needs STORE_MODEL_IN_DB=True and a Mantle bearer token for us-east-1)max_tokens: 1000max_tokensmax_completion_tokensmax_completion_tokens: 1000max_tokensno longer breaks a streamed tool-call request that carries no limit of its ownmax_tokens: 1000added tolitellm_paramsstream: true,stream_options: {"include_usage": true}, oneget_weatherfunction tool taking acity, andtool_choice: required[DONE]with no error chunk, whose tool-call deltas assemble to aget_weathercall with a non-emptycityargumentFinal Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
d4d2694 passes /live-pr-risk
Note
Medium Risk
Changes the upstream request shape for Mantle chat on GPT reasoning-series models; scoped to one provider config but affects all output-limit paths (client, deployment, messages adapter).
Overview
Fixes Bedrock Mantle chat completions for GPT reasoning-series models (gpt-5.x, gpt-6) by overriding
BedrockMantleChatConfig.map_openai_params: after normal OpenAI mapping,max_tokensis renamed tomax_completion_tokensand the old key is dropped, using the sameis_gpt_reasoning_series_namecheck as native OpenAI. Non–reasoning Mantle models (e.g. gpt-oss) keepmax_tokensbehavior unchanged.Adds unit coverage via
get_optional_params, a live e2e suite for client/deployment limits and streamed tool calls, and a large integration wire matrix (SDKs,/v1/messages, concurrency, chaos) that asserts the upstream body. Test harness tweaks: configurable proxy ready/stop deadlines and richer e2e model registration types (max_tokens,chatmode,supported_endpoints).Reviewed by Cursor Bugbot for commit 76b96e5. Bugbot is set up for automated code reviews on this repo. Configure here.