Repository navigation
fix(bedrock): backport #41870, the GPT-6 reasoning gate fix, and the Python 3.13 image pin to stable/1.98.x for v1.98.1 #42002
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
7996007
c32db1b
9f48cca
1af2cad
0c0dda9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,7 @@ | |
|
|
||
| import copy | ||
| import json | ||
| import re | ||
| import time | ||
| import types | ||
| from typing import Final, Literal, cast, overload | ||
|
|
@@ -92,6 +93,7 @@ | |
| "bash_", | ||
| "text_editor_", | ||
| ] | ||
| BEDROCK_OPENAI_COMPAT_MIN_MAX_TOKENS: Final = 16 | ||
|
|
||
| # Beta header patterns that are not supported by Bedrock Converse API | ||
| # These will be filtered out to prevent errors | ||
|
|
@@ -280,6 +282,10 @@ def _validate_request_metadata(self, metadata: dict) -> None: | |
| llm_provider="bedrock", | ||
| ) | ||
|
|
||
| @staticmethod | ||
| def _requires_min_max_tokens(model: str) -> bool: | ||
| return re.search(r"openai\.gpt-\d|xai\.grok-", model) is not None | ||
|
|
||
| def _is_nova_2_model(self, model: str) -> bool: | ||
| """ | ||
| Check if the model is a Nova 2 model that supports reasoningConfig. | ||
|
|
@@ -858,7 +864,11 @@ def map_openai_params( | |
| is_thinking_enabled=is_thinking_enabled, | ||
| ) | ||
| if param == "max_tokens" or param == "max_completion_tokens": | ||
| optional_params["maxTokens"] = value | ||
| optional_params["maxTokens"] = ( | ||
| max(value, BEDROCK_OPENAI_COMPAT_MIN_MAX_TOKENS) | ||
| if isinstance(value, int) and self._requires_min_max_tokens(model) | ||
| else value | ||
|
Comment on lines
878
to
+882
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Backport of #41870, where this was withdrawn. The regex clamps ARNs embedding the model id; opaque application profiles carry none, so get_model_info cannot resolve them
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That’s fair. Opaque application-profile ARNs contain no model identifier, and |
||
| ) | ||
| if param == "stream": | ||
| optional_params["stream"] = value | ||
| if param == "stop": | ||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This name-pattern check violates the directive to store model-specific flags in
model_prices_and_context_window.jsonand read them throughget_model_info; fix before mergingRule Used: What: Do not hardcode model-specific flags in the codebase. Instead, put them in model_prices_and_context_window.json and then read them in via get_model_info Why: Prevents need for users to upgrade litellm each time a new model supports this featu... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same shape as main's merged #41870. The pattern matches model families, not individual models, so new GPT or Grok releases need no cost-map entry
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That’s a fair distinction. I rechecked #41870:
_requires_min_max_tokensis not identifying a finite catalog of model capabilities; it encodes Bedrock Converse’s shared minimum-token contract for theopenai.gpt-*andxai.grok-*model families. Usingget_model_infowould require adding or updating every future model ID and would make the fix stale, whereas family matching intentionally covers new releases and mirrors the merged implementation on main. I’m withdrawing this finding; no change is needed.