Repository navigation
fix(gateway): default an Anthropic client timeout so the non-streaming pre-flight guard does not 502 #799
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(gateway): default an Anthropic client timeout so the non-streaming pre-flight guard does not 502 #799
Changes from 1 commit
619397f
a10c37e
4bdef8c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -237,6 +237,10 @@ | |
| "Top up the provider account, or route this model to a provider that has credit." | ||
| ) | ||
| PROVIDER_RATE_LIMITED_DETAIL = "The provider rate-limited this request" | ||
| PROVIDER_NONSTREAMING_MAX_TOKENS_DETAIL = ( | ||
| "max_tokens is too large for a non-streaming request to this provider; lower max_tokens " | ||
| "or set stream: true" | ||
| ) | ||
| ALL_PROVIDERS_FAILED_DETAIL = "All upstream providers failed" | ||
| ALL_PROVIDERS_TIMED_OUT_DETAIL = "All upstream providers timed out" | ||
| SANDBOX_NOT_CONFIGURED_DETAIL = ( | ||
|
|
@@ -343,6 +347,20 @@ def _unsupported_feature_detail(exc: BaseException) -> str: | |
| return _caller_fault_detail(exc, PROVIDER_BAD_REQUEST_DETAIL) | ||
|
|
||
|
|
||
| # The anthropic SDK's non-streaming pre-flight guard message (otari#533); status-less, | ||
| # so this classifier has to name it explicitly or it falls through to a generic 502. | ||
| # Matched as a substring: the SDK message also links a docs anchor that has moved. | ||
| _ANTHROPIC_NONSTREAMING_TIMEOUT_GUARD_MESSAGE = "Streaming is required for operations that may take longer" | ||
|
|
||
|
|
||
| def _is_anthropic_nonstreaming_timeout_guard(exc: BaseException) -> bool: | ||
| """True when a status-less ``ValueError`` is the anthropic SDK's own pre-flight guard.""" | ||
| return any( | ||
| isinstance(candidate, ValueError) and _ANTHROPIC_NONSTREAMING_TIMEOUT_GUARD_MESSAGE in str(candidate) | ||
| for candidate in upstream_exception_chain(exc) | ||
| ) | ||
|
|
||
|
|
||
| def _redacted_caller_fault_detail(message: str, fallback: str) -> str: | ||
| """Return redacted explanatory text, or a fixed detail when none remains.""" | ||
| redacted = redact_upstream_message(message) | ||
|
|
@@ -403,6 +421,10 @@ def classify_provider_error(exc: BaseException) -> ProviderErrorMapping | None: | |
| # whole signal here, and its message names the unsupported feature. | ||
| if _is_unsupported_feature_error(exc): | ||
| return ProviderErrorMapping(status.HTTP_400_BAD_REQUEST, _unsupported_feature_detail(exc)) | ||
| # Defense in depth for otari#533: an operator-supplied timeout equal to the | ||
| # SDK's own default re-arms this same status-less guard. | ||
| if _is_anthropic_nonstreaming_timeout_guard(exc): | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This branch is unreachable through either client-construction path once the timeout is injected, and it couples the classifier to an English sentence in a vendored SDK. Meanwhile any-llm 1.25's Classifying a status-less |
||
| return ProviderErrorMapping(status.HTTP_400_BAD_REQUEST, PROVIDER_NONSTREAMING_MAX_TOKENS_DETAIL) | ||
| if status_code is None: | ||
| return None | ||
| # Account billing exhaustion, which several providers report as a 400/422 | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -91,6 +91,28 @@ | |
| # and no key takes the keyless placeholder, so it never reaches a caller alone. | ||
| _NON_CREDENTIAL_KWARGS = frozenset({"client_args"}) | ||
|
|
||
| # The anthropic SDK's non-streaming pre-flight guard fires for a large max_tokens | ||
| # only while the client's timeout is untouched (otari#533); this is also the | ||
| # guard's own fallback value, so setting it explicitly disables the guard alone. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nothing ties assert anthropic.DEFAULT_TIMEOUT.read == ANTHROPIC_DEFAULT_TIMEOUT_SECONDS
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed on the drift risk, though the exact assertion changed shape along with the fix. Since with_anthropic_default_timeout now injects httpx.Timeout(601.0, connect=5.0) instead of a flat 600.0 (see the reply on the finding above), the constant that has to track the SDK is the connect value, not the read value. tests/unit/test_provider_instances.py now has test_anthropic_default_timeout_constants_track_the_sdks_own_default, asserting anthropic.DEFAULT_TIMEOUT.connect == ANTHROPIC_DEFAULT_CONNECT_TIMEOUT_SECONDS and that the httpx.Timeout we build stays unequal to anthropic.DEFAULT_TIMEOUT. Either one drifting fails the test. Pushed at 4bdef8c. |
||
| ANTHROPIC_DEFAULT_TIMEOUT_SECONDS = 600.0 | ||
|
|
||
|
|
||
| def with_anthropic_default_timeout( | ||
| provider: LLMProvider, client_args: dict[str, Any] | None | ||
| ) -> dict[str, Any] | None: | ||
| """Ensure an Anthropic ``client_args`` carries an explicit ``timeout``. | ||
|
|
||
| A no-op for every other provider. An operator's own ``timeout`` (set via | ||
| ``client_args`` on the provider config or a platform attempt's | ||
| ``extra_params``) is left untouched; this only fills the gap when nothing | ||
| was configured. | ||
| """ | ||
| if provider != LLMProvider.ANTHROPIC: | ||
| return client_args | ||
| merged = dict(client_args) if client_args else {} | ||
| merged.setdefault("timeout", ANTHROPIC_DEFAULT_TIMEOUT_SECONDS) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A flat float raises the Anthropic client's connect timeout from 5s to 600s on every request, including the streaming ones the guard never touched. Verified against the installed SDK: The guard's trigger is A per-request Fix: inject
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You are right on every part of this, and I reproduced it before changing anything. In a Docker container with anthropic 0.88.0 installed, AsyncAnthropic(timeout=600.0)._client.timeout is Timeout(timeout=600.0), and its connect attribute reads 600.0, not 5.0. anthropic._constants.DEFAULT_TIMEOUT is Timeout(connect=5.0, read=600, write=600, pool=600), and the guard in messages.py checks self._client.timeout == DEFAULT_TIMEOUT exactly as you quoted. I also patched httpx.AsyncClient.send and ran your three cases through the real SDK: a flat 600.0 skips the guard and reaches the transport layer with connect=600, httpx.Timeout(600.0, connect=5.0) compares equal to DEFAULT_TIMEOUT and puts the guard back, and httpx.Timeout(601.0, connect=5.0) skips the guard and reaches the transport with connect still at 5.0. I went with your client-level alternative rather than moving this to a completion kwarg. with_anthropic_default_timeout now injects httpx.Timeout(601.0, connect=5.0) instead of a flat float. That function already backs both call sites, get_provider_kwargs and build_attempt_client_args, so the platform-attempt path gets the fix too without touching responses.py or the completion-kwarg assembly. A new test in test_hybrid_client_args_routing.py drives an attempt through the real any-llm/anthropic stack and captures the constructed httpx.AsyncClient's own timeout to confirm connect lands at 5.0 end to end, not just in our own dict construction. Pushed at 4bdef8c. |
||
| return merged | ||
|
|
||
|
|
||
| def _kwargs_carry_a_credential(kwargs: dict[str, Any]) -> bool: | ||
| """Whether anything in a resolved provider's kwargs could authenticate a call. | ||
|
|
@@ -227,6 +249,9 @@ def get_provider_kwargs( | |
| if placeholder is not None: | ||
| kwargs["api_key"] = placeholder | ||
|
|
||
| if provider == LLMProvider.ANTHROPIC: | ||
| kwargs["client_args"] = with_anthropic_default_timeout(provider, kwargs.get("client_args")) | ||
|
|
||
| return kwargs | ||
|
|
||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,12 +17,14 @@ | |
| _AMBIENT_CREDENTIAL_PROVIDERS, | ||
| _KEYLESS_PLACEHOLDER_API_KEY, | ||
| _KEYLESS_SELF_HOSTED_PROVIDERS, | ||
| ANTHROPIC_DEFAULT_TIMEOUT_SECONDS, | ||
| get_provider_kwargs, | ||
| keyless_placeholder_api_key, | ||
| normalize_pricing_key, | ||
| provider_key, | ||
| resolve_provider_selector, | ||
| split_selector, | ||
| with_anthropic_default_timeout, | ||
| ) | ||
|
|
||
| # --------------------------------------------------------------------------- | ||
|
|
@@ -405,6 +407,47 @@ def test_get_provider_kwargs_strips_provider_type_and_models() -> None: | |
| assert kwargs == {"api_base": "http://x/v1", "api_key": "k"} | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # get_provider_kwargs / with_anthropic_default_timeout (otari#533) | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def test_get_provider_kwargs_fills_a_default_anthropic_timeout() -> None: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
def test_a_default_anthropic_timeout_does_not_look_like_a_credential() -> None:
kwargs = get_provider_kwargs(GatewayConfig(), LLMProvider.ANTHROPIC)
assert credential_ladder_exhausted(LLMProvider.ANTHROPIC, kwargs) is True
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added, close to what you proposed. tests/unit/test_provider_instances.py now has test_a_default_anthropic_timeout_does_not_look_like_a_credential, asserting that get_provider_kwargs(GatewayConfig(), LLMProvider.ANTHROPIC) still reports credential_ladder_exhausted() as True. It pins exactly the thing you flagged: a default-filled client_args stays in _NON_CREDENTIAL_KWARGS and never gets read as a credential by itself. Pushed at 4bdef8c. |
||
| config = GatewayConfig(providers={"anthropic": {"api_key": "sk-test"}}) | ||
| kwargs = get_provider_kwargs(config, LLMProvider.ANTHROPIC, instance="anthropic") | ||
| assert kwargs["client_args"] == {"timeout": ANTHROPIC_DEFAULT_TIMEOUT_SECONDS} | ||
|
|
||
|
|
||
| def test_get_provider_kwargs_preserves_an_operator_configured_anthropic_timeout() -> None: | ||
| config = GatewayConfig(providers={"anthropic": {"api_key": "sk-test", "client_args": {"timeout": 900}}}) | ||
| kwargs = get_provider_kwargs(config, LLMProvider.ANTHROPIC, instance="anthropic") | ||
| assert kwargs["client_args"] == {"timeout": 900} | ||
|
|
||
|
|
||
| def test_get_provider_kwargs_leaves_other_providers_client_args_untouched() -> None: | ||
| config = GatewayConfig(providers={"openai": {"api_key": "sk-test"}}) | ||
| kwargs = get_provider_kwargs(config, LLMProvider.OPENAI, instance="openai") | ||
| assert "client_args" not in kwargs | ||
|
|
||
|
|
||
| def test_with_anthropic_default_timeout_is_a_no_op_for_other_providers() -> None: | ||
| assert with_anthropic_default_timeout(LLMProvider.OPENAI, None) is None | ||
| assert with_anthropic_default_timeout(LLMProvider.OPENAI, {"timeout": 5}) == {"timeout": 5} | ||
|
|
||
|
|
||
| def test_with_anthropic_default_timeout_fills_a_missing_default() -> None: | ||
| assert with_anthropic_default_timeout(LLMProvider.ANTHROPIC, None) == { | ||
| "timeout": ANTHROPIC_DEFAULT_TIMEOUT_SECONDS | ||
| } | ||
| assert with_anthropic_default_timeout(LLMProvider.ANTHROPIC, {}) == { | ||
| "timeout": ANTHROPIC_DEFAULT_TIMEOUT_SECONDS | ||
| } | ||
|
|
||
|
|
||
| def test_with_anthropic_default_timeout_never_overrides_an_explicit_value() -> None: | ||
| assert with_anthropic_default_timeout(LLMProvider.ANTHROPIC, {"timeout": 42}) == {"timeout": 42} | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # get_provider_kwargs: keyless custom-endpoint placeholder (mozilla-ai/otari#421) | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The scenario named here cannot occur. A YAML
client_args: {timeout: 600}becomeshttpx.Timeout(600.0), which is not equal toDEFAULT_TIMEOUT(connect=5.0), so an operator-supplied timeout equal to the SDK's default is not expressible from config and never re-arms the guard.Keep the branch: it still catches any-llm's
InvalidRequestErrortranslation and any future Anthropic client built withoutwith_anthropic_default_timeout. Reword the justification, or the next reader will trust it.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right about the mechanism, and I rewrote the comment rather than the branch. An operator's client_args.timeout can only ever arrive as a flat number from config, and a flat number sets every httpx.Timeout dimension including connect, so it can equal DEFAULT_TIMEOUT only by also setting connect to 5.0, at which point read/write/pool are 5.0 too, not 600. Config alone can't reproduce the real default, so the old comment's stated scenario doesn't happen.
The branch itself stays, and the comment now says what it actually catches: a client built without with_anthropic_default_timeout, i.e. some future call path that skips get_provider_kwargs and build_attempt_client_args. That client is left at the SDK's real default and can still trip the guard for a large max_tokens.
Pushed at 4bdef8c.