Repository navigation
test(e2e): move live-provider legacy tests into tests/e2e - #44120
Merged
Merged
Conversation
Port legacy tests that exercise real providers into the tests/e2e suites that own them, using the harness (/model/new plus deferred cleanup) and asserting on what the caller receives. Delete legacy tests already covered at equal or stronger strength by e2e, integration or unit tests, and drop the now empty ocr_testing CircleCI job
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 01:30 — with 
GitHub Actions
Error
GitHub Actions
Error
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 01:30 — with
GitHub Actions
Error
GitHub Actions
Error
Contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Assert the SSE error frame a client actually receives when a post_call guardrail blocks a stream, and require a tool call for every requested city before checking the answer. Restore the OCR matrix and its CircleCI job, the Claude Agent SDK streaming test, and test_async_create_batch, since their SDK-level and callback assertions have no equivalent in tests/e2e
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 01:52 — with
GitHub Actions
Error
GitHub Actions
Error
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 01:52 — with
GitHub Actions
Error
GitHub Actions
Error
Contributor
Author
A post_call block before the first chunk reaches the client as HTTP 400 with either a JSON error body or a single SSE error frame, depending on whether the block surfaced as an exception or an error chunk. Assert the policy message is present and the blocked output is absent in both
Contributor
Author
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 02:02 — with
GitHub Actions
Error
GitHub Actions
Error
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 02:02 — with
GitHub Actions
Error
GitHub Actions
Error
Contributor
Comments Outside DiffThese findings could not be posted inline.
|
The e2e realtime tests go through the proxy and the remaining SDK tests either mock the upstream or assert less, so keep the direct litellm._arealtime tests with and without intent, and TestOpenAIRealtime::test_realtime_connection, in place
Contributor
Author
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 04:38 — with
GitHub Actions
Error
GitHub Actions
Error
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 04:39 — with
GitHub Actions
Error
GitHub Actions
Error
The direct SDK realtime tests now fail on a refused connection instead of skipping. The with-intent test asserts OpenAI rejects the exact intent value sent, which only happens when the intent is forwarded. The Nova /v1/messages stream test asserts stream structure, stop reason and usage instead of model wording
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 04:45 — with
GitHub Actions
Error
GitHub Actions
Error
Contributor
Author
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 04:46 — with
GitHub Actions
Error
GitHub Actions
Error
…ive rejection Assert litellm._arealtime passes the intent query param into the OpenAI realtime websocket URL, which is the behavior LiteLLM owns, and drop the live test that depended on OpenAI's rejection wording
Contributor
Author
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 04:52 — with
GitHub Actions
Error
GitHub Actions
Error
yuneng-berri
had a problem deploying
to
e2e-changed
October 2, 2026 04:53 — with
GitHub Actions
Error
GitHub Actions
Error
Contributor
Author
yucheng-berri
approved these changes
Oct 2, 2026
# Conflicts: # tests/test_openai_endpoints.py
yuneng-berri
requested a deployment
to
e2e-changed
October 2, 2026 06:36 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
yuneng-berri
requested a deployment
to
e2e-changed
October 2, 2026 06:36 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
5 tasks done
yuneng-berri
added a commit
that referenced
this pull request
Oct 2, 2026
* test(proxy): stop the proxy_server app fixture leaking LITELLM_LOG The session app fixture set LITELLM_LOG=ERROR with os.environ.setdefault and never removed it, so later tests on the same xdist worker inherited it. test_drop_params_env_var spawns a subprocess with os.environ and lost the warning it asserts on. Scope the variable to the import with a MonkeyPatch context * test(secret-detection): give the hand-built redaction request an ASGI path Since #43975 _read_request_body checks the route path via request.scope, and a scope without path raised KeyError that was swallowed into an empty body, so chat_completion failed with a missing messages parameter. Real ASGI scopes always carry path * test(integration): isolate litellm callback lists per sdk test usage-based-routing-v2 Routers register their selector in litellm.callbacks and nothing removes it, not even Router.reset(). The counter TTL and Redis service metrics tests left their selectors behind, and the next usage routing test ran their pre-call checks against its own rpm=1 deployments, raising "Deployment over defined rpm limit". An autouse fixture now gives each sdk test copies of the callback lists and restores the originals afterwards * test(integration): keep the owner-lookup fault proxy off the shared read replica The owned proxy points DATABASE_URL at a scratch database but inherited DATABASE_URL_READ_REPLICA from the replica job, so auth read the shared database and rejected the freshly created key with token_not_found_in_db. Drop the replica variable like the other scratch-database owned proxies * test(integration): request every seeded key in the team owner breakdown The aggregated team activity endpoint now caps breakdown.api_keys at the top 100 keys by default (#43398), so the 300 seeded keys came back as 100 rows. The test guarantees each key is reported with its own owner, so ask for an api_key_limit that covers all seeded keys * test(integration): give every owned Redis its own port in the redis-cache container On CircleCI every owned Redis ran on the fixed port 16379 inside the shared redis-cache container. When an earlier server still held that port, the new one failed to bind, readiness pinged the old server, the pidfile read failed and cleanup then reported "Owned Redis still serves after shutdown" Reserve an ephemeral port for the docker-exec path the same way the local binary path already does, and refuse to start when something already serves the chosen port so the failure names the real cause * test(e2e): skip the Vertex Mistral partner case the e2e project cannot reach The e2e Vertex project gets a 404 publisher model not found for vertex_ai/mistral-small-2503, so the case can only fail * test(e2e): skip the Vertex gpt-oss partner case the e2e project never serves vertex_ai/openai/gpt-oss-120b-maas has hit a 60s read timeout with no response headers on every run in the e2e Vertex project since the case was ported, and no other Vertex partner chat model passes there to switch to * test(e2e): check only stored message content for a leaked card number The Presidio spend-log check ran the card-number pattern over the whole serialized response, so a Luhn-valid usage.cost float (0.0003466000000000001) failed the streaming /v1/messages case although the stored content was <CREDIT_CARD>. The check now reads the content and text strings of the stored response, which is where a raw card would land, and still requires the placeholder there * test(e2e): assert the proxy decodes token-array embeddings for titan The port in #44120 carried over a legacy SDK-direct test that expected Bedrock to reject token ids with a 400. Through the proxy, /embeddings decodes token arrays to text for providers that cannot embed tokens, so titan answers 200. The test now sends a token array and its decoded sentence and requires the two vectors to match, which fails if the proxy stops decoding or decodes with the wrong tokenizer * test(e2e): run the Bedrock extended-thinking round trip on a model that honors enabled thinking us.anthropic.claude-sonnet-5-5 is adaptive-only, so litellm sends thinking.type=enabled with a 1024 budget as adaptive with low effort, and Bedrock returned no reasoning blocks on 5 of 5 identical Converse calls (boto3 direct agreed). us.anthropic.claude-sonnet-4-6 accepts the legacy shape verbatim and returned reasoning on 5 of 5. The non-thinking Bedrock case stays on sonnet-5-5 * test(proxy): stop unit modules forcing DEBUG logging into the event-loop lag tests Five tests/unit modules set verbose_proxy_logger to DEBUG at import, so every xdist worker that collected them logged the 2.4MB pass-through response from a worker thread, and secret redaction of that line held the GIL for ~0.8s+ inside the timed window. The lag tests now pin the LiteLLM loggers to WARNING and freeze gc while timing, and the module-level DEBUG overrides are removed * test(e2e): cite the tokenizer and date behind the titan token-array fixture * test(e2e): let migration seed replicas finish their request-log indexes before cloning Since #43948 a serving proxy builds the two LiteLLM_SpendLogs indexes on a background thread after it reports ready. The seed fixtures stopped the replica at readiness, so every cloned legacy database lacked an index no real deployment would be missing, and the v2 baseline diff refused it. Seeds now wait until both indexes exist and are valid in the database's schema * test(passthrough): give the pass-through MockRequest an httpx URL and ASGI scope #43626 made get_request_route read request.scope during pass-through kwarg setup; the MockRequest in tests/unit/passthrough had neither a scope nor a URL object, so both stream-param tests raised before reaching the code they check. Mirrors the repair #43626 made to the tests/pass_through_unit_tests fake * test(integration): ignore foreign allow_all_keys MCP servers in the access matrix tool list test_toolset_gateway_url_serves_a_team_granted_toolset_to_a_key_without_its_own_grant (#43908) registers an allow_all_keys server on the shared gateway, and allow_all_keys servers are listed to every key by design, so a matrix case running on another xdist worker at the same time saw its tools. The matrix now drops tools of allow_all_keys servers it did not create, read from LiteLLM_MCPServerTable before and after listing, and still compares everything else exactly
yuneng-berri
added a commit
that referenced
this pull request
Oct 2, 2026
…) (#44254) * test(proxy): stop the proxy_server app fixture leaking LITELLM_LOG The session app fixture set LITELLM_LOG=ERROR with os.environ.setdefault and never removed it, so later tests on the same xdist worker inherited it. test_drop_params_env_var spawns a subprocess with os.environ and lost the warning it asserts on. Scope the variable to the import with a MonkeyPatch context * test(secret-detection): give the hand-built redaction request an ASGI path Since #43975 _read_request_body checks the route path via request.scope, and a scope without path raised KeyError that was swallowed into an empty body, so chat_completion failed with a missing messages parameter. Real ASGI scopes always carry path * test(integration): isolate litellm callback lists per sdk test usage-based-routing-v2 Routers register their selector in litellm.callbacks and nothing removes it, not even Router.reset(). The counter TTL and Redis service metrics tests left their selectors behind, and the next usage routing test ran their pre-call checks against its own rpm=1 deployments, raising "Deployment over defined rpm limit". An autouse fixture now gives each sdk test copies of the callback lists and restores the originals afterwards * test(integration): keep the owner-lookup fault proxy off the shared read replica The owned proxy points DATABASE_URL at a scratch database but inherited DATABASE_URL_READ_REPLICA from the replica job, so auth read the shared database and rejected the freshly created key with token_not_found_in_db. Drop the replica variable like the other scratch-database owned proxies * test(integration): request every seeded key in the team owner breakdown The aggregated team activity endpoint now caps breakdown.api_keys at the top 100 keys by default (#43398), so the 300 seeded keys came back as 100 rows. The test guarantees each key is reported with its own owner, so ask for an api_key_limit that covers all seeded keys * test(integration): give every owned Redis its own port in the redis-cache container On CircleCI every owned Redis ran on the fixed port 16379 inside the shared redis-cache container. When an earlier server still held that port, the new one failed to bind, readiness pinged the old server, the pidfile read failed and cleanup then reported "Owned Redis still serves after shutdown" Reserve an ephemeral port for the docker-exec path the same way the local binary path already does, and refuse to start when something already serves the chosen port so the failure names the real cause * test(e2e): skip the Vertex Mistral partner case the e2e project cannot reach The e2e Vertex project gets a 404 publisher model not found for vertex_ai/mistral-small-2503, so the case can only fail * test(e2e): skip the Vertex gpt-oss partner case the e2e project never serves vertex_ai/openai/gpt-oss-120b-maas has hit a 60s read timeout with no response headers on every run in the e2e Vertex project since the case was ported, and no other Vertex partner chat model passes there to switch to * test(e2e): check only stored message content for a leaked card number The Presidio spend-log check ran the card-number pattern over the whole serialized response, so a Luhn-valid usage.cost float (0.0003466000000000001) failed the streaming /v1/messages case although the stored content was <CREDIT_CARD>. The check now reads the content and text strings of the stored response, which is where a raw card would land, and still requires the placeholder there * test(e2e): assert the proxy decodes token-array embeddings for titan The port in #44120 carried over a legacy SDK-direct test that expected Bedrock to reject token ids with a 400. Through the proxy, /embeddings decodes token arrays to text for providers that cannot embed tokens, so titan answers 200. The test now sends a token array and its decoded sentence and requires the two vectors to match, which fails if the proxy stops decoding or decodes with the wrong tokenizer * test(e2e): run the Bedrock extended-thinking round trip on a model that honors enabled thinking us.anthropic.claude-sonnet-5-5 is adaptive-only, so litellm sends thinking.type=enabled with a 1024 budget as adaptive with low effort, and Bedrock returned no reasoning blocks on 5 of 5 identical Converse calls (boto3 direct agreed). us.anthropic.claude-sonnet-4-6 accepts the legacy shape verbatim and returned reasoning on 5 of 5. The non-thinking Bedrock case stays on sonnet-5-5 * test(proxy): stop unit modules forcing DEBUG logging into the event-loop lag tests Five tests/unit modules set verbose_proxy_logger to DEBUG at import, so every xdist worker that collected them logged the 2.4MB pass-through response from a worker thread, and secret redaction of that line held the GIL for ~0.8s+ inside the timed window. The lag tests now pin the LiteLLM loggers to WARNING and freeze gc while timing, and the module-level DEBUG overrides are removed * test(e2e): cite the tokenizer and date behind the titan token-array fixture * test(e2e): let migration seed replicas finish their request-log indexes before cloning Since #43948 a serving proxy builds the two LiteLLM_SpendLogs indexes on a background thread after it reports ready. The seed fixtures stopped the replica at readiness, so every cloned legacy database lacked an index no real deployment would be missing, and the v2 baseline diff refused it. Seeds now wait until both indexes exist and are valid in the database's schema * test(passthrough): give the pass-through MockRequest an httpx URL and ASGI scope #43626 made get_request_route read request.scope during pass-through kwarg setup; the MockRequest in tests/unit/passthrough had neither a scope nor a URL object, so both stream-param tests raised before reaching the code they check. Mirrors the repair #43626 made to the tests/pass_through_unit_tests fake * test(integration): ignore foreign allow_all_keys MCP servers in the access matrix tool list test_toolset_gateway_url_serves_a_team_granted_toolset_to_a_key_without_its_own_grant (#43908) registers an allow_all_keys server on the shared gateway, and allow_all_keys servers are listed to every key by design, so a matrix case running on another xdist worker at the same time saw its tools. The matrix now drops tools of allow_all_keys servers it did not create, read from LiteLLM_MCPServerTable before and after listing, and still compares everything else exactly (cherry picked from commit 9b5562f)
yuneng-berri
added a commit
that referenced
this pull request
Oct 3, 2026
) * test: repair stale and polluting tests red on scheduled main CI (#44229) Partial backport to rc/1.104.0: only the owned Redis port, Presidio stored-content check and event-loop lag logging fixes apply here. The other repairs target tests or behavior this line does not have * test(proxy): stop the proxy_server app fixture leaking LITELLM_LOG The session app fixture set LITELLM_LOG=ERROR with os.environ.setdefault and never removed it, so later tests on the same xdist worker inherited it. test_drop_params_env_var spawns a subprocess with os.environ and lost the warning it asserts on. Scope the variable to the import with a MonkeyPatch context * test(secret-detection): give the hand-built redaction request an ASGI path Since #43975 _read_request_body checks the route path via request.scope, and a scope without path raised KeyError that was swallowed into an empty body, so chat_completion failed with a missing messages parameter. Real ASGI scopes always carry path * test(integration): isolate litellm callback lists per sdk test usage-based-routing-v2 Routers register their selector in litellm.callbacks and nothing removes it, not even Router.reset(). The counter TTL and Redis service metrics tests left their selectors behind, and the next usage routing test ran their pre-call checks against its own rpm=1 deployments, raising "Deployment over defined rpm limit". An autouse fixture now gives each sdk test copies of the callback lists and restores the originals afterwards * test(integration): keep the owner-lookup fault proxy off the shared read replica The owned proxy points DATABASE_URL at a scratch database but inherited DATABASE_URL_READ_REPLICA from the replica job, so auth read the shared database and rejected the freshly created key with token_not_found_in_db. Drop the replica variable like the other scratch-database owned proxies * test(integration): request every seeded key in the team owner breakdown The aggregated team activity endpoint now caps breakdown.api_keys at the top 100 keys by default (#43398), so the 300 seeded keys came back as 100 rows. The test guarantees each key is reported with its own owner, so ask for an api_key_limit that covers all seeded keys * test(integration): give every owned Redis its own port in the redis-cache container On CircleCI every owned Redis ran on the fixed port 16379 inside the shared redis-cache container. When an earlier server still held that port, the new one failed to bind, readiness pinged the old server, the pidfile read failed and cleanup then reported "Owned Redis still serves after shutdown" Reserve an ephemeral port for the docker-exec path the same way the local binary path already does, and refuse to start when something already serves the chosen port so the failure names the real cause * test(e2e): skip the Vertex Mistral partner case the e2e project cannot reach The e2e Vertex project gets a 404 publisher model not found for vertex_ai/mistral-small-2503, so the case can only fail * test(e2e): skip the Vertex gpt-oss partner case the e2e project never serves vertex_ai/openai/gpt-oss-120b-maas has hit a 60s read timeout with no response headers on every run in the e2e Vertex project since the case was ported, and no other Vertex partner chat model passes there to switch to * test(e2e): check only stored message content for a leaked card number The Presidio spend-log check ran the card-number pattern over the whole serialized response, so a Luhn-valid usage.cost float (0.0003466000000000001) failed the streaming /v1/messages case although the stored content was <CREDIT_CARD>. The check now reads the content and text strings of the stored response, which is where a raw card would land, and still requires the placeholder there * test(e2e): assert the proxy decodes token-array embeddings for titan The port in #44120 carried over a legacy SDK-direct test that expected Bedrock to reject token ids with a 400. Through the proxy, /embeddings decodes token arrays to text for providers that cannot embed tokens, so titan answers 200. The test now sends a token array and its decoded sentence and requires the two vectors to match, which fails if the proxy stops decoding or decodes with the wrong tokenizer * test(e2e): run the Bedrock extended-thinking round trip on a model that honors enabled thinking us.anthropic.claude-sonnet-5-5 is adaptive-only, so litellm sends thinking.type=enabled with a 1024 budget as adaptive with low effort, and Bedrock returned no reasoning blocks on 5 of 5 identical Converse calls (boto3 direct agreed). us.anthropic.claude-sonnet-4-6 accepts the legacy shape verbatim and returned reasoning on 5 of 5. The non-thinking Bedrock case stays on sonnet-5-5 * test(proxy): stop unit modules forcing DEBUG logging into the event-loop lag tests Five tests/unit modules set verbose_proxy_logger to DEBUG at import, so every xdist worker that collected them logged the 2.4MB pass-through response from a worker thread, and secret redaction of that line held the GIL for ~0.8s+ inside the timed window. The lag tests now pin the LiteLLM loggers to WARNING and freeze gc while timing, and the module-level DEBUG overrides are removed * test(e2e): cite the tokenizer and date behind the titan token-array fixture * test(e2e): let migration seed replicas finish their request-log indexes before cloning Since #43948 a serving proxy builds the two LiteLLM_SpendLogs indexes on a background thread after it reports ready. The seed fixtures stopped the replica at readiness, so every cloned legacy database lacked an index no real deployment would be missing, and the v2 baseline diff refused it. Seeds now wait until both indexes exist and are valid in the database's schema * test(passthrough): give the pass-through MockRequest an httpx URL and ASGI scope * test(integration): ignore foreign allow_all_keys MCP servers in the access matrix tool list test_toolset_gateway_url_serves_a_team_granted_toolset_to_a_key_without_its_own_grant (#43908) registers an allow_all_keys server on the shared gateway, and allow_all_keys servers are listed to every key by design, so a matrix case running on another xdist worker at the same time saw its tools. The matrix now drops tools of allow_all_keys servers it did not create, read from LiteLLM_MCPServerTable before and after listing, and still compares everything else exactly (cherry picked from commit 9b5562f) * fix(proxy-extras): retry P3009 when a peer already recovered the named migration row (#44283) * fix(proxy-extras): retry P3009 when a peer already recovered the named migration row * fix(proxy-extras): pin P3009 recovery locals as Final and cover a recovered sibling row * test(proxy-extras): stop the P3009 ledger fakes shadowing the partition detector cursor * fix(proxy-extras): annotate the migration ledger reader locals as Final --------- Co-authored-by: yuneng <yuneng@berri.ai> (cherry picked from commit 8b11b68) --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
tests/call real providers from CircleCItests/e2e, which owns that tierHow it solves it:
tests/e2esuitesBefore porting, each test was checked against
tests/e2e,tests/integrationandtests/unitfor an existing owner of the same provider, endpoint and behavior. Ported tests use the e2e harness (/model/newplus deferred cleanup) and assert what the caller receives instead of only a non-empty responsePre-Submission checklist
Screenshots / Proof of Fix
There is no runtime change, so the proof is the new e2e tests running against a local proxy and real providers, plus checks that the legacy side is consistent
Before (6a8e0a2)
After (b6be38c)
basedpyrightontests/e2e: 0 errorstests/unit/test_assert_ci_coverage.py: 37 passedPer-test mapping
All 61 tests
Ported (25)
tests/llm_responses_api_testing/test_openai_responses_api.py::TestOpenAIResponsesAPITest::test_responses_api_multi_turn_with_reasoning_and_structured_outputtests/e2e/llm_translation/test_responses_e2e.py::TestResponsesOpenAIHostedFeatures::test_reasoning_items_replay_into_structured_output_after_tool_call(ran against the real provider)tests/llm_responses_api_testing/test_openai_responses_api.py::TestOpenAIResponsesAPITest::test_responses_api_shell_tool_streaming_sees_shell_outputtests/e2e/llm_translation/test_responses_e2e.py::TestResponsesOpenAIHostedFeatures::test_shell_tool_stream_surfaces_shell_call_and_its_output(ran against the real provider)tests/llm_translation/test_bedrock_completion.py::test_bedrock_document_understandingtests/e2e/llm_translation/test_chat_completions_regression_e2e.py::TestBedrockConverseChatCompletions::test_bedrock_converse_reads_a_pdf_sent_by_url(not run locally)tests/audio_tests/test_audio_speech.py::test_aws_polly_tts_real_apitests/e2e/llm_translation/test_audio_speech_e2e.py::TestAwsPollySpeech::test_polly_generative_voice_returns_mp3(not run locally)tests/llm_responses_api_testing/test_anthropic_responses_api.py::TestAnthropicResponsesAPITest::test_basic_openai_list_input_items_endpointtests/e2e/llm_translation/test_responses_retrieve_e2e.py::TestStoredResponseLifecycle::test_input_items_list_the_stored_prompt(ran against the real provider)tests/local_testing/test_function_calling.py::test_aaparallel_function_calltests/e2e/llm_translation/test_chat_tool_round_trip_e2e.py::TestChatToolResultRoundTrip::test_gemini (+ test_mistral, test_bedrock_converse)(partly run)tests/local_testing/test_function_calling.py::test_aaparallel_function_call_with_anthropic_thinkingtests/e2e/llm_translation/test_chat_tool_round_trip_e2e.py::TestChatToolResultRoundTrip::test_anthropic_with_extended_thinking (+ test_bedrock_converse_with_extended_thinking)(partly run)tests/local_testing/test_amazing_vertex_completion.py::test_partner_models_httpx_streamingtests/e2e/llm_translation/test_chat_completions_regression_e2e.py::TestVertexPartnerChatCompletions::test_vertex_partner_model_streams_openai_shaped_chunks(not run locally)tests/local_testing/test_amazing_vertex_completion.py::test_vertexai_embedding_embedding_latesttests/e2e/llm_translation/test_embeddings_endpoint_e2e.py::TestEmbeddingsEndpoint::test_vertex_embeddings_honor_requested_dimensions(not run locally)tests/local_testing/test_amazing_vertex_completion.py::test_vertexai_multimodalembedding_embedding_latesttests/e2e/llm_translation/test_embeddings_endpoint_e2e.py::TestEmbeddingsEndpoint::test_vertex_multimodal_embeddings_honor_dimensions_and_are_costed(not run locally)tests/test_openai_endpoints.py::test_embeddingstests/e2e/llm_translation/test_embeddings_endpoint_e2e.py::TestEmbeddingsEndpoint::test_mistral_embeddings_returns_vector(not run locally)tests/audio_tests/test_whisper.py::test_transcription_openai_whispertests/e2e/llm_translation/test_audio_transcriptions_e2e.py::TestWhisperTranscriptionFormats(ran against the real provider)tests/local_testing/test_embedding.py::test_demo_tokens_as_input_to_embeddings_fails_for_titantests/e2e/llm_translation/test_embeddings_endpoint_e2e.py::TestEmbeddingsEndpoint::test_bedrock_titan_rejects_token_array_input_as_bad_request(not run locally)tests/openai_endpoints_tests/test_e2e_openai_responses_api.py::test_basic_responsetests/e2e/llm_translation/test_responses_retrieve_e2e.py::TestStoredResponseLifecycle::test_deleted_response_is_no_longer_retrievable(ran against the real provider)tests/openai_endpoints_tests/test_e2e_openai_responses_api.py::test_cancel_responsetests/e2e/llm_translation/test_responses_retrieve_e2e.py::TestBackgroundResponseCancel::test_cancel_background_response(ran against the real provider)tests/openai_endpoints_tests/test_e2e_openai_responses_api.py::test_cancel_streaming_responsetests/e2e/llm_translation/test_responses_retrieve_e2e.py::TestBackgroundResponseCancel::test_cancel_background_streaming_response_by_streamed_id(ran against the real provider)tests/llm_translation/test_containers_api.py::test_container_files_apitests/e2e/llm_translation/test_containers_e2e.py::TestOpenAIContainerFiles::test_container_file_lifecycle_through_the_gateway(ran against the real provider)tests/pass_through_unit_tests/messages_api_structured_output/test_anthropic_api_structured_output.py::TestAnthropicAPIStructuredOutput::test_structured_output_e2etests/e2e/llm_translation/test_messages_e2e.py::TestAnthropicMessages::test_messages_output_format_returns_schema_json(ran against the real provider)tests/llm_translation/test_mistral_api.py::TestMistralCompletion::test_basic_tool_callingtests/e2e/llm_translation/test_chat_tool_round_trip_e2e.py::TestChatToolResultRoundTrip::test_mistral(not run locally)tests/guardrails_tests/test_bedrock_guardrails.py::test_bedrock_guardrails_block_messages_apitests/e2e/guardrails/test_bedrock_guardrail_e2e.py::TestBedrockGuardrail::test_bedrock_pre_call_blocks_on_messages(not run locally)tests/guardrails_tests/test_bedrock_guardrails.py::test_bedrock_guardrails_block_responses_apitests/e2e/guardrails/test_bedrock_guardrail_e2e.py::TestBedrockGuardrail::test_bedrock_pre_call_blocks_on_responses(not run locally)tests/guardrails_tests/test_bedrock_guardrails.py::test_bedrock_guardrails_with_streamingtests/e2e/guardrails/test_bedrock_guardrail_e2e.py::TestBedrockGuardrail::test_bedrock_post_call_blocks_denied_streamed_output_and_passes_clean_streams(not run locally)tests/guardrails_tests/test_bedrock_guardrails.py::test_bedrock_guardrails_with_streaming_no_violationtests/e2e/guardrails/test_bedrock_guardrail_e2e.py::TestBedrockGuardrail::test_bedrock_post_call_blocks_denied_streamed_output_and_passes_clean_streams(not run locally)tests/pass_through_unit_tests/messages_api_structured_output/test_azure_anthropic_structured_output.py::TestAzureAnthropicStructuredOutput::test_structured_output_e2etests/e2e/llm_translation/test_messages_azure_foundry_e2e.py::TestAzureFoundryMessages::test_output_format_returns_schema_json(not run locally)tests/pass_through_unit_tests/messages_api_structured_output/test_bedrock_converse_structured_output.py::TestBedrockConverseStructuredOutput::test_structured_output_e2etests/e2e/llm_translation/test_messages_bedrock_e2e.py::TestBedrockMessages::test_converse_output_format_returns_schema_json_text(not run locally)Deleted, already covered (16)
tests/llm_translation/test_bedrock_completion.py::test_bedrock_passthroughtests/e2e/llm_translation/test_bedrock_native_e2e.py::TestBedrockNative::test_invoke_returns_message(equal or stronger)tests/local_testing/test_amazing_vertex_completion.py::test_gemini_pro_function_calling_httpxtests/e2e/llm_translation/test_chat_completions_regression_e2e.py::TestVertexChatCompletions::test_vertex_chat_returns_tool_call(equal)tests/local_testing/test_amazing_vertex_completion.py::test_gemini_pro_function_calling_streamingtests/e2e/llm_translation/test_chat_completions_regression_e2e.py::TestVertexChatCompletions::test_vertex_chat_streams_real_content(equal)tests/test_openai_endpoints.py::test_completiontests/e2e/llm_translation/test_completions_endpoint_e2e.py(equal)tests/test_openai_endpoints.py::test_image_generationtests/e2e/llm_translation/test_image_generation_e2e.py(equal)tests/test_openai_endpoints.py::test_moderations_endpointtests/e2e/llm_translation/test_moderations_e2e.py(equal)tests/batches_tests/test_openai_batches_and_files.py::test_create_batchtests/e2e/batches/test_batches_e2e.py::test_batch_lifecycle[openai-*](stronger)tests/openai_endpoints_tests/test_e2e_openai_responses_api.py::test_anthropic_with_responses_apitests/e2e/llm_translation/test_responses_e2e.py::TestResponses::test_responses_anthropic_returns_completion(equal)tests/openai_endpoints_tests/test_openai_batches_endpoint.py::test_batches_operationstests/e2e/batches/test_batches_e2e.py::test_batch_lifecycle[openai-*](stronger)tests/openai_endpoints_tests/test_openai_batches_endpoint.py::test_e2e_batches_filestests/e2e/batches/test_batches_e2e.py::TestBatchTerminalState::test_completed_batch_downloads_output_and_books_cost(stronger)tests/openai_endpoints_tests/test_bedrock_batches_api.py::test_bedrock_batches_apitests/e2e/batches/test_batches_e2e.py::test_batch_lifecycle[bedrock-unified](equal or stronger)tests/llm_translation/test_together_ai.py::TestTogetherAI::test_basic_tool_callingtests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_tool_call_is_returned(stronger)tests/logging_callback_tests/test_token_counting.py::test_stream_token_counting_anthropic_with_include_usagetests/integration/cost_calculation test_cost_tracking claude-*-stream cases(equal)tests/llm_translation/realtime/test_openai_realtime.py::test_openai_realtime_direct_call_with_intenttests/unit/realtime_api/test_main.py::test_arealtime_openai_forwards_the_intent_query_param_to_the_upstream_url(stronger for the contract LiteLLM owns)tests/llm_translation/test_openai.py::TestOpenAIChatCompletion::test_basic_tool_callingtests/e2e conversational matrix test_tool_call_is_returned_named_and_addressable / test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-4o-mini](stronger)tests/test_fallbacks.py::test_chat_completiontests/e2e/router/test_reliability_fallbacks_e2e.py::test_context_window_routes_to_fallback(equal)Left in place, needs a decision (20)
tests/batches_tests/test_manus_files_all_methods.py::test_manus_files_api_e2e_all_methodstests/proxy_e2e_anthropic_messages_tests/test_claude_agent_sdk.py::test_claude_agent_sdk_streamingtests/audio_tests/test_audio_speech.py::test_azure_ava_tts_asynctests/audio_tests/test_whisper.py::test_transcription_azure_whispertests/batches_tests/test_openai_batches_and_files.py::test_async_create_batchtests/ocr_tests/test_ocr_matrix.py::test_ocrtests/ocr_tests/test_ocr_matrix.py::test_router_aocrtests/llm_translation/test_bedrock_agentcore.py::test_bedrock_agentcore_basictests/llm_translation/test_bedrock_agentcore.py::test_bedrock_agentcore_with_streamingtests/proxy_e2e_anthropic_messages_tests/test_all_beta_headers.py::test_anthropic_messages_with_all_beta_headerstests/proxy_e2e_anthropic_messages_tests/test_all_beta_headers.py::test_bedrock_invoke_messages_with_all_beta_headerstests/litellm_utils_tests/test_secret_manager.py::test_aws_secret_managertests/llm_translation/realtime/test_openai_realtime.py::test_openai_realtime_direct_call_no_intenttests/openai_endpoints_tests/test_openai_fine_tuning.py::test_openai_fine_tuningtests/store_model_in_db_tests/test_adding_passthrough_model.py::test_e2e_assemblyai_passthroughtests/guardrails_tests/test_bedrock_guardrails.py::test_bedrock_guardrails_pii_maskingtests/guardrails_tests/test_bedrock_guardrails.py::test_bedrock_guardrails_pii_masking_content_listtests/llm_translation/realtime/test_openai_realtime_simple.py::TestOpenAIRealtime::test_realtime_connectiontests/llm_translation/realtime/test_xai_realtime.py::TestXAIRealtime::test_realtime_connectiontests/search_tests/test_searchapi_search.py::TestSearchAPIIntegration::test_real_search_requestType
✅ Test
🚄 Infrastructure
Caveats (if any)
Medium
/v1/messagesLow
test_x = None