Skip to content

test(e2e): add a codex row that calls an mcp tool whose name is over 64 characters - #49826

Draft
ryan-crabbe-berri wants to merge 9 commits into
mainfrom
litellm_e2e_codex_long_mcp_tool_name
Draft

ryan-crabbe-berri wants to merge 9 commits into
mainfrom
litellm_e2e_codex_long_mcp_tool_name

Conversation

@ryan-crabbe-berri

@ryan-crabbe-berri ryan-crabbe-berri commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Handwritten 1-3 sentence description

<make-me-human-readable>
This adds a Codex test where Codex calls a tool with a very long name on a small local MCP server through the gateway. Bedrock refuses tool names over 64 characters and the gateway passes them through as is, so every Codex turn on Claude on Bedrock fails once such a tool is set up. The test marks that one setup as an expected failure and goes red once the fix lands
</make-me-human-readable>

Details

TLDR

  • Codex on Claude on Bedrock breaks when an MCP tool has a long name
  • Bedrock refuses any tool name longer than 64 characters
  • The new test has Codex call a 78 character tool and repeat its secret
  • It passes on the four GPT setups and is an expected failure on Claude on Bedrock
  • Once the gateway fix lands, the expected failure turns red and must be removed

User Flow

Before: a Codex user on Claude on Bedrock adds an MCP server with a long tool name and every turn fails

  1. They add a stdio MCP server under [mcp_servers.vault] in Codex's config.toml whose tool name is 78 characters, and point Codex at the gateway
  2. They open codex on a Claude on Bedrock model and ask it to call the read_the_current_secret_value_from_the_vault_that_this_codex_session_was_given tool
  3. The turn fails with a 400 saying mcp__vault__read_the_current_secret_value_... must have length less than or equal to 64

After: nothing changes for that user yet, but the suite now runs this flow on every setup and will flip to a pass once the gateway shortens long names for Bedrock

  1. They add the same MCP server and point Codex at the gateway
  2. They ask the same thing on GPT through Bedrock Mantle, Bedrock Runtime, Azure OpenAI or OpenAI and get back the value the tool returned
  3. On Claude on Bedrock they still get the 400, which the test records as an expected failure

Relevant issues

Guards LIT-10289, also reported by a customer in Pylon #11252. The open fixes are #42963 and #46243

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This PR only adds a test, so the proof is the new row running live plus the same flow driven by hand in the Codex TUI. The gateway runs main's code on port 47330 with a Postgres DB and real OpenAI, Azure (gpt-6.1-sol deployment) and Bedrock (us-east-1) credentials. Each test registers its own deployment through /model/new, drives the real Codex CLI 0.165.0 and deletes the deployment after

Before (9ea4531)

Codex TUI

  1. The row does not exist on the base branch, so nothing checks a long MCP tool name through the gateway

  2. With a key scoped to a fresh Claude on Bedrock deployment, config.toml pointing a responses provider at the gateway and [mcp_servers.vault] running tests/e2e/codex/mcp_server.py with TOOL_NAME set to the 78 character name, codex --model <claude deployment> "<prompt>" in a tmux pane shows (captured with tmux capture-pane -p)

› Call the read_the_current_secret_value_from_the_vault_that_this_codex_session_was_given tool of the vault MCP server, then reply with exactly what
  it returned.

■ {"error":{"message":"litellm.BadRequestError: BedrockException - {\"message\":\"1 validation error detected: Value
'mcp__vault__read_the_current_secret_value_from_the_vault_that_this_codex_session_was_given' at 'toolConfig.tools.25.member.toolSpec.name' failed to
satisfy constraint: Member must have length less than or equal to 64\"}\n\nLiteLLM: model group 'tui-claude-56645790' failed with the error above. No
fallback was attempted.","type":"invalid_request_error","param":null,"code":"400"}}

After (3a1fab8)

Codex TUI

  1. The same pane on a fresh OpenAI deployment with the same config, where the random value in the secret file was 1ede38e08676f47c
› Call the read_the_current_secret_value_from_the_vault_that_this_codex_session_was_given tool of the vault MCP server, then reply with exactly what
  it returned.

• I'll find and call the vault tool, then return its output exactly.

• Called vault.read_the_current_secret_value_from_the_vault_that_this_codex_session_was_given
  └ 1ede38e08676f47c
    + Show details

• 1ede38e08676f47c

New row

  1. bash run_e2e.sh <worktree> tests/e2e/codex/test_long_mcp_tool_name_e2e.py -n 5 -rA -q --reruns 0, which runs uv run --no-sync pytest from the worktree with LITELLM_PROXY_URL=http://127.0.0.1:47330 and the gateway's master key, three full runs in a row
PASSED tests/e2e/codex/test_long_mcp_tool_name_e2e.py::TestCodexLongMcpToolName::test_calls_an_mcp_tool_whose_name_is_over_64_characters[bedrock_runtime]
PASSED tests/e2e/codex/test_long_mcp_tool_name_e2e.py::TestCodexLongMcpToolName::test_calls_an_mcp_tool_whose_name_is_over_64_characters[bedrock_mantle]
PASSED tests/e2e/codex/test_long_mcp_tool_name_e2e.py::TestCodexLongMcpToolName::test_calls_an_mcp_tool_whose_name_is_over_64_characters[azure_openai]
PASSED tests/e2e/codex/test_long_mcp_tool_name_e2e.py::TestCodexLongMcpToolName::test_calls_an_mcp_tool_whose_name_is_over_64_characters[openai]
XFAIL tests/e2e/codex/test_long_mcp_tool_name_e2e.py::TestCodexLongMcpToolName::test_calls_an_mcp_tool_whose_name_is_over_64_characters[claude_bedrock] - LiteLLM sends Bedrock Converse a tool name over 64 characters, which it rejects (LIT-10289)
4 passed, 1 xfailed in 23.45s
4 passed, 1 xfailed in 23.38s
4 passed, 1 xfailed in 24.99s

All three runs had the same per-setup result. The passing setups only pass when Codex's mcp_tool_call item names the 78 character tool, its result is exactly the random value, the value is in the final answer and Codex exits cleanly

  1. The Claude on Bedrock setup run on its own with --runxfail -vv fails on the Bedrock 400 this PR guards
validation error detected: Value 'mcp__vault__read_the_current_secret_value_from_the_vault_that_this_codex_session_was_given' at 'toolConfig.tools.13.member.toolSpec.name' failed to satisfy constraint: Member must have length less than or equal to 64
1 failed in 16.69s
  1. The existing MCP row, whose tool keeps its short default name, passes on Claude on Bedrock and OpenAI at the same tip. That shows the Claude on Bedrock failure comes from the long name and nothing else, and that the server still works without TOOL_NAME
PASSED tests/e2e/codex/test_mcp_tool_e2e.py::TestCodexMcpTool::test_calls_a_local_mcp_tool_again_on_a_resumed_turn[claude_bedrock]
PASSED tests/e2e/codex/test_mcp_tool_e2e.py::TestCodexMcpTool::test_calls_a_local_mcp_tool_again_on_a_resumed_turn[openai]
2 passed in 26.30s
  1. Harness and lint checks
$ uv run --no-sync ruff check --config ruff-tests.toml tests/e2e/codex tests/e2e_harness/codex
All checks passed!
$ uv run --no-sync ruff format --check --config ruff-tests.toml tests/e2e/codex/test_long_mcp_tool_name_e2e.py tests/e2e/codex/mcp_server.py
2 files already formatted
$ uv run --no-sync basedpyright tests/e2e/codex tests/e2e_harness/codex
0 errors, 0 warnings, 0 notes
$ LITELLM_MASTER_KEY=sk-harness uv run --no-sync pytest tests/e2e_harness/codex -q
10 passed
$ uv run --no-sync pytest tests/code_coverage_tests/test_e2e_metadata.py -q --noconftest -o addopts= -o "pythonpath=tests/e2e tests/e2e_harness"
53 passed
$ LITELLM_MASTER_KEY=sk-placeholder uv run --no-sync pytest tests/e2e/codex --collect-only -q -o addopts=
30 tests collected

Type

✅ Test

Caveats (if any)

Low

  • GPT on Bedrock Runtime passes because it skips Converse
    • That setup uses Bedrock's OpenAI-style chat endpoint, not Converse
  • Azure's small token quota can 429 this row when other runs share it
  • The test trusts Codex to send the full name upstream
    • Codex 0.165.0 does, as the 400 above shows
    • A later Codex that shortens long names would flip the expected failure early

QA runbook

  • tests/e2e/codex/test_long_mcp_tool_name_e2e.py::TestCodexLongMcpToolName::test_calls_an_mcp_tool_whose_name_is_over_64_characters - Codex calls an MCP tool whose name is over 64 characters through the gateway and answers with what it returned
    • Register a deployment with POST http://localhost:4000/model/new (for example bedrock/us.anthropic.claude-sonnet-5-5 with aws_region_name: us-east-1) and make a key scoped to it with POST /key/generate
    • Point Codex's config.toml at the gateway as a responses provider and add [mcp_servers.vault] running python tests/e2e/codex/mcp_server.py with env = { SECRET_FILE = "<path>", TOOL_NAME = "read_the_current_secret_value_from_the_vault_that_this_codex_session_was_given" } and default_tools_approval_mode = "approve"
    • Write a random value to that file and ask Codex "Call the read_the_current_secret_value_from_the_vault_that_this_codex_session_was_given tool of the vault MCP server, then reply with exactly what it returned."
    • On Claude on Bedrock expect the turn to fail with Bedrock's 400 about toolSpec.name being over 64 characters, and on the GPT setups expect the tool call to return the value and the same value in Codex's reply
    • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

Low Risk
Test and documentation only; no production gateway or routing code changes.

Overview
Adds a Codex compatibility row that drives the real codex exec CLI through the gateway to call a local stdio MCP tool whose name exceeds Bedrock Converse’s 64-character limit (78 characters), asserting the tool output appears in McpCall records and the final answer.

mcp_server.py now registers the tool with @server.tool(name=os.environ.get("TOOL_NAME")) so the long-name case can set the name via env; existing MCP rows keep using the short name when TOOL_NAME is omitted.

claude_bedrock is marked strict xfail (LIT-10289): the gateway forwards the full prefixed name and Bedrock returns 400. GPT setups are expected to pass; fixing name truncation should flip that xfail to a hard fail.

tests/e2e/AGENTS.md documents the new row and fixes the codex/ suite bullet (it was incorrectly nested under cli_harness/).

Reviewed by Cursor Bugbot for commit 3a1fab8. Bugbot is set up for automated code reviews on this repo. Configure here.

…64 characters

Claude on Bedrock is a strict xfail because LiteLLM passes the namespaced tool name to Bedrock uncapped
and Bedrock rejects names over 64 characters (LIT-10289)
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

@greptileai re review

@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@greptile-apps

greptile-apps Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge

Summary

Adds a Codex test for an MCP tool whose name exceeds 64 characters, using the existing five-provider setup

  • The Codex suite now checks calls to MCP tools with long names.

No actionable new issues found. No live tests were run during this review

Reviews (2) · Last reviewed commit: "test(e2e): check the long mcp tool name ..." · Reviewed by Greptile

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@codecov

codecov Bot commented Nov 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

@greptileai re review

@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 3a1fab8. Configure here.

@CLAassistant

CLAassistant commented Nov 5, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Base automatically changed from litellm_e2e_codex_mcp_tools to main November 6, 2026 03:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants