test(benchmarks): add CodSpeed benchmarks for inference, MCP and A2A hot paths

[yassin-berriai]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

These are CodSpeed micro-benchmarks, not a runtime behavior change, so the proof is that the new suite collects and runs green under the exact command the CI job uses. Run from the repo root in an isolated env that installs only the base dependency set the benchmark job uses (--no-default-groups) plus the benchmark extras:

env PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
  uv run --frozen --no-default-groups \
  --with pytest==8.3.5 \
  --with pytest-codspeed==4.3.0 \
  --with "mcp>=1.26.0,<2.0" \
  --with "a2a-sdk>=1.1.0,<2.0" \
  pytest -p pytest_codspeed.plugin tests/benchmarks/ --codspeed

Output (trimmed to the new benchmarks):

tests/benchmarks/test_a2a_benchmarks.py ....                             [ 13%]
tests/benchmarks/test_benchmarks.py ................                     [ 66%]
tests/benchmarks/test_inference_benchmarks.py .....                      [ 83%]
tests/benchmarks/test_mcp_benchmarks.py .....                            [100%]

│ test_a2a_inbound_message_to_openai_messages │   0ns │  0.7% │  2.90s │ 1,74… │
│              test_completion_simple_message │ 1.67… │  2.2% │  2.95s │ 11,0… │
│                  test_completion_multi_turn │ 1.70… │  2.6% │  2.98s │ 10,9… │
│                  test_completion_with_tools │ 1.75… │  1.5% │  2.98s │ 10,7… │
│                   test_completion_streaming │ 460.… │ 11.6% │  3.00s │   594 │
│      test_response_to_model_response_object │  20ns │  6.9% │  2.99s │ 96,0… │
│    test_transform_single_mcp_tool_to_openai │   0ns │  8.1% │  2.50s │ 1,51… │
│      test_transform_mcp_tool_list_to_openai │   1ns │  2.0% │  3.02s │ 456,… │
│      test_transform_openai_tool_call_to_mcp │   0ns │  1.2% │  2.83s │ 1,21… │
│            test_mcp_server_prefix_tool_list │   1ns │  1.2% │  2.92s │ 584,… │
│        test_mcp_server_strip_prefix_on_call │   0ns │  1.1% │  2.86s │ 2,14… │

================== 30 passed, 2 warnings in 122.96s (0:02:02) ==================

The sub-nanosecond transforms read as 0ns only in CodSpeed's local walltime mode; the CI run uses instrumentation mode, which counts CPU instructions and gives them a meaningful, stable number to diff against the base branch

Type

✅ Test

Changes

This adds CodSpeed benchmarks for the three areas of interest so perf regressions in their per-request hot paths surface on every commit going forward. Everything is designed to be lightweight and deterministic: each benchmark is pure in-process CPU work with zero network I/O (a hard requirement of CodSpeed's simulation mode, which counts CPU instructions), and the targets are the transformation functions rather than the full FastAPI proxy, so they import under the constrained dependency set the benchmark job installs

Inference (tests/benchmarks/test_inference_benchmarks.py) exercises the full SDK overhead through litellm.completion(..., mock_response=...) for simple, multi-turn, with-tools and streaming calls, which walks provider resolution, request/response transformation, ModelResponse construction, token counting and cost calculation without hitting a provider. It also benchmarks convert_to_model_response_object directly as a deterministic anchor for the provider-response to ModelResponse core that every non-streaming completion runs

MCP (tests/benchmarks/test_mcp_benchmarks.py) covers the client-side MCP to OpenAI tool translation (a single tool, a 20-tool list mirroring list-tools, and a tool-call dispatch) plus the proxy server-side tool-name prefix round-trip, add_server_prefix_to_name over a tool list and split_server_prefix_from_name on a call, which is the real per-request CPU cost on the MCP endpoint

A2A (tests/benchmarks/test_a2a_benchmarks.py) covers the client direction (convert_messages_to_prompt for the request and extract_text_from_a2a_response for message and task-artifact response shapes) and the proxy server-ingress A2ACompletionBridgeTransformation.a2a_message_to_openai_messages that runs on every inbound A2A request

The workflow change adds the mcp and a2a-sdk packages to the benchmark run since those transform modules need them at import time, and broadens the push and pull_request triggers to include litellm_internal_staging so the internal branch flow, where contributors open PRs against staging rather than main, is benchmarked too

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yassin-berriai]

@greptileai

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks
🆕 14 new benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
🆕	test_a2a_inbound_message_to_openai_messages	N/A	97.4 µs	N/A
🆕	test_convert_messages_to_a2a_prompt	N/A	111.1 µs	N/A
🆕	test_extract_text_from_a2a_message_response	N/A	87.3 µs	N/A
🆕	test_extract_text_from_a2a_task_response	N/A	87.2 µs	N/A
🆕	test_completion_multi_turn	N/A	3.1 ms	N/A
🆕	test_completion_simple_message	N/A	4.9 ms	N/A
🆕	test_completion_streaming	N/A	59.1 ms	N/A
🆕	test_completion_with_tools	N/A	4.2 ms	N/A
🆕	test_response_to_model_response_object	N/A	544 µs	N/A
🆕	test_mcp_server_prefix_tool_list	N/A	164.8 µs	N/A
🆕	test_mcp_server_strip_prefix_on_call	N/A	84.1 µs	N/A
🆕	test_transform_mcp_tool_list_to_openai	N/A	192.1 µs	N/A
🆕	test_transform_openai_tool_call_to_mcp	N/A	128.3 µs	N/A
🆕	test_transform_single_mcp_tool_to_openai	N/A	89.8 µs	N/A

---

_{Comparing litellm_codspeed_inference_mcp_a2a (af20e75) with litellm_internal_staging (cc57fe2)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (70eb4e5) during the generation of this report, so 15ba573 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds CodSpeed coverage for benchmark-only hot paths. The main changes are:

• New inference benchmarks for litellm.completion mock-response paths and ModelResponse conversion
• New MCP benchmarks for tool format conversion and server prefix helpers
• New A2A benchmarks for message prompt, response text, and inbound bridge transforms
• CodSpeed workflow updates for litellm_internal_staging and benchmark-only MCP/A2A dependencies

Confidence Score: 5/5

Safe to merge based on the benchmark-only scope and absence of code-path issues identified in the changed files.

The changes are isolated to benchmark coverage and CodSpeed workflow configuration, with deterministic in-process benchmark targets and no runtime behavior changes.

T-Rex Logs

What T-Rex did

• Ran the baseline uv benchmark and recorded 16 items collected with 16 passed and exit code 0.
• Ran the head uv benchmark after adding new A2A, inference, and MCP tests, and the run reported 30 items collected with 30 passed and exit code 0.
• The pre-change head state for push and pull_request showed only 'main' in uv dependencies, limited to pytest and pytest-codspeed, and the post-change head added litellm_internal_staging for both push and pull_request along with the uv benchmark command including --with 'mcp>=1.26.0,<2.0' and --with 'a2a-sdk>=1.1.0,<2.0'.
• A Python-based artifact captures the updated uv benchmark command configuration used in the head run.

_{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "test(benchmarks): add CodSpeed benchmark..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR adds CodSpeed benchmark coverage for LiteLLM hot paths. The main changes are:

• Adds A2A benchmarks for prompt conversion, response text extraction, and inbound message translation
• Adds inference benchmarks for mock completions, tool calls, streaming, and response object conversion
• Adds MCP benchmarks for tool conversion and server prefix handling
• Updates the CodSpeed workflow to run on litellm_internal_staging and install benchmark-only MCP/A2A dependencies

Confidence Score: 4/5

The benchmark additions are isolated and low-risk, with one workflow determinism issue to address before relying on the results long-term.

The tests are scoped to in-process benchmark paths and the workflow change is straightforward, but floating benchmark-only dependency ranges can make performance baselines change without a repository change.

.github/workflows/codspeed.yml

T-Rex Logs

What T-Rex did

• Ran the base benchmark suite to collect initial results, confirming the base run executed 16 items with 16 benchmarked and exit code 0.
• Installed uv locally and extended the PATH for the PR command so uv is used in the subsequent runs.
• Reran the benchmark suite after changes, now collecting 30 items with 30 benchmarked and exit code 0.
• Validated the codspeed workflow on the base run, confirming executable parse/assertion results show branch triggers and benchmark package arguments.
• Validated the codspeed workflow on the head run, confirming staging branch triggers and MCP/A2A extras are present.
• Confirmed the validation used workflow file parsing rather than git diff to inspect changes.

_{Ran code and verified through T-Rex}

_{Reviews (2): Last reviewed commit: "test(benchmarks): add CodSpeed benchmark..." | Re-trigger Greptile}

[greptile-apps]

Pin benchmark dependencies
The new benchmark-only dependencies use open version ranges, while the existing benchmark tools are exact-pinned. When a new mcp or a2a-sdk release changes import or runtime behavior, CodSpeed baselines can shift without any repo change. Pin these versions and update them deliberately when the benchmark target changes.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!