fix(proxy): strip tool config from health check probes (#31008)

[AyushGupta-Code]

Relevant issues

Fixes #31008

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Config (health_check_extra_body.yaml, needs OPENROUTER_API_KEY in .env):

model_list:
  - model_name: openrouter-websearch
    litellm_params:
      model: openrouter/openai/gpt-4o-mini
      api_key: os.environ/OPENROUTER_API_KEY
      extra_body:
        tools:
          - type: openrouter:web_search
general_settings:
  master_key: sk-1234

Before the fix (run on the unpatched branch), the deployment is reported unhealthy because the probe inherits extra_body.tools:

curl -s http://localhost:4000/health -H "Authorization: Bearer sk-1234" \
  | jq '{healthy: .healthy_endpoints, unhealthy: .unhealthy_endpoints}'

PASTE OUTPUT: openrouter-websearch under unhealthy_endpoints with a 500

After the fix, the same deployment is healthy and a real completion still uses web search (tools are preserved for live traffic, only probes strip them):

curl -s http://localhost:4000/health -H "Authorization: Bearer sk-1234" \
  | jq '{healthy: .healthy_endpoints, unhealthy: .unhealthy_endpoints}'

curl -s http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{"model":"openrouter-websearch","messages":[{"role":"user","content":"What did OpenRouter announce most recently? Use web search."}]}' \
  | jq '.choices[0].message.content'

PASTE OUTPUT: openrouter-websearch under healthy_endpoints, plus the completion text

Type

🐛 Bug Fix

Changes

Background and on-demand health checks reuse the deployment's litellm_params, so a deployment with extra_body.tools (e.g. openrouter:web_search) sent those server tools on the trivial probe and OpenRouter 500'd it, while real completions on the same deployment succeeded. _update_litellm_params_for_health_check now works on a copy and strips tools and tool_choice from both the top level and extra_body before probing, since a trivial probe never needs tool configuration

tool_choice is removed alongside tools because some providers reject a tool_choice with no tools. Other extra_body fields such as provider routing are preserved, and the caller's params are no longer mutated; a regression test in tests/litellm_utils_tests/test_health_check.py pins both the stripping and the no-mutation behavior

[AyushGupta-Code]

@greptileai

[greptile-apps]

Greptile Summary

This PR fixes health check probes failing for deployments that configure provider server tools (e.g. openrouter:web_search) via extra_body.tools — the probe inherited those tool params and caused the provider to return a 500, marking the deployment unhealthy even though live traffic worked fine.

• _update_litellm_params_for_health_check now works on a shallow copy and strips tools and tool_choice from both the top-level params and extra_body before probing, using the new _HEALTH_CHECK_STRIPPED_REQUEST_KEYS frozenset; other extra_body fields (e.g. provider routing) are preserved.
• Two new mock-only regression tests cover the stripping behaviour, the no-mutation guarantee, and the no-extra_body baseline case.

Confidence Score: 5/5

Safe to merge — the change is narrowly scoped to health probe construction, creates a copy so caller params are unaffected, and is covered by new regression tests.

The fix is minimal and well-contained: it creates a shallow copy of litellm_params, filters two keys from it and from extra_body, and leaves all other probe logic untouched. The tests directly verify stripping, non-mutation, and the no-extra_body path. No production path for live completions is altered.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/health_check.py	Adds _HEALTH_CHECK_STRIPPED_REQUEST_KEYS frozenset and strips tools/tool_choice from both top-level params and extra_body at the start of _update_litellm_params_for_health_check, working on a copy so caller params are never mutated.
tests/test_litellm/proxy/test_health_check_max_tokens.py	Adds two new mock-only regression tests: one verifying tools/tool_choice are stripped from both top-level and extra_body while other extra_body fields survive, and one verifying no crash when extra_body is absent.

_{Reviews (3): Last reviewed commit: "test(health_check): cover probe tool-str..." | Re-trigger Greptile}

[greptile-apps]

The older OpenAI functions/function_call keys (the predecessor to tools/tool_choice) are not included in the strip set. Some providers still accept or even require this older format, and a deployment configured with functions in its litellm_params or extra_body would have the same 500-on-probe failure that this PR is fixing. Worth adding them here while the mechanism is being established.

Suggested change

	_HEALTH_CHECK_STRIPPED_REQUEST_KEYS: frozenset[str] = frozenset(
	{"tools", "tool_choice"}
	)
	_HEALTH_CHECK_STRIPPED_REQUEST_KEYS: frozenset[str] = frozenset(
	{"tools", "tool_choice", "functions", "function_call"}
	)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Greptile Summary

This PR fixes a bug where health check probes inherited tools/tool_choice from deployment litellm_params (including inside extra_body), causing providers like OpenRouter to 500 on the probe while real traffic succeeded. The fix makes _update_litellm_params_for_health_check work on a shallow copy and strips those keys before probing.

• _HEALTH_CHECK_STRIPPED_REQUEST_KEYS constant added; tools and tool_choice are removed from both the top-level params and extra_body, preserving all other extra_body fields (e.g. provider routing).
• The caller's original dict is no longer mutated — a regression test pins both the stripping and the no-mutation behavior.

Confidence Score: 4/5

Safe to merge; the change is well-scoped and guarded by two targeted unit tests

The fix is straightforward and the tests verify both the stripping and no-mutation guarantees. One edge case remains: when all keys in extra_body are stripped, an empty {} dict is left in the params rather than removing the key entirely. This is benign for most providers but inconsistent with the "no extra_body" path tested in the second new test.

litellm/proxy/health_check.py — the empty extra_body case after stripping

Important Files Changed

Filename	Overview
litellm/proxy/health_check.py	Adds _HEALTH_CHECK_STRIPPED_REQUEST_KEYS constant and rebuilds litellm_params as a shallow copy at the top of _update_litellm_params_for_health_check, stripping tools/tool_choice from both the top-level dict and extra_body; minor edge case where an all-tools extra_body becomes {}
tests/litellm_utils_tests/test_health_check.py	Adds two new unit tests covering tool-config stripping and no-mutation guarantees; both are pure function tests with no network calls, consistent with the test directory's rules

_{Reviews (2): Last reviewed commit: "fix(proxy): strip tool config from healt..." | Re-trigger Greptile}

[greptile-apps]

When every key in extra_body is a stripped key (e.g. a deployment that only sets extra_body.tools), the result is litellm_params["extra_body"] = {}. Most providers will ignore an empty dict, but some may treat the presence of the key differently than its absence. Dropping the empty dict is safer and keeps the probe equivalent to one without an extra_body.

Suggested change

	if isinstance(_extra_body, dict):
	litellm_params["extra_body"] = {
	k: v
	for k, v in _extra_body.items()
	if k not in _HEALTH_CHECK_STRIPPED_REQUEST_KEYS
	}
	if isinstance(_extra_body, dict):
	_stripped_extra_body = {
	k: v
	for k, v in _extra_body.items()
	if k not in _HEALTH_CHECK_STRIPPED_REQUEST_KEYS
	}
	if _stripped_extra_body:
	litellm_params["extra_body"] = _stripped_extra_body
	else:
	litellm_params.pop("extra_body", None)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[AyushGupta-Code]

Relevant issues

Fixes #31008

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:
• CI run for the last commit
  Link:
• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Config (health_check_extra_body.yaml, needs OPENROUTER_API_KEY in .env):

model_list:
  - model_name: openrouter-websearch
    litellm_params:
      model: openrouter/openai/gpt-4o-mini
      api_key: os.environ/OPENROUTER_API_KEY
      extra_body:
        tools:
          - type: openrouter:web_search
general_settings:
  master_key: sk-1234

Before the fix (run on the unpatched branch), the deployment is reported unhealthy because the probe inherits extra_body.tools:

curl -s http://localhost:4000/health -H "Authorization: Bearer sk-1234" \
  | jq '{healthy: .healthy_endpoints, unhealthy: .unhealthy_endpoints}'

PASTE OUTPUT: openrouter-websearch under unhealthy_endpoints with a 500

After the fix, the same deployment is healthy and a real completion still uses web search (tools are preserved for live traffic, only probes strip them):

curl -s http://localhost:4000/health -H "Authorization: Bearer sk-1234" \
  | jq '{healthy: .healthy_endpoints, unhealthy: .unhealthy_endpoints}'

curl -s http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{"model":"openrouter-websearch","messages":[{"role":"user","content":"What did OpenRouter announce most recently? Use web search."}]}' \
  | jq '.choices[0].message.content'

PASTE OUTPUT: openrouter-websearch under healthy_endpoints, plus the completion text

Type

🐛 Bug Fix

Changes

Background and on-demand health checks reuse the deployment's litellm_params, so a deployment with extra_body.tools (e.g. openrouter:web_search) sent those server tools on the trivial probe and OpenRouter 500'd it, while real completions on the same deployment succeeded. _update_litellm_params_for_health_check now works on a copy and strips tools and tool_choice from both the top level and extra_body before probing, since a trivial probe never needs tool configuration

tool_choice is removed alongside tools because some providers reject a tool_choice with no tools. Other extra_body fields such as provider routing are preserved, and the caller's params are no longer mutated; a regression test in tests/litellm_utils_tests/test_health_check.py pins both the stripping and the no-mutation behavior

The "Check UI API Types Sync" failure is independent of this PR. The check runs only because its path filter matches litellm/proxy/**, and the single proxy change here is _update_litellm_params_for_health_check, an internal helper that appears in no route signature or Pydantic model. It contributes nothing to the proxy OpenAPI spec, so regenerating schema.d.ts from this branch produces the same result as regenerating from litellm_internal_staging; the diff the check reports already exists on the base

That base drift has been accumulating. schema.d.ts is behind roughly a dozen proxy and types commits that merged after its last regeneration in #30902, several of which add or change routes and models. Pulling that into a focused health-check fix would broaden the scope well beyond the bug, so I have left schema.d.ts untouched here

Happy to open a separate PR against staging that runs npm run gen:api and commits the refreshed schema.d.ts if that is the preferred way to clear the check

[Sameerlite]

Thanks for the contribution! A couple of things:

• The proof of fix section in the PR body has unfilled placeholders ("PASTE OUTPUT: ...") — could you fill those in with actual curl output showing the health check probes no longer include tool configs?
• Triggering a fresh Greptile review on the latest commit.

@greptileai

[Code-weaver1]

Ran into this same class of problem building an OpenRouter-first agent harness, so I read through the patch — two notes and a question from the sidelines:

What this covers, from the diff: the probe works on a copy, tools/tool_choice are stripped at both the top level and inside extra_body, and unrelated extra_body keys (e.g. provider routing) survive. That's a nicer scope than the blunt pop("extra_body") workaround in the issue thread — deployments that rely on extra_body for provider routing keep meaningful probes. The no-mutation assertions in the tests matter too, since the router reuses those param dicts.

Scope note for maintainers: the second repro in #31008 (deployment tools + a very low client max_tokens failing on live traffic) is a different code path, and arguably correctly out of scope here — stripping tool config from real client requests would change request semantics. That part probably deserves its own issue rather than holding this fix up.

One question: OpenRouter's web search can also be configured via extra_body.plugins ([{"id": "web"}]) rather than the openrouter:web_search server tool. If that config triggers the same probe failure, plugins may belong in _HEALTH_CHECK_STRIPPED_REQUEST_KEYS as well — worth a quick check before this merges.