feat(mcp): per-server env vars with global + per-user scopes

[mateo-berri]

Summary

Branch created off commit 719d562b3a5167dbcf06c853f5fd20cf98308bfc from #28401.

This includes the MCP per-server env vars work up to and including the fix that stops tool listing from being blocked on missing per-user env vars:

• Global scope: value supplied once by the admin, used for every user.
• Per-user scope: each user fills in their own value via the MCP Gateway dashboard.
• ${NAME} interpolation into static_headers at request time.
• Listing is best-effort (unfilled ${NAME} refs are left untouched so tools still appear); the friendly MCPMissingUserEnvVarsError with a setup deep-link is raised only on the tool-call path.
• Create resilience: POST /v1/mcp/server now treats the DB write as the commit point and the in-memory registry refresh (add_server + reload_servers_from_database) as best-effort, so a single pre-existing malformed env_vars row can no longer turn a successful insert into a 500 and orphan the server. Previously that 500 made callers assume failure and retry, which left duplicate rows
• List/edit round-trip fix (this revision): GET /v1/mcp/server (and the health endpoint) build LiteLLM_MCPServerTable from the in-memory registry via _build_mcp_server_table / health_check_server. Both copied static_headers but dropped env_vars, so the list always returned env_vars: null. The admin edit form is populated from that list data, so it loaded an empty env-var list; saving any edit then persisted env_vars: [], silently wiping the stored variables. With nothing left to interpolate, the ${VAR} static headers were forwarded upstream verbatim as literal text. Both conversions now carry env_vars, mirroring static_headers
• UI clarity (this revision): the per-user value column rendered as a plain input, so it read like a static value shared by every user. It now shows a persistent "Hint" addon with an explanatory tooltip and lighter text, and the row aligns to the top so the layout stays put when the name field shows a validation error. The section is also retitled from "Environment Variables" to "Variables": these values interpolate into Static Headers and Auth via ${NAME} rather than being exported into the MCP server's process environment, so the old label overpromised (the backend env_vars field is unchanged)

Commits included on top of litellm_internal_staging:

• 48a6334 feat(mcp): per-server env vars with global + per-user scopes
• accecaf test(mcp): add DB layer + load-user-env-vars coverage
• 28883ea feat(mcp/ui): match env-vars UI to card-grid design from proto(mcp): per-user env-var fields demo flow (UI mockup) #28399
• 855cf82 test(mcp): cover per-user env-var management endpoints
• 719d562 fix(mcp): don't block tool listing on missing per-user env vars
• ba20078 fix(mcp): keep a created MCP server when post-write registry refresh fails
• 8852c0d fix(mcp): preserve env_vars in registry to table conversion
• ba7cab2 fix(mcp/ui): clarify per-user env var field as a hint, not a value
• e33ec2e fix(mcp/ui): rename env vars section to Variables
• 67e81f7 fix(ui): move MCP env-var loading to React Query to drop lint suppressions

Proof of fix

End-to-end interpolation, verified against a tiny local MCP server whose only tool, whoami, returns the HTTP headers it received. That makes the forwarded header directly observable, so we can see whether ${VAR} placeholders were substituted on the way through the proxy. Proxy on localhost:4000, master key sk-1234, echo upstream on localhost:8765/mcp.

1. Create a server pointing at the echo upstream, with instance env vars and a header that interpolates all of them:

curl -s -X POST http://127.0.0.1:4000/v1/mcp/server \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{
    "alias": "echo_interp",
    "url": "http://localhost:8765/mcp",
    "transport": "http",
    "auth_type": "none",
    "static_headers": {"X-Db-Url": "${DB_PROTOCOL}://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}"},
    "env_vars": [
      {"name": "DB_PROTOCOL", "value": "postgresql", "scope": "global"},
      {"name": "DB_USER", "value": "svc_reporting", "scope": "global"},
      {"name": "DB_PASSWORD", "value": "s3cr3t-p@ss", "scope": "global"},
      {"name": "DB_HOST", "value": "db.corp.internal", "scope": "global"},
      {"name": "DB_PORT", "value": "5432", "scope": "global"},
      {"name": "DB_NAME", "value": "analytics", "scope": "global"},
      {"name": "CORP_USERNAME", "value": "", "scope": "user", "description": "Your DB username"}
    ]
  }'

2. The list endpoint now returns the env vars (this field was null before the fix, which is what emptied the edit form):

curl -s -H "Authorization: Bearer sk-1234" http://127.0.0.1:4000/v1/mcp/server \
  | python3 -c "import sys,json;[print(s['alias'],'->',len(s['env_vars'] or []),'env_vars') for s in json.load(sys.stdin) if s['alias']=='echo_interp']"
# echo_interp -> 7 env_vars

3. Call the tool through the proxy; the echo reports the header arriving fully interpolated:

curl -s -X POST http://127.0.0.1:4000/mcp-rest/tools/call \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"server_id":"","name":"echo_interp-whoami","arguments":{}}'
# received_headers.x-db-url:
#   postgresql://svc_reporting:s3cr3t-p@ss@db.corp.internal:5432/analytics

4. Reproduce the exact action that used to wipe the vars: load the server from the list (what the edit form reads) and PUT it straight back, then call the tool again:

loaded env_vars from list : 7 vars
after PUT, env_vars stored : 7 vars (preserved)
whoami again -> x-db-url   : postgresql://svc_reporting:s3cr3t-p@ss@db.corp.internal:5432/analytics

Before the fix step 4 loaded null, saved [], and the next call forwarded the header verbatim. A server whose vars were wiped by the old behaviour shows that literal-text symptom directly:

curl -s -X POST http://127.0.0.1:4000/mcp-rest/tools/call \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"server_id":"","name":"-whoami","arguments":{}}'
# received_headers.x-db-url:
#   ${DB_PROTOCOL}://${CORP_USERNAME}:${CORP_PASSWORD}@${DB_HOSTNAME}

Regression coverage: TestRegistryTableConversionPreservesEnvVars in tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py asserts both _build_mcp_server_table and health_check_server round-trip env-var name, scope, value, and description. Reverting either one-line fix makes the matching test fail with env_vars is None.

Concurrency fix: atomic per-user env var merge

POST /v1/mcp/server/{id}/user-env-vars previously read the user's stored values, merged the submitted updates in Python, and wrote the result back as three separate steps. Two simultaneous requests from the same user to the same server could both read the same snapshot, so the second write silently clobbered the first and one update was lost with no error. The read-modify-write now lives in a merge_user_env_vars DB helper that runs inside a transaction guarded by a (user_id, server_id) advisory lock, so concurrent writes are serialized and every update survives. Covered by test_merge_user_env_vars_serializes_concurrent_writes (two concurrent merges with disjoint updates both persist) and test_merge_user_env_vars_preserves_existing_and_prunes_disallowed; dropping the advisory lock makes the concurrency test fail with a lost update

The lock itself is taken with execute_raw, not query_raw. pg_advisory_xact_lock() returns void, and running it through query_raw makes the Prisma engine try to deserialize that column, raising RawQueryError: Failed to deserialize column of type 'void' and 500ing the endpoint. execute_raw doesn't deserialize a result set, so the lock is acquired cleanly. test_merge_user_env_vars_acquires_lock_without_deserializing_void guards this by raising the real void RawQueryError from a fake query_raw while execute_raw succeeds; the merge must still complete

Empty-valued global env vars

A scope="global" entry with an empty value is now treated as unset rather than as a usable fallback. Two code paths consult the admin globals when deciding whether a referenced ${NAME} still needs a per-user value: _resolve_static_headers_with_env_vars on the tool-call path and _compute_user_env_var_status behind the user-env-var status endpoint. Both keyed off the mere presence of the name in the globals map, so a same-named global with an empty value masked a per-user var; the status endpoint reported no credential needed and the merge substituted an empty string while a real tool call still required it, and the two could disagree, telling the user they were all set while every call returned a 412. Both now drop empty globals before the coverage check, so an empty global neither covers a required per-user var nor overrides a value the user did supply, and the unresolved ${NAME} is left untouched like any other undefined reference. Covered by test_resolve_static_headers_empty_global_does_not_cover_user_var, test_resolve_static_headers_user_value_wins_over_empty_global, and test_dual_scope_var_with_empty_global_is_required; dropping either filter makes the matching test fail

Frontend lint cleanup

The earlier frontend-lint gate commit baselined seven new eslint suppressions on the touched MCP files instead of fixing them. This revision removes the ones that are genuinely fixable rather than grandfathering them. The per-user env-var loading in UserEnvVarsModal and mcp_servers moved onto React Query (useQuery / useMutation), which clears all three react-hooks/set-state-in-effect findings (UserEnvVarsModal 1 -> 0, mcp_servers 4 -> 2); the ?fill_env_vars deep link is now captured in lazy initial state with the modal target derived during render rather than set from an effect. The unused clearMCPUserEnvVars wrapper, the one new raw fetch with no caller, is deleted (networking.tsx 274 -> 273). The remaining three wrappers (GET status, GET server vars, POST save) still call fetch because networking.tsx is the only HTTP layer and React Query consumes it rather than replacing it, so they stay grandfathered and the suppressions baseline only ratchets down. mcp_servers.test.tsx gains a listMCPUserEnvVarStatus mock so the new query resolves under test

Type

🐛 Bug Fix

---

> [!NOTE]
> High Risk
> Touches encrypted credential storage, upstream auth headers, and master-key rotation; mis-interpolation or redaction bugs could leak secrets or break MCP connectivity.
>
> Overview
> Adds MCP server variables so admins define global and per-user values that substitute into static_headers via ${NAME} at request time, with encrypted storage and master-key rotation.
>
> Data & API: New env_vars on LiteLLM_MCPServerTable and LiteLLM_MCPUserEnvVars for per-user secrets; management routes to get/set/clear user values and list missing vars (write-only status, no secret echo). Global values are redacted for non–full admins; variable names are hidden from non-admins on server list payloads.
>
> Runtime: MCPServerManager merges globals and user values into headers on tool call (412 / friendly error + UI deep-link if required per-user vars are missing, with DB recheck before failing); tool list and health/instruction probes stay best-effort and skip misleading unhealthy probes when headers need per-user vars. Short-lived per-user cache, advisory-locked merge on POST, cleanup on server delete, and env_vars preserved when converting registry → table (fixes list/edit wiping vars).
>
> Other: POST /v1/mcp/server treats DB create as success even if registry reload fails; OTEL redacts global env values in management responses.
>
> ^{Reviewed by Cursor Bugbot for commit 16bc527. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 5 committers have signed the CLA.

✅ mateo-berri
✅ yuneng-berri
❌ claude
❌ shin-berri
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 90.14085% with 56 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/_experimental/mcp_server/db.py	81.03%	33 Missing ⚠️
...oxy/_experimental/mcp_server/mcp_server_manager.py	91.12%	11 Missing ⚠️
...m/proxy/_experimental/mcp_server/rest_endpoints.py	82.35%	3 Missing ⚠️
litellm/proxy/_experimental/mcp_server/server.py	0.00%	3 Missing ⚠️
litellm/proxy/_experimental/mcp_server/utils.py	95.00%	3 Missing ⚠️
litellm/proxy/management_helpers/utils.py	85.71%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds MCP per-server environment variables with global (admin-supplied) and per-user scopes, ${NAME} interpolation into static_headers at request time, and several associated fixes: preserving env_vars through the registry↔table round-trip, treating the DB write as the atomic commit point for server creation, filtering empty-valued globals consistently across the tool-call and status-endpoint paths, and encrypting new fields during master-key rotation.

• New data model and endpoints: env_vars JSONB column on LiteLLM_MCPServerTable, LiteLLM_MCPUserEnvVars table for per-user encrypted blobs, and three /server/{id}/user-env-vars endpoints (GET/POST/DELETE) with access-scoped authorization and a bulk status endpoint for the dashboard. Global values are encrypted at rest, redacted for non-full-admin callers, and scrubbed from OTEL spans before telemetry.
• Tool-call and listing paths: tool calls raise a user-facing MCPMissingUserEnvVarsError (→ HTTP 412 with a setup deep-link) when required per-user vars are absent; listing is best-effort and leaves unresolved ${NAME} refs in place so tools still appear. Health checks skip servers whose headers reference per-user vars to avoid false unhealthy status.
• Access-control note: the non-admin path in _authorize_and_fetch_mcp_server loads every accessible server to find one by ID, which becomes expensive for users with broad access.

Confidence Score: 4/5

Safe to merge with attention to the advisory lock limitation and the per-server access-control query pattern noted in the comments.

The core correctness concerns flagged in earlier review rounds (double-decryption, master-key rotation gaps, virtual-key secret exposure, health-check false-unhealthy, status-endpoint empty-global filter) are all addressed in this revision with targeted fixes and regression tests. The remaining new findings are non-blocking: the pg_advisory_xact_lock call in merge_user_env_vars works correctly on PostgreSQL but will error on any non-PG backend, and the non-admin path in _authorize_and_fetch_mcp_server loads all accessible servers to authorize a single one.

litellm/proxy/_experimental/mcp_server/db.py (merge_user_env_vars advisory lock) and litellm/proxy/management_endpoints/mcp_management_endpoints.py (_authorize_and_fetch_mcp_server full-table scan for non-admin users).

Important Files Changed

Filename	Overview
litellm/proxy/_experimental/mcp_server/db.py	Adds global env-var encrypt/decrypt helpers, per-user env-var CRUD (merge_user_env_vars with advisory lock), master-key rotation for both new tables, and post-write decryption on create/update paths. PostgreSQL-specific advisory lock is the main caution.
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py	Adds _resolve_static_headers_with_env_vars (global+per-user merge, raise_on_missing flag), per-user env-var cache, _references_per_user_env_var guard for health-check skip, and env_vars_are_encrypted plumbing that fixes the double-decryption issue in add_server/update_server.
litellm/proxy/management_endpoints/mcp_management_endpoints.py	Adds three per-user env-var endpoints with access control, _compute_user_env_var_status, and correctly redacts global env-var values for non-full-admin callers. Non-admin auth check loads all accessible servers per request (O(N)).
litellm/proxy/_experimental/mcp_server/utils.py	Adds MCPMissingUserEnvVarsError, interpolation helpers (parse_admin_env_vars, interpolate_env_vars, collect_env_var_references), and build_env_var_setup_url. Clean, well-tested utilities.
litellm/proxy/_types.py	Adds MCPEnvVar, MCPEnvVarScope, MCPUserEnvVarsRequest/Status/Spec types. MCPEnvVarScope.global_ uses a trailing underscore to avoid keyword clash; value is global which serializes correctly.
litellm/proxy/management_endpoints/key_management_endpoints.py	Adds step 4c to _rotate_master_key to rotate the new LiteLLM_MCPUserEnvVars table alongside existing MCP credential tables.
litellm/proxy/management_helpers/utils.py	Adds _redact_env_var_values to scrub global env-var plaintext values from OTEL spans before telemetry emission.
litellm-proxy-extras/litellm_proxy_extras/migrations/20260520120000_add_mcp_env_vars/migration.sql	Adds env_vars JSONB column to LiteLLM_MCPServerTable and creates LiteLLM_MCPUserEnvVars table with proper unique index and lookup indexes. Uses IF NOT EXISTS guards throughout.

_{Reviews (57): Last reviewed commit: "fix(mcp): treat empty global as unset in..." | Re-trigger Greptile}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 9 · PR risk: 0/10

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai