fix(helm): authenticate ServiceMonitor scrapes of the /metrics endpoint

[mubashir1osmani]

Relevant issues

Supersedes #30565, which fixes the same problem but was written against deploy/charts/litellm-helm before the chart moved to helm/litellm-helm in #32234, and carries unrelated README changes. This PR rebuilds the fix on the current chart location and addresses the review feedback left on that PR

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Before, at base commit 7d15f2f, the chart renders the scrape endpoint with no credentials:

$ helm template my-release helm/litellm-helm --set serviceMonitor.enabled=true --show-only templates/servicemonitor.yaml
  ...
  endpoints:
  - port: http
    path: /metrics/
    interval: 15s
    scrapeTimeout: 10s
    scheme: http

so every scrape Prometheus makes is an unauthenticated GET of /metrics/, which the proxy rejects since require_auth_for_metrics_endpoint defaults to true. Reproduced against a live proxy on localhost (prometheus callback enabled, master key sk-1234):

$ curl -sS -w "\nHTTP %{http_code}\n" http://localhost:4010/metrics/
"Unauthorized access to metrics endpoint: Authentication Error, Malformed API Key passed in. Ensure Key has `Bearer ` prefix. To allow unauthenticated access, set `litellm_settings.require_auth_for_metrics_endpoint: false` in your proxy_config.yaml."
HTTP 401

After, at a06517c, pointing the ServiceMonitor at a secret holding a scrape token renders bearer credentials:

$ helm template my-release helm/litellm-helm --set serviceMonitor.enabled=true \
    --set serviceMonitor.authSecret.name=prometheus-litellm-token --set serviceMonitor.authSecret.key=token \
    --show-only templates/servicemonitor.yaml
  ...
  endpoints:
  - port: http
    path: /metrics/
    interval: 15s
    scrapeTimeout: 10s
    scheme: http
    authorization:
      type: Bearer
      credentials:
        name: prometheus-litellm-token
        key: token

and explicitly opting into the master key reuses the chart's masterkey secret, honoring masterkeySecretName/masterkeySecretKey when set:

$ helm template my-release helm/litellm-helm --set serviceMonitor.enabled=true \
    --set serviceMonitor.authSecret.useMasterKey=true --show-only templates/servicemonitor.yaml
  ...
    authorization:
      type: Bearer
      credentials:
        name: my-release-litellm-masterkey
        key: masterkey

Enabling the ServiceMonitor without choosing a credential fails at render time with instructions instead of silently deploying a broken scrape or handing Prometheus the admin key:

$ helm template my-release helm/litellm-helm --set serviceMonitor.enabled=true --show-only templates/servicemonitor.yaml
Error: ... The proxy requires a bearer token to scrape /metrics. Set serviceMonitor.authSecret.name and
serviceMonitor.authSecret.key to a secret holding a dedicated virtual key (recommended), or set
serviceMonitor.authSecret.useMasterKey=true to scrape with the master key, or set
serviceMonitor.authSecret.enabled=false if the proxy runs with require_auth_for_metrics_endpoint: false

$ helm template my-release helm/litellm-helm --set serviceMonitor.enabled=true \
    --set serviceMonitor.authSecret.name=prometheus-litellm-token --show-only templates/servicemonitor.yaml
Error: ... serviceMonitor.authSecret.key is required when serviceMonitor.authSecret.name is set

To prove the header the ServiceMonitor now sends is exactly what /metrics needs, a real completion through the live proxy followed by a scrape with that bearer token:

$ curl -sS http://localhost:4010/v1/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"model": "gpt-5.5", "messages": [{"role": "user", "content": "Say hi in three words"}]}'
{"id": "chatcmpl-Dz4pBu6L40cWCb9bsEv48UcVU5Yep", ... "content": "Hi there, friend!", ... "total_tokens": 19 ...}

$ curl -sS -o /dev/null -w "HTTP %{http_code}\n" -H "Authorization: Bearer sk-1234" http://localhost:4010/metrics/
HTTP 200

$ curl -sS -H "Authorization: Bearer sk-1234" http://localhost:4010/metrics/ | grep -E "^litellm_requests_metric|^litellm_total_tokens" | head -2
litellm_total_tokens_metric_total{api_key_alias="None",api_provider="openai",...,model="gpt-5.5",...} 19.0
litellm_requests_metric_total{api_key_alias="None",api_provider="openai",client_ip="127.0.0.1",...,model="gpt-5.5",...} 1.0

The existing chart test suite still passes with the same invocation as the helm_unit_test.yml workflow (helm unittest -f 'tests/*.yaml' helm/litellm-helm, 54 tests) and helm lint is clean

Type

🐛 Bug Fix

Changes

Enabling serviceMonitor.enabled in the helm chart produces a ServiceMonitor whose scrapes always fail with 401 because the proxy now requires a bearer token on /metrics by default (require_auth_for_metrics_endpoint defaults to true in litellm/__init__.py), so users get no metrics at all

The ServiceMonitor endpoint now sends Authorization: Bearer <token> using the prometheus-operator authorization field (the modern replacement for the deprecated bearerTokenSecret). A new serviceMonitor.authSecret values block controls where the token comes from: authSecret.name/authSecret.key reference your own secret (a dedicated low privilege virtual key is recommended since whatever token Prometheus mounts can be replayed against the proxy), while authSecret.useMasterKey: true explicitly opts into scraping with the chart's master key secret, honoring masterkeySecretName/masterkeySecretKey with the same fallback logic as deployment.yaml. There is deliberately no silent fallback to the master key; if the ServiceMonitor is enabled without choosing a credential, rendering fails with a message listing the options, and setting authSecret.name without authSecret.key fails rather than guessing a key. authSecret.enabled: false restores the old credential-free endpoint for deployments that run with require_auth_for_metrics_endpoint: false

Chart version bumped to 1.2.0 and the new options are documented in the chart README

[greptile-apps]

Greptile Summary

This PR fixes the Helm chart's ServiceMonitor resource so that Prometheus scrapes of /metrics/ carry an Authorization: Bearer header, resolving persistent 401 rejections that occurred because require_auth_for_metrics_endpoint defaults to true. The fix is targeted at helm/litellm-helm after the chart was relocated in #32234.

• Adds a new serviceMonitor.authSecret values block with three explicit modes: a user-supplied secret (authSecret.name/authSecret.key), the chart's own master-key secret (authSecret.useMasterKey: true), or credential-free operation (authSecret.enabled: false). Enabling serviceMonitor.enabled: true without picking one of these modes fails at helm template render time with a clear error message rather than silently deploying broken scrapes.
• Uses the Prometheus Operator's modern authorization field (replacing the deprecated bearerTokenSecret), correctly structured as a SecretKeySelector referencing the chosen secret by name and key.

Confidence Score: 5/5

Safe to merge; the change is confined to the Helm chart and fixes a broken default with a well-designed fail-fast guard.

The template logic correctly handles all three credential modes and aborts rendering with a descriptive error when no mode is selected. The PR description includes concrete helm template and live-proxy proofs for each code path. The only gap is the absence of helm-unittest cases for the new servicemonitor.yaml branches, which is minor given the manual verification already provided.

helm/litellm-helm/templates/servicemonitor.yaml — the new authSecret branching logic has no corresponding unit test file, unlike every other chart template.

Important Files Changed

Filename	Overview
helm/litellm-helm/templates/servicemonitor.yaml	Adds authorization block to the ServiceMonitor endpoint with three paths (custom secret, master key, or fail-fast); logic is correct and the fail-fast default prevents silent 401 scrapes, but no unit tests cover the new branches
helm/litellm-helm/values.yaml	Adds authSecret block under serviceMonitor with sensible defaults (enabled: true, name/key empty, useMasterKey: false) that trigger the fail-fast guard when ServiceMonitor is enabled without credentials
helm/litellm-helm/Chart.yaml	Version bumped from 1.1.0 to 1.2.0 to reflect the new authSecret feature
helm/litellm-helm/README.md	Adds documentation rows for serviceMonitor.authSecret.* values to the README parameter table

_{Reviews (2): Last reviewed commit: "fix(helm): require explicit metrics scra..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing litellm_helm_servicemonitor_metrics_auth (a06517c) with litellm_internal_staging (7d15f2f)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (1280126) during the generation of this report, so 7d15f2f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mubashir1osmani]

@greptile-apps