fix(helm): stop migrations Job pods matching the litellm Service selector

[JoaVirtudes19]

The problem

helm/litellm-helm's migrations Job labels its pods with litellm.labels, which embeds litellm.selectorLabels:

# templates/migrations-job.yaml
  template:
    metadata:
      labels:
        {{- include "litellm.labels" . | nindent 8 }}   # -> name + instance

That is the exact selector of four other objects in the same chart:

Object	Selector
service.yaml	litellm.selectorLabels
service-metrics.yaml	litellm.selectorLabels
servicemonitor.yaml	litellm.selectorLabels
poddisruptionbudget.yaml	litellm.selectorLabels

So every migrations pod is selected by all four, even though its prisma-migrations container exposes no ports. The litellm Service resolves a named targetPort: http that the pod cannot satisfy, so each pre-upgrade / PreSync run perturbs the Service's EndpointSlices for the lifetime of the Job pod.

How it surfaced

On a cluster fronted by NGINX Ingress Controller 5.3.4 (OSS, NGINX 1.29.5), that was enough to make the controller compute zero endpoints for the litellm upstream and serve its no-endpoints placeholder for every request:

"POST /responses HTTP/1.1" 502 150 ... upstream_addr=127.0.0.1:8181 upstream_status=502
request_time=0.000 upstream_connect_time=- upstream_header_time=-

127.0.0.1:8181 is the controller's internal "service has no active endpoints" backend. It kept returning 502 long after the Job pod had exited (~12 s) and the new proxy pods were healthy — in one case for 63 minutes, across every controller pod, until the ingress controller itself was restarted. Correlation over 14 days was exact: every 502 burst followed a migrations Job, and pod replacements without a chart sync produced none.

The same overlap is why pdb.minAvailable can only take an integer in this chart: a percentage would be computed against a pod count that includes a Job pod which never serves traffic.

The fix

Give the migrations pods their own labels — app.kubernetes.io/name: <name>-migrations plus app.kubernetes.io/component: migrations — mirroring what the newer helm/litellm chart already does for its own migrations Job (litellm.commonLabels + a component label, where every component selector carries a component).

Two notes on why it is done this way:

• Overriding app.kubernetes.io/name is what breaks the match. Adding only app.kubernetes.io/component: migrations would not: a Service selector matches on a subset, so a pod carrying name + instance + component still matches a {name, instance} selector.
• The proxy Deployment's selector cannot be narrowed instead. Adding a component: proxy label to litellm.selectorLabels would change Deployment.spec.selector, which is immutable — helm upgrade on any existing release would fail outright, and the Services would briefly select nothing mid-upgrade.

podLabels are now merged rather than appended, so an override replaces a chart label instead of rendering the key twice. This is the same pattern and comment already used in helm/litellm/templates/migrations-job.yaml.

The Job object's own metadata.labels are deliberately left untouched, to keep the change to the one thing that causes the bug.

Compatibility

The migrations pods' app.kubernetes.io/name changes from litellm to litellm-migrations, and they gain app.kubernetes.io/component: migrations. Anything selecting migrations pods by app.kubernetes.io/name=litellm (a custom NetworkPolicy or log query, say) would need updating — which is precisely the overlap being removed. kubectl get jobs -l app.kubernetes.io/name=litellm is unaffected.

A Job's spec.template is immutable, so an in-place update of an existing Job would be rejected. Both hook paths delete and recreate the Job (helm.sh/hook-delete-policy: before-hook-creation / argocd.argoproj.io/hook-delete-policy: BeforeHookCreation) plus ttlSecondsAfterFinished, so they are unaffected. Running with both hooks disabled already fails to upgrade today, because the Job's checksum/config pod annotation changes with any values change and is equally immutable.

Testing

Two cases added to helm/litellm-helm/tests/migrations-job_tests.yaml:

• the pods do not carry the Service selector labels (name is litellm-migrations, component is migrations, instance still the release)
• podLabels still land on the pods and can override a chart label without duplicating the key

$ helm unittest -f 'tests/*.yaml' helm/litellm-helm
Charts:      1 passed, 1 total
Test Suites: 14 passed, 14 total
Tests:       113 passed, 113 total

No new suite is declared, so the helm_unit_test.yml declared-vs-executed guard still balances at 14. helm/litellm is untouched.

Rendered before/after for the Job's pod labels:

   template:
     metadata:
       labels:
-        helm.sh/chart: litellm-helm-1.1.3
-        app.kubernetes.io/name: litellm
-        app.kubernetes.io/instance: r
-        app.kubernetes.io/version: "v1.85.1"
-        app.kubernetes.io/managed-by: Helm
+        app.kubernetes.io/component: migrations
+        app.kubernetes.io/instance: r
+        app.kubernetes.io/managed-by: Helm
+        app.kubernetes.io/name: litellm-migrations
+        app.kubernetes.io/version: v1.85.1
+        helm.sh/chart: litellm-helm-1.1.3

[CLAassistant]

All committers have signed the CLA.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing JoaVirtudes19:fix/helm-migrations-job-service-selector-overlap (c86ccdf) with main (328a5f5)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This pull request contains a broad set of changes beyond the stated Helm fix, including migration-pod label isolation, managed Bedrock file deletion, Azure AI pricing and transcription updates, MCP lifecycle and logging changes, guardrail scan metadata, spend-log metadata recovery, and extensive E2E coverage.

• Introduces dedicated labels for migration pods and merges global pod label overrides.
• Adds Bedrock S3 file deletion and normalizes managed-file deletion responses.
• Revises Azure AI Model Router accounting and adds catalog pricing entries.
• Expands MCP persistence, toolset enforcement, lifecycle logging, and replica-convergence tests.
• Adds guardrail scan metadata headers and spend-log-based key metadata recovery.

Confidence Score: 3/5

The pull request is not safe to merge until migration labels remain valid for long name overrides and cannot be changed back to the proxy Service selector by global pod labels.

Two reachable Helm configurations break the core migration fix: long supported names render an invalid Kubernetes label, while a global name label can make migration pods match the Service selector again.

Files Needing Attention: helm/litellm-helm/templates/_helpers.tpl, helm/litellm-helm/templates/migrations-job.yaml

Important Files Changed

Filename	Overview
helm/litellm-helm/templates/_helpers.tpl	Adds dedicated migration-pod labels, but the suffixed name can violate Kubernetes label-length limits.
helm/litellm-helm/templates/migrations-job.yaml	Merges migration labels with global pod labels, but override precedence can restore the Service-selector collision.
helm/litellm-helm/tests/migrations-job_tests.yaml	Adds direct coverage for default migration labels and global label overrides, but does not cover long names or selector-restoring overrides.
enterprise/litellm_enterprise/proxy/hooks/managed_files.py	Forwards deployment credentials for managed deletion and returns the unified managed file identifier.
litellm/llms/bedrock/files/transformation.py	Implements bucket-restricted, SigV4-signed S3 file deletion through the shared files handler.
litellm/llms/azure_ai/cost_calculator.py	Separates routed-model token pricing from the Azure Model Router fee and avoids duplicate charges.
litellm/proxy/_experimental/mcp_server/server.py	Adds terminal logging for MCP proxy calls, adjusts manager shutdown order, and resolves namespaced server metadata.
litellm/proxy/spend_tracking/key_metadata_recovery.py	Adds bounded, cached SpendLogs fallback queries for recovering missing key metadata.
litellm/proxy/common_utils/callback_utils.py	Records structured guardrail scan metadata and emits it through a bounded response header.

_{Reviews (1): Last reviewed commit: "fix(helm): stop migrations Job pods matc..." | Re-trigger Greptile}

[greptile-apps]

Migration Label Exceeds Limit

A nameOverride can already be 63 characters because litellm.name truncates it to that limit. Appending -migrations afterward can make app.kubernetes.io/name 74 characters long. Kubernetes rejects label values longer than 63 characters, so a release using a long supported override cannot create the migration pod and its installation or upgrade fails.

Suggested change

	app.kubernetes.io/name: {{ include "litellm.name" . }}-migrations
	app.kubernetes.io/name: {{ printf "%s-migrations" (include "litellm.name" .) | trunc 63 | trimSuffix "-" }}

Knowledge Base Used: Bound Helm migration Jobs so blocked releases fail visibly

[JoaVirtudes19]

Good catch, fixed. app.kubernetes.io/name is now built as printf "%s-migrations" (include "litellm.name" .) | trunc 63 | trimSuffix "-". Added a regression test with a 60-character nameOverride, which produced a 71-character label before the fix.

[greptile-apps]

Overrides Restore Service Match

The merge lets the global podLabels map override app.kubernetes.io/name. An existing setting such as podLabels.app.kubernetes.io/name: litellm therefore restores the Service selector on migration pods. Because the same global map is used for deployment pods, this is a realistic configuration that defeats the fix and puts migration pods back into the Service and PodDisruptionBudget selections.

Knowledge Base Used: Bound Helm migration Jobs so blocked releases fail visibly

[JoaVirtudes19]

Agreed, that defeated the fix. podLabels still wins for every other key, but app.kubernetes.io/name is now forced back from the chart labels after the merge, since that key is what the Service, metrics Service, ServiceMonitor and PDB select on. Added a regression test that sets podLabels."app.kubernetes.io/name": litellm and asserts the pods stay out of the selector.

[JoaVirtudes19]

@yassin-berriai @yuneng-berri mind taking a look at this one?

Quick heads-up on the red checks: this PR only touches Helm templates and their unit tests, yet Vertex AI, core-utils and proxy-behavior fail — they're broken on the base right now. Same three show up on unrelated PRs like #41396 and #41041, with fixes already open (#41415, #41278, #41019).