feat(helm): add pod-hardening and migration-Job knobs to the componentized chart

[yassin-berriai]

TLDR

Problem this solves:

• Componentized chart exposes no pod-hardening knobs
• podSecurityContext, securityContext, sidecars, podLabels silently render nothing
• Migrations Job cannot get its own volumes or securityContext
• First install fails when the chart creates the backend ServiceAccount

How it solves it:

• Adds the six pod-hardening knobs to gateway, backend, ui
• Adds volumes, volumeMounts, podLabels, securityContext to the Job
• Job resolves its own ServiceAccount, defaulting past the hook cycle
• Every knob defaults to empty, so existing renders are unchanged

Relevant issues

Fixes #34252

The ServiceAccount half is the componentized-chart analogue of #21344 and #20571, which fixed the same pre-install hook cycle in helm/litellm-helm. This mirrors that chart's semantics and field names so operator values stay portable between the two

Linear ticket

Resolves LIT-4727

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This is a deploy artifact, so the proof is tool-native. Before runs are at base commit 0a42f28850 (extracted with git archive), after runs are at 4b9a37d86c for sections 1 and 2 and at 01b4def385 for sections 3, 4 and 5. The only differences between those two are the podLabels selector guard in section 3 and the ServiceAccount token fix in section 4

1. The silent no-op

Same hardened values file rendered against both charts, counting occurrences in the output of helm template lit ./helm/litellm -f hardened.yaml. The values file sets pod-level and container-level securityContext, an auth sidecar, podLabels, lifecycle hooks, and terminationGracePeriodSeconds on all three components plus the Job

                              BEFORE   AFTER
securityContext                  0       8
runAsNonRoot                     0       4
readOnlyRootFilesystem           0       4
auth-sidecar (extraContainers)   0       3
egress-policy (podLabels)        0       4
lifecycle:                       0       3
terminationGracePeriodSeconds    0       3

Helm does not error on unknown values, so before this PR that render is green with none of the hardening applied

2. The first-install failure

kind cluster, dedicated namespace, an in-cluster Postgres so the migration actually runs, serviceAccounts.backend.create: true, gateway.enabled: false, backend.enabled: false, ui.enabled: false, migrationJob.backoffLimit: 0, migrations image tag v1.94.1

Before, at 0a42f28850:

$ helm install lit ./helm/litellm -n wt4727 -f live.yaml --timeout 3m
Error: INSTALLATION FAILED: failed pre-install: resource Job/wt4727/lit-litellm-migrations not ready. status: InProgress, message: Job in progress

$ kubectl -n wt4727 get events | grep migrations
Warning  FailedCreate  job/lit-litellm-migrations  Error creating: pods "lit-litellm-migrations-" is forbidden: error looking up service account wt4727/lit-litellm-backend: serviceaccount "lit-litellm-backend" not found

$ kubectl -n wt4727 get pods
postgres-759695db99-gsbp4   1/1   Running   1   5m30s

The Job references the backend ServiceAccount, which is an ordinary Sync-phase resource, so during the pre-install hook it does not exist yet and the pod is never created. ArgoCD maps helm.sh/hook: pre-install,pre-upgrade to PreSync and hits the same cycle

After, at 4b9a37d86c, same values file:

$ helm install lit ./helm/litellm -n wt4727 -f live.yaml --timeout 10m
NAME: lit
STATUS: deployed

$ kubectl -n wt4727 get pods
lit-litellm-migrations-m5q2d   0/1   Completed   0   16s
postgres-759695db99-gsbp4      1/1   Running     2   10m

$ kubectl -n wt4727 get job lit-litellm-migrations -o jsonpath='{.status.succeeded}'
1

$ kubectl -n wt4727 logs lit-litellm-migrations-m5q2d | tail -3
All migrations have been successfully applied.
2026-08-01 19:57:27,469 - litellm_proxy_extras - INFO - Migration job completed successfully.

$ kubectl -n wt4727 get pod lit-litellm-migrations-m5q2d -o jsonpath='...'
serviceAccountName=default
podSecurityContext={"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":{"type":"RuntimeDefault"}}
containerSecurityContext={"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}
podLabels include egress-policy=restricted
volumes=tmp   volumeMounts=/tmp

$ kubectl -n wt4727 get sa
default               10m
lit-litellm-backend   11s

The discriminator is explicit: before, the Job pod is never created at all and the install dies on a forbidden pod creation; after, the pod is created, runs as default with the requested hardening on it, and the migration applies cleanly against a real Postgres. The 11s age on lit-litellm-backend is the created ServiceAccount arriving in the Sync phase, which is to say after the hook already needed it

3. podLabels cannot quietly break the selector

The three selector keys are also emitted on the pod template, so a podLabels entry reusing one used to render a duplicate YAML key whose later value wins, leaving the pod template no longer matching the immutable selector. helm lint, helm template, and helm unittest are all happy with that, and the apiserver is not. Server-side dry-run against a kind cluster, rendering gateway.podLabels: {app.kubernetes.io/component: not-gateway} from the chart at 4b9a37d86c:

$ kubectl -n wt4727b apply --dry-run=server -f collide.yaml
The Deployment "lit-litellm-gateway" is invalid: spec.template.metadata.labels: Invalid value: {"app.kubernetes.io/component":"not-gateway","app.kubernetes.io/instance":"lit","app.kubernetes.io/name":"litellm"}: `selector` does not match template `labels`

The chart now refuses at template time, naming the key and the component, rather than dropping it silently:

$ helm template lit ./helm/litellm --set 'gateway.podLabels.app\.kubernetes\.io/component=not-gateway' ...
Error: execution error at (litellm/templates/gateway/deployment.yaml:25:12): gateway.podLabels cannot set app.kubernetes.io/component: it is part of the Deployment's immutable selector

The migrations Job is deliberately left permissive, and that was verified rather than assumed. Its selector is generated by the controller, so the same override is accepted:

$ kubectl -n wt4727b apply --dry-run=server -f jobcollide.yaml
job.batch/lit-litellm-migrations created (server dry run)

$ kubectl -n wt4727b get job lit-litellm-migrations -o jsonpath='{.spec.selector.matchLabels}'
{"batch.kubernetes.io/controller-uid":"48ef3b5a-a1a2-4369-a62c-78c17b89e139"}

The Job merges podLabels over the chart labels rather than appending them, so an override there replaces the key instead of rendering it twice. A duplicate key is worth avoiding even where the apiserver tolerates it: helm-unittest's parser refuses to load such a document at all (mapping key "app.kubernetes.io/component" already defined), so appending would have left that path untestable

4. The Job no longer inherits a ServiceAccount token it should not have

Resolving the Job's ServiceAccount independently changes which account it runs as, and that changes what it inherits. With serviceAccounts.backend.create: true and automount: false the Job used to run as the created backend account, which the chart renders with automountServiceAccountToken: false, so no token was projected. Falling back to default would have handed it one, which is a regression this PR would have introduced in exactly the hardened configuration it exists to serve

$ helm template ... --set serviceAccounts.backend.create=true --set serviceAccounts.backend.automount=false
kind: ServiceAccount
  name: lit-litellm-backend
automountServiceAccountToken: false
...
      serviceAccountName: default
      automountServiceAccountToken: false

migrationJob.automountServiceAccountToken defaults to false and closes it. The Job runs prisma migrate deploy against Postgres and never calls the Kubernetes API, which is the same reasoning the chart already applies to the ui ServiceAccount. It is a value rather than a literal so IAM database auth has an escape hatch; that path should not need the default token mount, since EKS Pod Identity injects its own projected token volume and GKE Workload Identity goes through the metadata server, and an operator who finds otherwise can flip the key instead of forking the chart

5. Gates

$ helm lint ./helm/litellm --set database.writer.host=h --set database.writer.dbname=d
1 chart(s) linted, 0 chart(s) failed

$ helm unittest -f 'tests/*.yaml' helm/litellm
Test Suites: 6 passed, 6 total
Tests:       59 passed, 59 total

The two new suites were also run against the base chart to confirm they are mutation-sensitive: 22 of the 27 new tests fail at 0a42f28850 and pass here. The 5 that pass on both sides are deliberate, and they are the ones asserting that nothing renders by default and that a ServiceAccount configuration that works today keeps resolving to the same name

Type

🆕 New Feature

🐛 Bug Fix

Changes

Each of gateway, backend, and ui gains podSecurityContext, container-level securityContext, extraContainers, podLabels, lifecycle, and terminationGracePeriodSeconds. The migrations Job gains volumes, volumeMounts, podLabels, podSecurityContext, and securityContext

Everything defaults to empty and renders nothing, so an existing release re-renders byte for byte. A hardened default was tempting for securityContext, and rejected: a non-empty default would change the runtime of every deployment that upgrades into it, which is not something a chart should do silently. terminationGracePeriodSeconds follows the same rule and inherits the Kubernetes default of 30 when unset, rather than the legacy chart's 90; an explicit 0 is honored instead of being swallowed as an empty value. extraContainers renders through tpl, matching helm/litellm-helm, so sidecar entries can reference chart values and release metadata

podLabels on a Deployment is validated against that Deployment's selector. The three selector keys are also emitted on the pod template, so reusing one would render a duplicate YAML key, the later value would win, and the apiserver would reject the release for a pod template that no longer matches its own selector. A shared helper fails at template time naming the offending key and component, which follows what the chart already does elsewhere for invalid value combinations. Silently dropping the key was the alternative and it is the same wart as ignoring an operator's ServiceAccount name, so it renders an error instead. The migrations Job stays permissive here on purpose, since a Job's selector is generated by the controller rather than declared; it merges podLabels over the chart labels so an override replaces a key rather than duplicating it

The migrations Job now resolves its ServiceAccount through a helper of its own. migrationJob.serviceAccountName always wins; with it empty the Job falls back to default when serviceAccounts.backend.create is true, and otherwise keeps borrowing the backend name, which in that branch is either an existing account the operator supplied or default. Both of those already exist when the hook runs, so every configuration that works today resolves to exactly the name it resolves to now

Resolving the Job's identity independently also changes what that pod inherits, so migrationJob.automountServiceAccountToken closes the gap it would otherwise open. With serviceAccounts.backend.create: true and automount: false the Job used to run as the created backend account and get no projected token, and falling back to default would have handed it one, in precisely the hardened configuration this PR serves. It defaults to false, since the Job talks to Postgres and never calls the Kubernetes API, which is the reasoning the chart already applies to the ui ServiceAccount. It is a value rather than a literal so that IAM database auth keeps an escape hatch, per section 4

One upgrade caveat, documented next to the key in values.yaml: a release already running with serviceAccounts.backend.create: true used to hand the Job the created backend account on every upgrade, since by then it exists, and now hands it default. If that Job relies on IRSA or Workload Identity annotations, for instance under database.writer.useIAMAuth, set migrationJob.serviceAccountName to an account that already exists

readOnlyRootFilesystem: true on the Job is now settable, and running the Job that way also needs the migrations image from #35485, which bakes the prisma CLI and engines offline. On today's published image the Job still writes under $HOME at startup, so the chart knobs alone do not deliver a read-only Job until that lands

Tests live in helm/litellm/tests/pod_hardening_tests.yaml and helm/litellm/tests/migration_job_tests.yaml, both picked up by the existing helm unittest -f 'tests/*.yaml' helm/litellm step in .github/workflows/helm_unit_test.yml. They assert rendered values rather than key presence, and cover every branch of the ServiceAccount helper: name set, create true, create false with an explicit name, and neither

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR adds pod-hardening options to the gateway, backend, UI, and migration Job while independently resolving the migration Job’s ServiceAccount.

• Adds pod and container security contexts, labels, sidecars, lifecycle hooks, volumes, mounts, and termination grace periods.
• Prevents custom Deployment pod labels from overriding immutable selector labels.
• Adds migration Job ServiceAccount and token-mount controls with coverage for the new rendering branches.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the prior selector mismatch is prevented for gateway, backend, and UI across all immutable selector keys.

Important Files Changed

Filename	Overview
helm/litellm/templates/_helpers.tpl	Adds migration ServiceAccount resolution and complete reserved-selector-label validation for all component Deployments.
helm/litellm/templates/gateway/deployment.yaml	Renders the new gateway hardening options and applies the selector-label validation that resolves the prior finding.
helm/litellm/templates/backend/deployment.yaml	Renders backend hardening options and uses the shared selector-label guard.
helm/litellm/templates/ui/deployment.yaml	Renders UI hardening options and uses the shared selector-label guard.
helm/litellm/templates/migrations-job.yaml	Adds independent ServiceAccount resolution, token control, pod labels, security contexts, volumes, and mounts to the migration Job.
helm/litellm/values.yaml	Defines empty-by-default values for the new component and migration Job controls and documents the ServiceAccount behavior.
helm/litellm/tests/pod_hardening_tests.yaml	Covers hardening rendering, zero-valued grace periods, sidecar templating, and all three reserved selector keys across the component Deployments.
helm/litellm/tests/migration_job_tests.yaml	Covers migration ServiceAccount branches, token mounting, hardening fields, volumes, and permissive Job-label overrides.

_{Reviews (3): Last reviewed commit: "feat(helm): add pod-hardening and migrat..." | Re-trigger Greptile}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please re-review at c628a0f. The podLabels selector collision is fixed: a shared helper now fails at template time naming the key and component, instead of emitting a duplicate label key that the apiserver rejects. The migrations Job stays permissive, verified with a server-side dry-run, since its selector is controller-generated; it merges podLabels over the chart labels so an override replaces a key rather than duplicating it.

[yassin-berriai]

@greptileai please re-review at 01b4def. Since your 5/5 at c628a0f this adds one thing: migrationJob.automountServiceAccountToken, defaulting to false, so the Job stops inheriting a projected API token when the ServiceAccount fallback sends it to the namespace default account while the operator disabled automounting on the backend. It is a value rather than a hardcoded literal so IAM database auth keeps an escape hatch. Three unit tests cover the default, an explicit true, and the backend-disabled configuration.