fix(helm): bound the migrations Job so a blocked migration cannot stall the release

[yuneng-berri]

TLDR

Problem this solves:

• A blocked schema migration hangs the release forever
• backoffLimit never fires: the pod never terminates
• Operators must delete the Job by hand to recover
• Unrelated gateway, backend and UI changes silently stop shipping

How it solves it:

• Add migrationJob.activeDeadlineSeconds to both charts
• Default 1800s, so a stuck Job fails instead of hanging
• Guarded by with, so null restores the old behaviour

User Flow

Before: an operator upgrading the chart, whose migration blocks on the database, gets an upgrade that never returns

1. They run helm upgrade litellm ./helm/litellm -f values.yaml
2. The migration Job starts and its container blocks talking to the database
3. The command sits waiting on the pre-upgrade hook and never returns
4. kubectl get job litellm-migrations still reads Running 0/1 hours later, with no conditions set
5. Nothing else in the release rolls out, so unrelated gateway, backend and UI changes never reach the cluster
6. The only way out is kubectl delete job litellm-migrations by hand

After: the same upgrade gives up on its own, so the operator gets a failure they can act on instead of a hang

1. They run the same helm upgrade litellm ./helm/litellm -f values.yaml
2. The migration Job starts and blocks on the database exactly as before
3. After migrationJob.activeDeadlineSeconds (1800 by default) the Job reports Failed, reason DeadlineExceeded
4. helm upgrade returns non-zero naming the failed hook rather than waiting forever
5. The rest of the release is no longer pinned behind a Job that cannot finish, so the next attempt can proceed once the database issue is addressed
6. An operator who wants the old unbounded behaviour sets migrationJob.activeDeadlineSeconds: null

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

There is no HTTP route to exercise here, so the proof is the rendered manifest plus a live Kubernetes cluster showing the two Jobs behave differently.

1. The field renders, in both charts. Captured at ae3e19a83f

$ helm template t helm/litellm -f helm/litellm/tests/values/required.yaml | grep -A1 ttlSecondsAfterFinished
  ttlSecondsAfterFinished: 120
  activeDeadlineSeconds: 1800

$ helm template t helm/litellm-helm | grep -B1 -A1 activeDeadlineSeconds
  ttlSecondsAfterFinished: 120
  backoffLimit: 4
  activeDeadlineSeconds: 1800

2. A blocked migration now terminates. Live kind cluster, two Jobs identical except for the new field, each running a container that starts and then blocks the way a migration waiting on an unreachable database does. The 30s deadline stands in for the 1800s default so the run is observable

$ kubectl apply -f before.yaml -f after.yaml
job.batch/mig-no-deadline-v2 created
job.batch/mig-with-deadline-v2 created

$ kubectl logs job/mig-no-deadline-v2
connecting to database...

$ for i in 1 2 3 4 5; do sleep 15; echo "t=$((i*15))s"; kubectl get jobs --no-headers; done
t=15s
mig-no-deadline-v2     Running   0/1   20s
mig-with-deadline-v2   Running   0/1   20s
t=30s
mig-no-deadline-v2     Running         0/1   35s
mig-with-deadline-v2   FailureTarget   0/1   35s
t=60s
mig-no-deadline-v2     Running   0/1   65s
mig-with-deadline-v2   Failed    0/1   65s
t=75s
mig-no-deadline-v2     Running   0/1   80s
mig-with-deadline-v2   Failed    0/1   80s

Terminal state of each:

$ kubectl get job mig-with-deadline-v2 -o jsonpath='{range .status.conditions[*]}type={.type} reason={.reason} message={.message}{"\n"}{end}'
type=FailureTarget reason=DeadlineExceeded message=Job was active longer than specified deadline
type=Failed reason=DeadlineExceeded message=Job was active longer than specified deadline

$ kubectl get job mig-no-deadline-v2 -o jsonpath='conditions={.status.conditions} active={.status.active}'
conditions= active=1

The Job without the field never reaches a terminal state, which is what leaves a release waiting on it. The Job with the field fails on its own.

3. The tests fail without the change. Reverting only the two templates and the two values files, keeping the new tests, at base f03df1bb42

$ helm unittest -f 'tests/*.yaml' helm/litellm
 FAIL  test migrations Job ServiceAccount resolution and pod hardening
	- bounds the Job with a deadline by default, so a blocked migration cannot stall the release forever
	- honours an operator-supplied deadline
Tests:       2 failed, 72 passed, 74 total

The third new test asserts the field is absent when nulled, which passes either way, so it was mutation-tested separately: dropping the with guard so the field always renders makes it fail.

$ helm unittest -f 'tests/migration_job_tests.yaml' helm/litellm
 FAIL	- omits the deadline entirely when it is nulled out, restoring the unbounded behaviour
Tests:       1 failed, 15 passed, 16 total

Restored, both suites are green at ae3e19a83f

$ helm unittest -f 'tests/*.yaml' helm/litellm
Tests:       74 passed, 74 total
$ helm unittest -f 'tests/*.yaml' helm/litellm-helm
Tests:       97 passed, 97 total

Type

🐛 Bug Fix

Caveats (if any)

• The container blocking was simulated, not a genuinely wedged database connection
• Default changes behaviour: migrations running past 1800s now fail
• Measured healthy runs finish in about 30s, so headroom is roughly 60x
• The budget is shared across all backoffLimit retries, not per attempt
• A killed migration can leave a partially applied entry needing manual resolution
• Chart.yaml untouched, matching recent template-only changes to litellm-helm

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

Adds a configurable whole-Job deadline to migration Jobs in both Helm charts so blocked migrations eventually fail instead of indefinitely stalling release hooks.

• Defaults the deadline to 1800 seconds in both charts.
• Allows operators to restore unbounded behavior by setting the value to null.
• Adds tests for the default, custom, and disabled configurations.

Confidence Score: 5/5

The PR appears safe to merge because the bounded migration behavior is configurable, consistently implemented across both charts, and covered by focused rendering tests.

The templates render the intended default and custom deadlines while omitting the field for the documented null opt-out, with no concrete blocking defect remaining.

Important Files Changed

Filename	Overview
helm/litellm/templates/migrations-job.yaml	Conditionally renders the configured deadline on the pre-install and pre-upgrade migration Job.
helm/litellm/values.yaml	Defines and explains the 1800-second default and null opt-out.
helm/litellm/tests/migration_job_tests.yaml	Covers default, overridden, and omitted deadline rendering.
helm/litellm-helm/templates/migrations-job.yaml	Applies the same conditional deadline to the legacy chart's migration Job.
helm/litellm-helm/values.yaml	Defines the legacy chart's matching deadline configuration.
helm/litellm-helm/tests/migrations-job_tests.yaml	Verifies matching deadline behavior in the legacy chart.

_{Reviews (1): Last reviewed commit: "fix(helm): bound the migrations Job so a..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!