Skip to content

fix(terraform): keep prior state when a virtual key update is rejected - #40512

Merged
yassin-berriai merged 1 commit into
litellm_internal_stagingfrom
litellm_terraform_key_update_failure_keeps_prior_state
Sep 10, 2026
Merged

yassin-berriai merged 1 commit into
litellm_internal_stagingfrom
litellm_terraform_key_update_failure_keeps_prior_state

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • A rejected /key/update still writes the planned values into Terraform state
  • The next terraform plan says "No changes", so the update is silently lost

How it solves it:

  • resourceKeyUpdate calls d.Partial(true) when the update fails
  • The SDK then keeps the prior state instead of the rejected plan
  • Regression test drives the real SDK Apply and fails without the fix

User Flow

Before: an operator changes a virtual key in Terraform, the proxy rejects it, and Terraform then behaves as if the change had been applied

  1. They set budget_duration = "bad" on an existing litellm_key and run terraform apply
  2. The proxy answers POST http://localhost:16398/key/update with 400 Invalid budget_duration 'bad' and terraform apply exits 1 with that error
  3. terraform state show litellm_key.demo nevertheless shows budget_duration = "bad"
  4. They run terraform plan again and get "No changes. Your infrastructure matches the configuration."
  5. GET http://localhost:16398/key/info?key=<token_id> still returns "budget_duration": null, so state and the proxy disagree and the change is gone

After: the same rejected update leaves state untouched, so Terraform keeps trying until the config is valid

  1. They set budget_duration = "bad" on an existing litellm_key and run terraform apply
  2. The proxy answers POST http://localhost:16398/key/update with 400 Invalid budget_duration 'bad' and terraform apply exits 1 with that error
  3. terraform state show litellm_key.demo shows no budget_duration (the prior state)
  4. They run terraform plan again and it still proposes the in-place update (Plan: 0 to add, 1 to change, 0 to destroy)
  5. GET http://localhost:16398/key/info?key=<token_id> returns "budget_duration": null, matching state

Relevant issues

Upstream report: BerriAI/terraform-provider-litellm#55

Linear ticket

Resolves LIT-6398

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Setup shared by both runs: Postgres in Docker plus the proxy from this checkout (PYTHONPATH=<checkout> uv run --no-sync litellm --config <cfg> --port 16398 --use_prisma_db_push, one fake-model deployment). The provider binary is built with (cd terraform/provider && go build -o <out> .) from the merge base and from the PR tip, and Terraform 1.9.8 loads it through dev_overrides. main.tf and the starting terraform.tfstate are byte-identical across both runs (sha256 ba853a6f... and f5df4c7f...), and the proxy is the same process for both. The build serving each run is identified by go version -m <binary>, which prints the git revision it was compiled from

resource "litellm_key" "demo" {
  key_alias       = "lit6398-demo"
  models          = ["fake-model"]
  budget_duration = var.budget_duration
}

Before (b22ca7a)

  1. Confirm the build

    $ go version -m bin/active/terraform-provider-litellm | grep vcs
    	build	vcs.revision=b22ca7ac6dad77ce7f377d77dba34fc95a64e405
    	build	vcs.modified=false
    
  2. Starting point on the proxy

    $ curl -s http://localhost:16398/key/info?key=$TOKEN_ID -H 'Authorization: Bearer <master>' | jq '.info | {key_alias, budget_duration}'
    {"key_alias": "lit6398-demo", "budget_duration": null}
    
  3. Apply an update the proxy rejects

    $ terraform apply -auto-approve -no-color -var budget_duration=bad
      # litellm_key.demo will be updated in-place
      ~ resource "litellm_key" "demo" {
          + budget_duration  = "bad"
            id               = "ff0d616f4572db90d713e354311c44bb20216cbed1937d6ad78e91acc1d69a8c"
            # (9 unchanged attributes hidden)
        }
    Plan: 0 to add, 1 to change, 0 to destroy.
    litellm_key.demo: Modifying... [id=ff0d616f4572db90d713e354311c44bb20216cbed1937d6ad78e91acc1d69a8c]
    Error: error updating key: API request failed with status code 400: {"error":{"message":"{'error': \"Invalid budget_duration 'bad'. Use a format like '1h', '24h', '7d', or '30d'.\"}","type":"auth_error","param":"None","code":"400"}}
    exit=1
    
  4. State recorded the rejected value anyway

    $ terraform state show -no-color litellm_key.demo | grep budget_duration
        budget_duration  = "bad"
    
  5. The next plan thinks everything is applied

    $ terraform plan -no-color -var budget_duration=bad
    No changes. Your infrastructure matches the configuration.
    
  6. The proxy never received the change

    $ curl -s http://localhost:16398/key/info?key=$TOKEN_ID -H 'Authorization: Bearer <master>' | jq '.info | {key_alias, budget_duration}'
    {"key_alias": "lit6398-demo", "budget_duration": null}
    

After (08152ee)

  1. Confirm the build

    $ go version -m bin/active/terraform-provider-litellm | grep vcs
    	build	vcs.revision=08152ee5d6d5feb22013cb3aee5f28f5e454c85b
    	build	vcs.modified=false
    
  2. Starting point on the proxy

    $ curl -s http://localhost:16398/key/info?key=$TOKEN_ID -H 'Authorization: Bearer <master>' | jq '.info | {key_alias, budget_duration}'
    {"key_alias": "lit6398-demo", "budget_duration": null}
    
  3. Apply the same rejected update

    $ terraform apply -auto-approve -no-color -var budget_duration=bad
      # litellm_key.demo will be updated in-place
      ~ resource "litellm_key" "demo" {
          + budget_duration  = "bad"
            id               = "ff0d616f4572db90d713e354311c44bb20216cbed1937d6ad78e91acc1d69a8c"
            # (9 unchanged attributes hidden)
        }
    Plan: 0 to add, 1 to change, 0 to destroy.
    litellm_key.demo: Modifying... [id=ff0d616f4572db90d713e354311c44bb20216cbed1937d6ad78e91acc1d69a8c]
    Error: error updating key: API request failed with status code 400: {"error":{"message":"{'error': \"Invalid budget_duration 'bad'. Use a format like '1h', '24h', '7d', or '30d'.\"}","type":"auth_error","param":"None","code":"400"}}
    exit=1
    
  4. State kept the prior values

    $ terraform state show -no-color litellm_key.demo | grep budget_duration
    (budget_duration not in state)
    
  5. The next plan still wants to apply the change

    $ terraform plan -no-color -var budget_duration=bad
      ~ update in-place
    Plan: 0 to add, 1 to change, 0 to destroy.
    
  6. Proxy and state agree

    $ curl -s http://localhost:16398/key/info?key=$TOKEN_ID -H 'Authorization: Bearer <master>' | jq '.info | {key_alias, budget_duration}'
    {"key_alias": "lit6398-demo", "budget_duration": null}
    
  7. Control: a valid update on the fixed build still goes through

    $ terraform apply -auto-approve -no-color -var budget_duration=30d
    Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
    exit=0
    $ terraform state show litellm_key.demo | grep budget_duration
        budget_duration  = "30d"
    $ curl -s http://localhost:16398/key/info?key=$TOKEN_ID -H 'Authorization: Bearer <master>' | jq '.info | {key_alias, budget_duration}'
    {"key_alias": "lit6398-demo", "budget_duration": "30d"}
    

Type

🐛 Bug Fix

Caveats (if any)

Medium

  • Scope is litellm_key only, as the ticket asks
  • The other 22 resources in this provider have their own update functions and none call d.Partial(true), so they share the same SDK behavior
    • Confirmed on litellm_team with a throwaway test: a 400 from /team/update still lands the planned team_alias in state
    • Those need the same one-line change each, in a follow-up

Low

  • The ticket says the failed apply is "recorded as successful"; on the current provider terraform apply does already exit 1 with the proxy error. What is wrong is the state written after that error, which is what this PR fixes
  • The non-required osv-scan check is red for a smol-toml dev dependency in ui/litellm-dashboard/package-lock.json. That lockfile is untouched here and the same check fails on every PR opened against the base branch in the same window, so it is base drift, not this change

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/7c22271f9676444688e115bbf10168ab
Open in Devin Desktop: https://app.devin.ai/desktop/session/7c22271f9676444688e115bbf10168ab?variant=devin
Requested by: @yassin-berriai

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR preserves the prior Terraform state when a litellm_key update is rejected.

  • Enables partial-state handling before returning an update error.
  • Adds an SDK-level regression test covering failed updates.
  • Removes the explanatory source comments identified in the previous review.

Confidence Score: 5/5

The PR appears safe to merge; no actionable regressions or outstanding findings remain.

The update failure path now preserves prior state, and the regression test verifies the rejected planned value is omitted while existing attributes remain. The previous source-comment finding was manually resolved without explanation, and the comments have been removed from the current head.

Important Files Changed

Filename Overview
terraform/provider/litellm/resource_key.go Marks failed key updates as partial so Terraform retains the prior state.
terraform/provider/litellm/resource_key_test.go Verifies through the Terraform SDK Apply path that rejected values are not persisted.

Reviews (2): Last reviewed commit: "fix(terraform): keep prior state when a ..." | Re-trigger Greptile

Comment thread terraform/provider/litellm/resource_key.go Outdated
resourceKeyUpdate now calls d.Partial(true) on a failed /key/update so the SDK does not persist the rejected planned values into state, which made the next plan report no changes and silently dropped the update.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot force-pushed the litellm_terraform_key_update_failure_keeps_prior_state branch from f76bcdc to 08152ee Compare September 10, 2026 01:52
@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai removed both explanatory comments per the repo comment policy; re-review the squashed head 08152ee please.

@yassin-berriai
yassin-berriai merged commit fb603fd into litellm_internal_staging Sep 10, 2026
83 of 85 checks passed
@yassin-berriai
yassin-berriai deleted the litellm_terraform_key_update_failure_keeps_prior_state branch September 10, 2026 02:48
SK2iP added a commit to SK2iP/litellm that referenced this pull request Sep 10, 2026
BerriAI#40512 added d.Partial(true) to resourceKeyUpdate, but it landed on the
plannedKeyMetadata error path rather than the c.UpdateKey one. The test it
shipped with drives a 400 from /key/update, which returns through the
UpdateKey branch, so TestResourceKeyUpdateFailureKeepsPriorState has been
failing on litellm_internal_staging since it merged

Set the flag on the UpdateKey branch too. Without it the SDK hands back the
diff-applied state, so a rejected update persists the planned values and the
next plan reports no changes, silently dropping the update
yassin-berriai added a commit that referenced this pull request Sep 10, 2026
…0527)

The squash of #40512 onto a base that already carried #40514 left the Partial call on the metadata pre-read error path only, so a rejected /key/update again persisted the planned values into state and TestResourceKeyUpdateFailureKeepsPriorState fails on the default branch.

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants