Repository navigation
docs(proxy): document the master key boot check and rotating off sk-1234 - #1577
Merged
Merged
Conversation
The proxy is about to refuse to start when the master key is not set, is empty, or is sk-1234, and its error message links to /docs/proxy/master_key_rotations#proxy-refuses-to-start Add that section to the rotation page with the two ways out: swap and restart when a salt key is set or there is no database, and boot once with the local development override, re-encrypt through POST /key/regenerate, then swap when sk-1234 is also the encryption key Add dangerously_allow_unsafe_proxy and LITELLM_DANGEROUSLY_ALLOW_UNSAFE_PROXY to the config settings reference, and say on the master_key rows that the proxy will not start without a real key Fix the existing regenerate example, whose body key had been replaced with a virtual key placeholder. The proxy only rotates the master key when key is the current master key
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…generate call The proxy's error no longer prints the save-to-.env command when sk-1234 is also the encryption key. It prints a command that only generates a key and says to save it once this guide says to, so the rotation steps now follow that order Back up the database first, generate the key without saving it anywhere the proxy reads, boot once on the old key with the override, call POST /key/regenerate, then stop the proxy right away and only then set LITELLM_MASTER_KEY. The running process still holds the old key after the call and cannot decrypt the re-encrypted rows Also warn that the regenerate response echoes the new master key in plaintext, repeat that the regenerate call must not be used when LITELLM_SALT_KEY is set, and mirror the error's "make sure the config reads the key from the environment" wording
… key The startup error now picks its "set a new key" step by whether LITELLM_MASTER_KEY is already set in the proxy's environment. When it is not set, the error still prints the command that generates a key and appends it to .env. When it is set to an unsafe value, the error prints a generate-only command and says to put the new key in place of the current value wherever that is set Show both commands in the no-rotation case and explain why appending to .env does not work there: the proxy loads .env without overriding, so a value already exported in the environment wins. Say the same in "Where to set the new key", and stop describing the rotation case by the generate-only command, since it is no longer unique to it
4 of 6 tasks
…time migration Getting out of a refused boot no longer needs the local development override or a call to POST /key/regenerate. On refusal the proxy counts the stored values that decrypt under the unsafe key, and when there are any it tells the user to set LITELLM_MIGRATE_FROM_MASTER_KEY to the old key next to a new LITELLM_MASTER_KEY. The next boot re-encrypts those values before serving traffic and logs when the variable can be deleted Rewrite the section around that: the case where nothing is encrypted with the old key, the migration steps with the log lines to expect, what happens when the variable is left set, when the master key is still unsafe, when a value changes during the migration, and why LITELLM_SALT_KEY must not be added halfway. Add a short subsection on what the migration covers and that it also works as an offline alternative to the regenerate call, and point to it from the existing regenerate section Add LITELLM_MIGRATE_FROM_MASTER_KEY to the environment variable reference. The general POST /key/regenerate docs for a running proxy and the salt key warning stay as they were
Contributor
|
bugbot run |
…tion Quote the refusal lead as it is now wrapped, and say the unreachable-database variant only changes the first line Add the case where a database error interrupts the migration: the proxy keeps starting, logs a WARNING that the values cannot be read until the migration succeeds, and the fix is to keep LITELLM_MIGRATE_FROM_MASTER_KEY set and restart once the database is reachable Also say that the migration goes through the primary database even with a read replica configured, and that it skips tables or columns an older schema does not have, so it works before and after a schema upgrade
Contributor
|
bugbot run |
The proxy no longer keeps starting when the boot-time migration hits a database error. It logs the same WARNING and exits with a non-zero status, so a worker never serves with stored values it cannot read. The exception is a connection outage with general_settings.allow_requests_on_db_unavailable on, where the boot continues as it already does for the database connection Say that restarting with the same two variables resumes a partial migration, and move the quoted log line into a code block so its <ErrorType: message> placeholder cannot be parsed as JSX
Contributor
|
bugbot run |
…mit_weak_or_unset_master_key BerriAI/litellm#42019 renamed the override so the name says exactly what it permits. The environment variable is now LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY and the YAML key is general_settings.dangerously_permit_weak_or_unset_master_key Update the general_settings example, the general_settings reference row and the environment variable row. The behaviour is unchanged
Contributor
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 833da8f. Configure here.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Documents an upcoming BerriAI/litellm change, BerriAI/litellm#42019: the proxy will refuse to start when the master key it resolved is not set, is empty or only whitespace, or is the literal
sk-1234. It exits non-zero and prints how to fix it. This PR adds the page section that error links to, documents the newLITELLM_MIGRATE_FROM_MASTER_KEYboot-time migration, and adds the reference rows for the new settingsWhat changed
docs/proxy/master_key_rotations.mdgets a new section, "Proxy refuses to start on sk-1234", with the explicit heading id{#proxy-refuses-to-start}. It explains the refusal and that, when a database is configured andLITELLM_SALT_KEYis not set, the proxy connects during the refusal and counts the stored values that decrypt under the unsafe key. The steps it prints depend on that count, and the section follows the same split"Nothing is encrypted with the old key" covers a count of 0, no database, or a salt key. The error only says to replace the key, and the section shows both variants of that step. When
LITELLM_MASTER_KEYis not set at all the error printsecho "LITELLM_MASTER_KEY=sk-$(openssl rand -hex 32)" | tee -a .env. When the variable is already set to an unsafe value it prints onlyecho "sk-$(openssl rand -hex 32)"and says to put the new key in place of the current value wherever that is set, because the proxy loads.envwithout overriding and an exported value wins over an appended line. The section repeats thatPOST /key/regeneratewithnew_master_keymust not be used when a salt key is set"The database holds values encrypted with the old key" covers a count above 0, and the variant where the database could not be checked. The user sets
LITELLM_MIGRATE_FROM_MASTER_KEYto the old key next to a newLITELLM_MASTER_KEYand starts the proxy again. Right after the database connects and before traffic is served, the proxy re-encrypts every stored value that decrypts under the previous key and logs at WARNING when it starts and when the variable can be deleted. The steps quote those log lines, and the paragraphs after them cover leaving the variable set (a no-op with one notice per boot), setting it while the master key is still unsafe (refuses again), a value edited during the migration (keep the variable and restart), a database error during the migration (the proxy logs a WARNING and exits non-zero so it never serves with values it cannot read, except for a connection outage withgeneral_settings.allow_requests_on_db_unavailableon, and restarting with the same two variables resumes the partial migration), and whyLITELLM_SALT_KEYmust not be added halfway. The earlier version of this PR told users to boot with the local development override and callPOST /key/regenerate. Those instructions are gone"What the boot-time migration covers", with the id
{#boot-time-migration}, says that it works for any previous key and so is also an offline alternative to the regenerate call, that it is idempotent and safe with several workers or replicas booting at once, that it goes through the primary database even when a read replica is configured, that it skips tables or columns an older schema does not have and so works before and after a schema upgrade, that virtual keys keep working, and lists the stored values it re-encrypts, which is more than the regenerate call covers. The existing "If the master key is your encryption key" section gets one sentence pointing to it. The generalPOST /key/regeneratedocs for a running proxy and the salt key warning at the top of the page are otherwise unchangedA short "Where to set the new key" subsection covers when a
.envfile is actually read, replacing an already exportedLITELLM_MASTER_KEYwhere it is set instead of appending to.env, theos.environ/LITELLM_MASTER_KEYfix for a literal inconfig.yaml, and the Admin UI sign-outdocs/proxy/config_settings.mdgets adangerously_permit_weak_or_unset_master_keyrow in thegeneral_settingsreference, andLITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEYandLITELLM_MIGRATE_FROM_MASTER_KEYrows in the environment variables reference, all linked to the new section. Themaster_keyandLITELLM_MASTER_KEYrows now say the proxy will not start without a real keyOne fix to existing content: the
POST /key/regenerateexample on the rotation page had"key": "sk-<virtual-key>"in the body since the placeholder scrub in #1431. The proxy only rotates the master key whenkeyis the current master key (regenerate_key_fninkey_management_endpoints.pychecks the bodykeyagainst the master key), so with a virtual key there the call is a regular key regeneration and nothing is re-encrypted. The placeholder is nowsk-<current-master-key>and the sentence above it says soDo not rename the anchor
The proxy's error message hardcodes
https://docs.litellm.ai/docs/proxy/master_key_rotations#proxy-refuses-to-start, so the{#proxy-refuses-to-start}id has to stay as is even if the heading text changesMerge order
Merge together with, or just before, BerriAI/litellm#42019, so the link in the error resolves from the first release that ships the check
Related: BerriAI/litellm#42011 stops advertising
sk-1234as the master key in the configs and examples shipped in the main repo. The pages in this docs repo still usesk-1234in examples. That sweep is a separate follow-up and is left out of this PR on purposeChecks
npm run lint:writingandnpm run lint:docspass on the current tip, including the new heading anchors.npm run buildsucceeds on the current tip withonBrokenLinksandonBrokenAnchorsset tothrow, and the built rotation page hasid=proxy-refuses-to-startandid=boot-time-migration. The quoted error and log lines were checked against BerriAI/litellm#42019 at99f99cfb46, and the override names againstecf17513fb. The<ErrorType: message>placeholder in the quoted log line sits inside a code block, because in prose MDX parses it as JSX and the build failsNote
Low Risk
Documentation-only changes with no runtime or security logic in this repo; merge timing should align with the upstream proxy release so error links resolve.
Overview
Documents the upcoming proxy behavior that refuses to start when the resolved master key is missing, empty, or
sk-1234, and adds operator runbooks for fixing that safely.master_key_rotations.mdgains a anchored section "Proxy refuses to start on sk-1234" ({#proxy-refuses-to-start}) describing boot-time DB checks, split recovery paths (no encrypted data vs. DB values encrypted under the old key), andLITELLM_MIGRATE_FROM_MASTER_KEYboot re-encryption before traffic is served. It also documents migration scope ({#boot-time-migration}), env vs.envprecedence, partial-failure restarts, and "Where to set the new key". The encryption-key rotation section now points to boot-time migration as an offline alternative toPOST /key/regenerate, and the regenerate curl example fixeskeyto the current master key (not a virtual key).config_settings.mdupdatesmaster_key/LITELLM_MASTER_KEYto state the refusal rule, and addsdangerously_permit_weak_or_unset_master_key,LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY, andLITELLM_MIGRATE_FROM_MASTER_KEYwith links to the new section.Reviewed by Cursor Bugbot for commit 833da8f. Bugbot is set up for automated code reviews on this repo. Configure here.