feat(setup): interactive setup wizard + install.sh#23644
feat(setup): interactive setup wizard + install.sh#23644ishaan-jaff merged 24 commits intolitellm_ishaan_march_17from
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds a The implementation has gone through significant iteration — a large number of issues identified in prior review rounds have been addressed: YAML-safe escaping via Remaining issues found in this pass:
Confidence Score: 3/5
|
| Filename | Overview |
|---|---|
| litellm/setup_wizard.py | New interactive TUI setup wizard; many issues from prior review rounds have been addressed (YAML escaping, readline fix, port validation, lazily-evaluated divider, Windows import guard, Azure deployment collection, etc.), but three new issues remain: check_valid_key swallows all exceptions producing a misleading "invalid API key" message on network errors; no guard prevents writing a config with an empty model_list; and the Azure API key leaks into environment_variables even when Azure setup is incomplete. |
| litellm/proxy/proxy_cli.py | Adds --setup flag that delegates to run_setup_wizard() and returns; also adds # type: ignore[possibly-unbound] to suppress a Pyright warning on litellm_settings. The setup integration is minimal and correct; the type-ignore was flagged in a prior review thread. |
| scripts/install.sh | Curl-installable installer; most prior issues are resolved — LITELLM_PACKAGE now points to PyPI, pipefail dropped in favour of POSIX-compatible set -eu, /dev/tty guarded before read, --force-reinstall removed. Looks clean. |
| tests/test_litellm/test_setup_wizard.py | Pure unit tests covering _yaml_escape and _build_config; no network calls, no mocks needed. Tests verify YAML escaping, provider skipping, sentinel key filtering, Azure deployment name handling, and display-name collision avoidance. Good coverage of the pure functions. |
| CLAUDE.md | Documentation-only addition describing setup wizard conventions (class structure, check_valid_key usage, test_model field pattern). No issues. |
Sequence Diagram
sequenceDiagram
participant User
participant CLI as proxy_cli.py (--setup)
participant Wizard as SetupWizard
participant Terminal as Terminal / /dev/tty
participant LiteLLM as litellm.utils.check_valid_key
participant FS as Filesystem
User->>CLI: litellm --setup
CLI->>Wizard: run_setup_wizard()
Wizard->>Terminal: _print_welcome()
Wizard->>Terminal: _select_providers() [arrow-key TUI or number fallback]
Terminal-->>Wizard: selected providers[]
Wizard->>Terminal: _collect_keys(providers)
loop For each provider with env_key
Terminal-->>Wizard: API key input
Wizard->>LiteLLM: check_valid_key(test_model, api_key)
LiteLLM-->>Wizard: True / False (auth OR any exception)
alt Valid
Wizard->>Terminal: ✔ connected
else Invalid / network error
Wizard->>Terminal: ✘ invalid API key (misleading on network errors)
Terminal-->>Wizard: retry? (y/N)
end
end
Wizard->>Terminal: _proxy_settings() — port + master key
Terminal-->>Wizard: port, master_key
Wizard->>FS: write litellm_config.yaml
FS-->>Wizard: OK / OSError
Wizard->>Terminal: _print_success()
Wizard->>Terminal: Start proxy now? (Y/n)
alt Yes
Wizard->>CLI: os.execlp(litellm --config ... --port ...)
else No
Wizard->>Terminal: print manual start instructions
end
Comments Outside Diff (3)
-
litellm/setup_wizard.py, line 296-316 (link)No guard against writing an empty
model_listconfigIf the user selects only API-key-requiring providers (no Ollama) and then skips every key prompt,
_build_configwill emit:model_list: general_settings: master_key: "sk-xxx"
YAML parses
model_list:with no scalar or sequence following it asnull. LiteLLM will start successfully but silently have no routable models, and every API request will fail. There is no warning beforeconfig_path.write_text(...).A simple guard after
_build_configreturns would catch this:config_content = SetupWizard._build_config(providers, env_vars, master_key) if "\n - model_name:" not in config_content: print(f"\n {bold(_CROSS + ' No providers configured.')} " "Please re-run and enter at least one API key.\n") return
Alternatively, check that
env_varscontains at least one non-_LITELLM_key (or that Ollama is selected) before proceeding to write the file. -
litellm/setup_wizard.py, line 494-510 (link)Azure API key written to
environment_variableseven when Azure setup is incompleteIf a user provides an Azure API key but leaves the deployment name prompt blank,
_collect_keysstill storesAZURE_API_KEYinenv_vars(line 509). In_build_configthe Azure model entry is correctly skipped (no deployment), butAZURE_API_KEYis not_LITELLM_-prefixed, so it passes thereal_varsfilter and ends up in theenvironment_variables:block of the generated YAML — a stored credential that no model actually references.The generated config will confuse users who compare it to the model list and find no
azure/entry, while still seeing their Azure key. Consider either:- Clearing
env_vars["AZURE_API_KEY"]when no deployment name is provided, so the key is never carried into the config:
if deployment: env_vars[f"_LITELLM_AZURE_DEPLOYMENT_{p['id'].upper()}"] = deployment else: # No deployment → abandon Azure entirely; don't store the API key either print(grey(" Skipping Azure (no deployment name provided).")) continue # skip _validate_and_report and env_vars assignment below
- Or print an explicit warning after writing the config when
AZURE_API_KEYis present but noazure/model appears in the output.
- Clearing
-
litellm/setup_wizard.py, line 473-481 (link)check_valid_keyswallows all exceptions — misleading "invalid API key" errorcheck_valid_key(inlitellm/utils.py) catches bothAuthenticationErrorand the catch-allexcept Exceptionbranch, returningFalsein both cases. This means a transient network timeout, a 429 rate-limit response, or a DNS failure during key validation will display— invalid API keyto the user, even though the key itself is perfectly valid.A user with a working key but a momentary network hiccup may discard a valid key and re-enter it unnecessarily, or give up on the provider entirely.
At minimum, updating the printed message from
— invalid API keyto— validation failed (invalid key or network error)avoids a false diagnosis and sets correct expectations. Thecheck_valid_keyinterface does not expose the actual error type, so that distinction would require wrappinglitellm.completiondirectly here — but the message fix is a low-cost improvement.
Last reviewed commit: 0e77077
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
litellm/setup_wizard.py
Outdated
| if real_env_vars: | ||
| lines.append("environment_variables:") | ||
| for k, v in real_env_vars.items(): | ||
| lines.append(f" {k}: \"{v}\"") | ||
| lines.append("") |
There was a problem hiding this comment.
API keys written as unescaped YAML strings
API keys are embedded as bare double-quoted YAML scalars with no escaping. If a user pastes a key that happens to contain a ", \, or newline (even accidentally), the generated litellm_config.yaml will be malformed/unparseable. For example, a key value of sk-abc"def would produce:
OPENAI_API_KEY: "sk-abc"def"which is invalid YAML. Use Python's yaml library (already a transitive dependency of LiteLLM) to serialise the block safely:
import yaml
if real_env_vars:
lines.append("environment_variables:")
for k, v in real_env_vars.items():
lines.append(f" {k}: {yaml.dump(v, default_flow_style=True).strip()}")
lines.append("")Or build the full document as a dict and call yaml.dump() at the end to avoid manually constructing YAML entirely.
litellm/setup_wizard.py
Outdated
| def run_setup_wizard() -> Optional[str]: | ||
| """ | ||
| Run the interactive setup wizard. | ||
|
|
||
| Returns the path to the generated config file, or None if aborted. | ||
| """ | ||
| try: | ||
| _run_wizard() | ||
| except (KeyboardInterrupt, EOFError): | ||
| print(f"\n\n {grey('Setup cancelled.')}\n") | ||
| return None | ||
| return None # caller receives path via side effect (printed to stdout) |
There was a problem hiding this comment.
run_setup_wizard() always returns None despite its own docstring
The docstring says "Returns the path to the generated config file, or None if aborted", but the function always returns None regardless of whether the wizard completed successfully or was cancelled. The caller in proxy_cli.py currently ignores the return value, but this contract is broken and will mislead future callers.
Either update the docstring to match the implementation, or actually return the path on success:
def run_setup_wizard() -> Optional[str]:
"""
Run the interactive setup wizard.
Returns the path to the generated config file, or None if aborted.
"""
try:
config_path = _run_wizard()
return config_path
except (KeyboardInterrupt, EOFError):
print(f"\n\n {grey('Setup cancelled.')}\n")
return Noneand have _run_wizard() return the str(config_path) at the end.
litellm/setup_wizard.py
Outdated
| env_vars: Dict[str, str] = {} | ||
| print() | ||
| print(DIVIDER) | ||
| print() | ||
| print(f" {bold('Enter your API keys')}") | ||
| print(grey(" Keys are stored only in the generated config file.")) | ||
| print() |
There was a problem hiding this comment.
API keys stored in plaintext with no .gitignore warning
The wizard tells users "Keys are stored only in the generated config file", which is technically true but omits the critical follow-up: the file is written to the current working directory with a predictable name (litellm_config.yaml) and contains all secrets in plaintext. Users who accidentally git add . or check this file into a public repository will silently leak credentials.
Consider:
- Printing an explicit warning after writing the file, e.g.:
⚠ Add litellm_config.yaml to .gitignore to avoid leaking keys:
echo "litellm_config.yaml" >> .gitignore
- Optionally auto-appending to
.gitignoreif one already exists in the current directory.
litellm/setup_wizard.py
Outdated
| ╚══════╝╚═╝ ╚═╝ ╚══════╝╚══════╝╚══════╝╚═╝ ╚═╝ | ||
| """ | ||
|
|
||
| DIVIDER = dim(" " + "╌" * 74) |
There was a problem hiding this comment.
DIVIDER captures color support at module-import time
DIVIDER = dim(" " + "╌" * 74)dim() calls _supports_color(), which calls sys.stdout.isatty(), at import time. If setup_wizard is imported in a non-TTY context (e.g., during tests, CI, or any other import-time evaluation), DIVIDER will permanently lack ANSI codes for the lifetime of the process — even if the actual wizard later runs in a TTY.
Move this to a function or lazily evaluate it at call time:
def _divider() -> str:
return dim(" " + "╌" * 74)and replace all uses of DIVIDER with _divider().
litellm/setup_wizard.py
Outdated
| key = "" | ||
| while not key: | ||
| key = input(f" {blue('❯')} {bold(p['name'])} API key {hint}: ").strip() | ||
| if not key: | ||
| print(grey(" Key is required. Leave blank to skip this provider.")) | ||
| skip = input(grey(" Skip? (y/N): ")).strip().lower() | ||
| if skip == "y": | ||
| break |
There was a problem hiding this comment.
Skipped provider still emits models into the config without credentials
If a user selects (say) OpenAI but then skips the API key prompt (Skip? y), the key variable is left as "" and env_vars["OPENAI_API_KEY"] is never set. However in _build_config, the provider's models (gpt-4o, gpt-4o-mini) are still added to model_list — just without an api_key line. The resulting config will attempt to load these models and silently fall back to the OPENAI_API_KEY environment variable, or fail at runtime with a confusing authentication error.
Consider either:
- Excluding the provider from
_build_configwhen its key was skipped (pass a filtered list), or - Adding a comment in the YAML to make the missing key obvious to the user.
scripts/install.sh
Outdated
| PYTHON_BIN="" | ||
| for candidate in python3 python; do | ||
| if command -v "$candidate" >/dev/null 2>&1; then | ||
| py_ver="$("$candidate" -c 'import sys; print(sys.version_info[:2])' 2>/dev/null || true)" |
There was a problem hiding this comment.
Unused variable py_ver
py_ver is assigned but never referenced. Remove it to keep the script clean and avoid the spurious Python subprocess invocation:
| py_ver="$("$candidate" -c 'import sys; print(sys.version_info[:2])' 2>/dev/null || true)" | |
| major="$("$candidate" -c 'import sys; print(sys.version_info.major)' 2>/dev/null || true)" |
litellm/setup_wizard.py
Outdated
| "description": "Claude Opus, Sonnet, Haiku", | ||
| "env_key": "ANTHROPIC_API_KEY", | ||
| "key_hint": "sk-ant-...", | ||
| "models": ["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001"], |
There was a problem hiding this comment.
Hardcoded Anthropic model IDs will go stale
"models": ["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001"],These model strings are hardcoded in the wizard, meaning every new Claude release requires a code change + LiteLLM upgrade before it shows up as a suggestion. The same applies to bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0 and other versioned identifiers.
Per the project's convention, consider deriving the displayed model list from model_prices_and_context_window.json (via get_model_info / litellm.model_list) so the wizard automatically reflects the latest supported models without code changes.
Rule Used: What: Do not hardcode model-specific flags in the ... (source)
scripts/install.sh
Outdated
| # LiteLLM Installer | ||
| # Usage: curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh | ||
| # Branch QA: LITELLM_BRANCH=worktree-dynamic-tickling-journal curl -fsSL .../install.sh | sh | ||
| set -euo pipefail |
There was a problem hiding this comment.
pipefail breaks on POSIX sh (Debian/Ubuntu)
The documented install command is curl -fsSL ... | sh. When a script is piped this way, the OS executes it with the shell named in $SHELL or invoked as sh — the #!/usr/bin/env bash shebang on line 1 is completely ignored by the receiving shell process.
On Debian and Ubuntu (the most common Linux distributions), /bin/sh is dash, which does not recognise pipefail as a valid set option. Running the one-liner will immediately print:
sh: 1: set: Illegal option -o pipefail
and exit with a non-zero status before a single line of installer logic runs.
Fix: either change the documented usage to | bash, or drop pipefail and keep only the POSIX-compatible options:
| set -euo pipefail | |
| set -eu |
litellm/setup_wizard.py
Outdated
| "description": "Claude Opus, Sonnet, Haiku", | ||
| "env_key": "ANTHROPIC_API_KEY", | ||
| "key_hint": "sk-ant-...", | ||
| "models": ["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001"], |
There was a problem hiding this comment.
Hardcoded model IDs will go stale and violate project convention
The model strings "claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001" (and "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0" on line 63) are hardcoded directly in the wizard. Per the project's convention, model-specific identifiers should not be hardcoded in the codebase — they belong in model_prices_and_context_window.json and should be read via get_model_info / litellm.model_list. Every new Claude release will require a code change and a LiteLLM upgrade before it shows up here.
This pattern also appears at litellm/setup_wizard.py:63 for the Bedrock model ID.
Consider deriving the displayed model list from the existing model registry so the wizard automatically reflects the latest supported models without code changes.
Rule Used: What: Do not hardcode model-specific flags in the ... (source)
litellm/setup_wizard.py
Outdated
| if not p["models"] and p["id"] == "azure": | ||
| # Azure — add a generic placeholder | ||
| models_to_add = ["azure/gpt-4o"] | ||
| else: |
There was a problem hiding this comment.
Azure deployment name is never collected — generated config will fail
Azure OpenAI requires a user-defined deployment name that is separate from the underlying model name. In LiteLLM the model string must be azure/<deployment_name>, not azure/<model_name>. The wizard never prompts for this value, so it unconditionally falls back to the placeholder azure/gpt-4o, which assumes the user named their deployment gpt-4o.
In virtually every real Azure workspace the deployment has a custom name (e.g. my-gpt4-prod), so any request using this generated config will receive a 404 / "DeploymentNotFound" error from Azure.
The deployment name should be collected during the API key step for Azure (alongside the endpoint URL) and threaded through to _build_config.
scripts/install.sh
Outdated
|
|
||
| # NOTE: set to "litellm[proxy]" before merging to main (once --setup ships in a PyPI release). | ||
| # On this branch we install from git so the installer gets the --setup flag. | ||
| LITELLM_PACKAGE="litellm[proxy] @ git+https://github.com/BerriAI/litellm.git@worktree-dynamic-tickling-journal" |
There was a problem hiding this comment.
Hardcoded dev branch will break the installer after merge
LITELLM_PACKAGE is hardcoded to the feature branch worktree-dynamic-tickling-journal. Once this PR is merged and that branch is deleted, every user who runs the documented curl | sh one-liner from main will get a pip install failure because the branch no longer exists.
The comment on line 9-10 even acknowledges this must be fixed before merging, but the fix has not been applied. The line should be changed to:
| LITELLM_PACKAGE="litellm[proxy] @ git+https://github.com/BerriAI/litellm.git@worktree-dynamic-tickling-journal" | |
| LITELLM_PACKAGE="litellm[proxy]" |
scripts/install.sh
Outdated
| if [ -n "$LITELLM_BRANCH" ]; then | ||
| header "Installing litellm from branch '${LITELLM_BRANCH}'…" | ||
| else | ||
| header "Installing litellm[proxy]…" | ||
| fi | ||
| echo "" | ||
|
|
||
| "$PYTHON_BIN" -m pip install --quiet --progress-bar off "${LITELLM_PACKAGE}" \ | ||
| || die "pip install failed. Try manually: $PYTHON_BIN -m pip install '${LITELLM_PACKAGE}'" |
There was a problem hiding this comment.
LITELLM_BRANCH env var changes the display banner but not the installed package
The script checks $LITELLM_BRANCH to decide what to print in the header (lines 81–85), but the actual pip install on line 88 always uses $LITELLM_PACKAGE, which is hardcoded above. A user who exports LITELLM_BRANCH=my-feature will see "Installing litellm from branch 'my-feature'…" but actually install from whatever LITELLM_PACKAGE was set to. This is misleading and could cause confusion during debugging.
If LITELLM_BRANCH is intended to allow branch overrides, the package variable needs to be set based on it:
| if [ -n "$LITELLM_BRANCH" ]; then | |
| header "Installing litellm from branch '${LITELLM_BRANCH}'…" | |
| else | |
| header "Installing litellm[proxy]…" | |
| fi | |
| echo "" | |
| "$PYTHON_BIN" -m pip install --quiet --progress-bar off "${LITELLM_PACKAGE}" \ | |
| || die "pip install failed. Try manually: $PYTHON_BIN -m pip install '${LITELLM_PACKAGE}'" | |
| if [ -n "$LITELLM_BRANCH" ]; then | |
| header "Installing litellm from branch '${LITELLM_BRANCH}'…" | |
| LITELLM_PACKAGE="litellm[proxy] @ git+https://github.com/BerriAI/litellm.git@${LITELLM_BRANCH}" | |
| else | |
| header "Installing litellm[proxy]…" | |
| fi | |
| echo "" | |
| "$PYTHON_BIN" -m pip install --quiet --progress-bar off "${LITELLM_PACKAGE}" \ | |
| || die "pip install failed. Try manually: $PYTHON_BIN -m pip install '${LITELLM_PACKAGE}'" |
litellm/setup_wizard.py
Outdated
| port_raw = input(f" {blue('❯')} Port {grey('[4000]')}: ").strip() | ||
| port = int(port_raw) if port_raw.isdigit() else 4000 |
There was a problem hiding this comment.
Port number is not range-validated
port_raw.isdigit() only verifies the string is a non-negative integer, but valid TCP ports are 1–65535. A user entering 99999 or 0 will pass this check and produce a config file with an invalid port that causes litellm to fail on startup with an unhelpful OS error rather than a clear message here.
| port_raw = input(f" {blue('❯')} Port {grey('[4000]')}: ").strip() | |
| port = int(port_raw) if port_raw.isdigit() else 4000 | |
| port = int(port_raw) if port_raw.isdigit() and 1 <= int(port_raw) <= 65535 else 4000 |
litellm/setup_wizard.py
Outdated
| for p in providers: | ||
| if not p["models"] and p["id"] == "azure": | ||
| # Azure — add a generic placeholder | ||
| models_to_add = ["azure/gpt-4o"] | ||
| else: | ||
| models_to_add = p["models"] | ||
|
|
||
| for model in models_to_add: | ||
| # User-facing model name (strip provider prefix for display) | ||
| display_name = model.split("/")[-1] if "/" in model else model |
There was a problem hiding this comment.
display_name collision when OpenAI and Azure are both selected
display_name is derived by stripping the provider prefix: azure/gpt-4o → gpt-4o. When a user selects both OpenAI and Azure OpenAI, the generated config will contain two entries both with model_name: gpt-4o:
model_list:
- model_name: gpt-4o # from OpenAI
litellm_params:
model: gpt-4o
...
- model_name: gpt-4o # from Azure — same key!
litellm_params:
model: azure/gpt-4oLiteLLM supports duplicate model_name entries for load-balancing, but with no indication to the user that both will serve under the same name, this will cause silent routing surprises. Prefixing the Azure entry would make the config unambiguous:
| for p in providers: | |
| if not p["models"] and p["id"] == "azure": | |
| # Azure — add a generic placeholder | |
| models_to_add = ["azure/gpt-4o"] | |
| else: | |
| models_to_add = p["models"] | |
| for model in models_to_add: | |
| # User-facing model name (strip provider prefix for display) | |
| display_name = model.split("/")[-1] if "/" in model else model | |
| display_name = model.split("/")[-1] if "/" in model else model | |
| if p["id"] == "azure": | |
| display_name = f"azure-{display_name}" |
litellm/setup_wizard.py
Outdated
| "description": "Claude Opus, Sonnet, Haiku", | ||
| "env_key": "ANTHROPIC_API_KEY", | ||
| "key_hint": "sk-ant-...", | ||
| "models": ["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001"], |
There was a problem hiding this comment.
Hardcoded model IDs violate project convention
The model strings "claude-opus-4-5", "claude-sonnet-4-5", and "claude-haiku-4-5-20251001" (line 36) are hardcoded in the wizard. Per the project's convention, model-specific identifiers should not be hardcoded in the codebase — they belong in model_prices_and_context_window.json and should be read via get_model_info / litellm.model_list. Every new Claude release will require a code change and a LiteLLM upgrade before it shows up here.
The same issue applies to "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0" on line 63.
Consider querying the model registry at wizard startup to populate the model lists dynamically, so the wizard automatically reflects the latest supported models without code changes.
Rule Used: What: Do not hardcode model-specific flags in the ... (source)
litellm/setup_wizard.py
Outdated
| config_path = Path(os.getcwd()) / "litellm_config.yaml" | ||
| config_path.write_text(config_content) |
There was a problem hiding this comment.
Unhandled exception on config write failure
config_path.write_text(config_content) is not wrapped in any error handling. If the current working directory is not writable, or the disk is full, Python raises a PermissionError or OSError respectively. This exception propagates through _run_wizard() and exits run_setup_wizard() via the except (KeyboardInterrupt, EOFError) block — but those exception types are not caught, so the user ends up with a raw Python traceback after spending time answering all the prompts.
A try/except around the write would produce a friendly error message instead:
try:
config_path.write_text(config_content)
except OSError as e:
print(f"\n Error: could not write config: {e}")
return
scripts/install.sh
Outdated
| # ── launch setup wizard ──────────────────────────────────────────────────── | ||
| echo "" | ||
| printf " ${BOLD}Run the interactive setup wizard?${RESET} ${GREY}(Y/n)${RESET}: " | ||
| read -r answer </dev/tty |
There was a problem hiding this comment.
read </dev/tty exits with error when /dev/tty is unavailable
read -r answer </dev/tty redirects from /dev/tty to allow interactive input even when stdin is a pipe (the curl | sh use-case). However, /dev/tty may not exist or may not be accessible in Docker containers, CI systems, or any environment where there is no controlling terminal. Because set -e is in effect, a failure to open /dev/tty will exit the script immediately, leaving the user with no LiteLLM binary and no indication of what went wrong.
The fallback should skip the prompt gracefully:
| read -r answer </dev/tty | |
| read -r answer </dev/tty 2>/dev/null || answer="n" |
This way, non-interactive environments simply skip the wizard launch without aborting.
litellm/setup_wizard.py
Outdated
| if p.get("needs_api_base") and key: | ||
| api_base = input( | ||
| f" {blue('❯')} Azure endpoint URL {grey(p.get('api_base_hint', ''))}: " | ||
| ).strip() | ||
| if api_base: | ||
| env_vars[f"_LITELLM_AZURE_API_BASE_{p['id'].upper()}"] = api_base |
There was a problem hiding this comment.
Azure endpoint URL silently omitted when user skips it
Azure OpenAI always requires an endpoint URL (api_base). If the user enters a valid API key but then hits Enter without providing the endpoint, api_base is never set in env_vars. In _build_config (lines 285–288), the api_base field is then silently omitted from the YAML, and the generated config will fail at runtime with a confusing authentication or routing error.
The endpoint URL should be treated as required for Azure, with a validation loop mirroring how the API key is collected — rejecting empty input and only allowing a deliberate skip rather than silently proceeding without it.
scripts/install.sh
Outdated
| "$PYTHON_BIN" -m pip install --upgrade --force-reinstall "${LITELLM_PACKAGE}" \ | ||
| || die "pip install failed. Try manually: $PYTHON_BIN -m pip install '${LITELLM_PACKAGE}'" |
There was a problem hiding this comment.
--force-reinstall can downgrade existing installations
Using --upgrade --force-reinstall together forces reinstallation of litellm and all of its dependencies even when the user already has a newer or equivalent version installed. This is problematic because:
- It may downgrade
litellm's dependencies (e.g.httpx,pydantic) if the version pinned in the package's metadata is older than what the user has. - It causes significant extra download and install time for users who already have litellm installed.
--upgradealone is sufficient to ensure the latest version is installed;--force-reinstallon top of it adds no benefit for a normal install path.
| "$PYTHON_BIN" -m pip install --upgrade --force-reinstall "${LITELLM_PACKAGE}" \ | |
| || die "pip install failed. Try manually: $PYTHON_BIN -m pip install '${LITELLM_PACKAGE}'" | |
| "$PYTHON_BIN" -m pip install --upgrade "${LITELLM_PACKAGE}" \ | |
| || die "pip install failed. Try manually: $PYTHON_BIN -m pip install '${LITELLM_PACKAGE}'" |
litellm/setup_wizard.py
Outdated
| lines.append(f" api_base: {p['api_base']}") | ||
| elif p.get("needs_api_base"): | ||
| azure_base_key = f"_LITELLM_AZURE_API_BASE_{p['id'].upper()}" | ||
| if azure_base_key in env_vars: | ||
| lines.append(f" api_base: {env_vars.pop(azure_base_key)}") |
There was a problem hiding this comment.
env_vars.pop() mutates a caller-owned dict inside _build_config
_build_config removes the Azure API base sentinel key directly from the env_vars dict that was passed in by reference from _run_wizard. This is a hidden side effect: the caller's dict is silently mutated as part of config generation. While harmless today (the dict isn't used after this call), it couples the two functions via mutation rather than value.
Consider using env_vars.get(azure_base_key) and letting the later filter on line 298 (if not k.startswith("_LITELLM_")) handle exclusion, which is already in place for this reason:
elif p.get("needs_api_base"):
azure_base_key = f"_LITELLM_AZURE_API_BASE_{p['id'].upper()}"
azure_base_val = env_vars.get(azure_base_key)
if azure_base_val:
lines.append(f" api_base: {azure_base_val}")
litellm/setup_wizard.py
Outdated
| print(f" {bold('Then set your client:')}") | ||
| print() | ||
| print(f" export OPENAI_BASE_URL=http://localhost:{port}") | ||
| print(f" export OPENAI_API_KEY={master_key}") |
There was a problem hiding this comment.
Auto-generated master key printed in cleartext to terminal
The auto-generated master_key is printed verbatim in the post-config summary. Unlike a key the user typed themselves, this is a machine-generated token that now lives in the terminal's scrollback buffer — making it easy to leak via screen-shares, CI logs, or terminal recordings.
Consider displaying only a short truncated prefix (e.g. the first 8 characters followed by ...) and pointing the user to the config file for the full value. This reduces accidental exposure without losing usability.
litellm/setup_wizard.py
Outdated
| raw = input(f" {blue('❯')} Provider(s): ").strip() | ||
| if not raw: | ||
| if not selected_nums: | ||
| print(grey(" Please select at least one provider.")) | ||
| continue | ||
| break | ||
| try: | ||
| nums = [int(x.strip()) for x in raw.replace(" ", ",").split(",") if x.strip()] | ||
| valid = [n for n in nums if 1 <= n <= len(PROVIDERS)] | ||
| if not valid: | ||
| print(grey(f" Enter numbers between 1 and {len(PROVIDERS)}.")) | ||
| continue | ||
| selected_nums = sorted(set(valid)) | ||
| _print_provider_menu(selected_nums) | ||
| except ValueError: | ||
| print(grey(" Enter numbers separated by commas, e.g. 1,3")) |
There was a problem hiding this comment.
ANSI codes in input() prompts break readline cursor positioning
Every input() call in this file passes a prompt containing raw ANSI escape sequences (from blue(), bold(), grey(), etc.) directly to Python's input(). When the readline library is active (the default on macOS and Linux), it uses the prompt string's byte length to calculate the terminal column for cursor positioning. Because readline doesn't know the ANSI codes are zero-width, it overcounts and places the cursor in the wrong column. As a result:
- Backspace may delete the wrong characters visually
- Line-editing navigation (Home/End, arrow keys) produces garbled output
- The cursor jumps to incorrect positions
The POSIX fix is to wrap every non-printing sequence between \001 (RL_PROMPT_START_IGNORE) and \002 (RL_PROMPT_END_IGNORE). Update _c() to emit these when output is going to input():
def _c_prompt(code: str, text: str) -> str:
"""ANSI colour helper safe for use inside input() prompts."""
if _supports_color():
return f"\001{code}\002{text}\001{_RESET}\002"
return textAnd use _c_prompt variants for all prompt strings passed to input(). This affects all six input() call-sites (lines 180, 222, 225, 245, 320, 323, 388).
litellm/setup_wizard.py
Outdated
| os.execlp( # noqa: S606 | ||
| sys.executable, | ||
| sys.executable, | ||
| "-m", | ||
| "litellm", | ||
| "--config", | ||
| str(config_path), | ||
| "--port", | ||
| str(port), | ||
| ) |
There was a problem hiding this comment.
os.execlp failure produces raw traceback after wizard completes
os.execlp raises FileNotFoundError (if the Python executable path is somehow invalid) or OSError (permission issue, etc.) when the exec syscall fails. The user will have already completed every step of the wizard, and an unhandled exception here produces a confusing Python traceback at the worst possible moment — right after seeing the "Config saved" success message.
Wrap the call in a try/except for a friendlier error:
try:
os.execlp(
sys.executable,
sys.executable,
"-m",
"litellm",
"--config",
str(config_path),
"--port",
str(port),
)
except OSError as e:
print(f"\n Error: could not start proxy: {e}")
print(f" Run manually: litellm --config {config_path} --port {port}")
print()
litellm/setup_wizard.py
Outdated
| "description": "Claude Opus, Sonnet, Haiku", | ||
| "env_key": "ANTHROPIC_API_KEY", | ||
| "key_hint": "sk-ant-...", | ||
| "models": ["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001"], |
There was a problem hiding this comment.
Hardcoded model IDs violate project convention and will go stale
Model-specific strings such as "claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001" (line 36) and "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0" (line 63) are hardcoded directly in the wizard. Per the project convention, model-specific identifiers must not be hardcoded in the codebase — they belong in model_prices_and_context_window.json and should be read via get_model_info / litellm.model_list.
Every new Claude (or Bedrock) release will require a code change and a LiteLLM upgrade before it surfaces here. Instead, consider querying the model registry at wizard startup to populate the model lists dynamically:
import litellm
def _get_models_for_provider(prefix: str) -> list[str]:
return [m for m in litellm.model_list if m.startswith(prefix)]The same issue appears at line 63 for the Bedrock model ID.
Rule Used: What: Do not hardcode model-specific flags in the ... (source)
scripts/install.sh
Outdated
|
|
||
| # NOTE: set to "litellm[proxy]" before merging to main (once --setup ships in a PyPI release). | ||
| # On this branch we install from git so the installer gets the --setup flag. | ||
| LITELLM_PACKAGE="litellm[proxy] @ git+https://github.com/BerriAI/litellm.git@worktree-dynamic-tickling-journal" |
There was a problem hiding this comment.
Hardcoded dev branch will break the installer after merge
LITELLM_PACKAGE points to the feature branch worktree-dynamic-tickling-journal. Once this PR is merged and that branch is deleted, any user running the documented curl | sh one-liner from main will get a pip install failure because the branch no longer exists. The comment on lines 9–10 even acknowledges this fix is required before merging, but it has not been applied.
This must be changed to the stable PyPI package before merging:
| LITELLM_PACKAGE="litellm[proxy] @ git+https://github.com/BerriAI/litellm.git@worktree-dynamic-tickling-journal" | |
| LITELLM_PACKAGE="litellm[proxy]" |
scripts/install.sh
Outdated
| #!/usr/bin/env bash | ||
| # LiteLLM Installer | ||
| # Usage: curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh | ||
| set -euo pipefail |
There was a problem hiding this comment.
pipefail is not POSIX-compatible — breaks on Debian/Ubuntu dash
The documented install command is curl -fsSL ... | sh. When a script is piped this way, the OS executes it with the interpreter named as sh — the #!/usr/bin/env bash shebang on line 1 is completely ignored by the receiving shell process. On Debian and Ubuntu (the most common Linux distributions), /bin/sh is dash, which does not recognise pipefail as a valid set option. Running the one-liner will immediately print:
sh: 1: set: Illegal option -o pipefail
and exit with a non-zero status before a single line of installer logic runs.
Either change the documented usage to | bash, or drop pipefail and keep only POSIX-compatible options:
| set -euo pipefail | |
| set -eu |
litellm/setup_wizard.py
Outdated
| # Extra keys (e.g. AWS secret + region) | ||
| if p.get("needs_extra") and key: | ||
| for extra_key, extra_hint in zip( | ||
| p.get("extra_keys", []), p.get("extra_hints", []) | ||
| ): | ||
| val = input( | ||
| f" {blue('❯')} {extra_key} {grey(extra_hint)}: " | ||
| ).strip() | ||
| if val: | ||
| env_vars[extra_key] = val |
There was a problem hiding this comment.
Bedrock extra credentials silently optional — config will fail at runtime
AWS_SECRET_ACCESS_KEY and AWS_REGION_NAME are silently treated as optional for Bedrock. If the user provides AWS_ACCESS_KEY_ID but skips (or accidentally leaves blank) either of the extra key prompts, the generated config will omit those values with no warning. Any request routed to the Bedrock model will then fail at runtime with a confusing authentication error rather than a clear message during setup.
Unlike the Azure endpoint (which has a dedicated needs_api_base guard), the needs_extra path has no required-field enforcement. At minimum, print a warning when AWS_SECRET_ACCESS_KEY is left blank:
if p.get("needs_extra") and key:
for extra_key, extra_hint in zip(
p.get("extra_keys", []), p.get("extra_hints", [])
):
val = input(
f" {blue('❯')} {extra_key} {grey(extra_hint)}: "
).strip()
if val:
env_vars[extra_key] = val
elif extra_key != "AWS_REGION_NAME": # region has a default
print(grey(f" Warning: {extra_key} is required for AWS Bedrock — config may fail at runtime."))
litellm/setup_wizard.py
Outdated
| def _build_config( | ||
| providers: List[Dict], | ||
| env_vars: Dict[str, str], | ||
| port: int, | ||
| master_key: str, | ||
| ) -> str: | ||
| lines = ["model_list:"] | ||
|
|
||
| for p in providers: | ||
| if not p["models"] and p["id"] == "azure": | ||
| # Azure — add a generic placeholder | ||
| models_to_add = ["azure/gpt-4o"] | ||
| else: | ||
| models_to_add = p["models"] | ||
|
|
||
| for model in models_to_add: | ||
| # User-facing model name (strip provider prefix for display) | ||
| display_name = model.split("/")[-1] if "/" in model else model | ||
| lines.append(f" - model_name: {display_name}") | ||
| lines.append(f" litellm_params:") | ||
| lines.append(f" model: {model}") | ||
|
|
||
| if p["env_key"] and p["env_key"] in env_vars: | ||
| lines.append(f" api_key: os.environ/{p['env_key']}") | ||
|
|
||
| if p.get("api_base"): | ||
| lines.append(f" api_base: {p['api_base']}") | ||
| elif p.get("needs_api_base"): | ||
| azure_base_key = f"_LITELLM_AZURE_API_BASE_{p['id'].upper()}" | ||
| if azure_base_key in env_vars: | ||
| lines.append(f" api_base: {env_vars.pop(azure_base_key)}") | ||
| if p.get("api_version"): | ||
| lines.append(f" api_version: {p['api_version']}") | ||
|
|
||
| lines.append("") | ||
| lines.append("general_settings:") | ||
| lines.append(f" master_key: {master_key}") | ||
| lines.append("") | ||
|
|
||
| # Write env vars inline so the config is self-contained | ||
| real_env_vars = {k: v for k, v in env_vars.items() if not k.startswith("_LITELLM_")} | ||
| if real_env_vars: | ||
| lines.append("environment_variables:") | ||
| for k, v in real_env_vars.items(): | ||
| lines.append(f" {k}: \"{v}\"") | ||
| lines.append("") | ||
|
|
||
| return "\n".join(lines) |
There was a problem hiding this comment.
No unit tests for the pure _build_config function
_build_config is a completely pure function — it takes plain Python values and returns a string — yet no tests are provided anywhere in the PR. Per project convention, all testable logic should have corresponding mock/unit tests that can run in CI without any network calls.
There are several behaviours in this function worth protecting with tests:
- Azure model placeholder (
azure/gpt-4o) whenmodelsis empty os.environ/KEYreference is emitted only when the key is present inenv_varsapi_versionis appended only for providers that declare itenvironment_variables:block is omitted when no real vars exist- The
_LITELLM_*sentinel keys are correctly filtered out of the output YAML
These can all be exercised with a straightforward parameterised test using only stdlib — no mocks or network access needed.
Rule Used: What: prevent any tests from being added here that... (source)
litellm/setup_wizard.py
Outdated
| "description": "Claude Opus, Sonnet, Haiku", | ||
| "env_key": "ANTHROPIC_API_KEY", | ||
| "key_hint": "sk-ant-...", | ||
| "models": ["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001"], |
There was a problem hiding this comment.
Hardcoded model IDs violate project convention and will go stale
"claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001" (line 36) and "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0" (line 63) are hardcoded directly in the wizard. Per the project's convention, model-specific identifiers must not be hardcoded in the codebase — they belong in model_prices_and_context_window.json and should be read via get_model_info / litellm.model_list. Every new Claude or Bedrock release will require a code change and a LiteLLM upgrade before it surfaces in the wizard.
Consider querying the model registry at wizard startup so the list stays current automatically:
import litellm
def _get_models_for_provider(prefix: str) -> list[str]:
return [m for m in litellm.model_list if m.startswith(prefix)]The same issue also applies to "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0" on line 63.
Rule Used: What: Do not hardcode model-specific flags in the ... (source)
| database_url = get_secret("DATABASE_URL", default_value=None) | ||
| modified_url = append_query_params(database_url, params) | ||
| modified_url = append_query_params(database_url, params) # type: ignore[arg-type] | ||
| os.environ["DATABASE_URL"] = modified_url |
There was a problem hiding this comment.
type: ignore suppresses potential None passed to append_query_params
get_secret("DATABASE_URL", default_value=None) can return None (e.g., when a secret manager is configured and the key resolves to None). The # type: ignore[arg-type] comment silences the type-checker warning instead of fixing the underlying issue. Even though the guard os.getenv("DATABASE_URL") is not None runs first, get_secret wraps secret-manager backends that may independently return None for a key that exists in the environment.
Passing None to append_query_params will produce a TypeError at runtime. The proper fix is to add an explicit guard:
database_url = get_secret("DATABASE_URL", default_value=None)
if database_url is None:
die("DATABASE_URL is set but get_secret returned None")
modified_url = append_query_params(database_url, params)Note: this change appears unrelated to the --setup wizard feature, suggesting it was introduced as a workaround for a pre-existing type error and should be addressed separately.
litellm/setup_wizard.py
Outdated
| # Proxy settings | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
| def _proxy_settings() -> tuple[int, str]: |
There was a problem hiding this comment.
tuple[int, str] annotation incompatible with Python 3.8
The built-in tuple generic syntax (tuple[int, str]) was introduced in Python 3.9 (PEP 585). Function annotations are evaluated eagerly at definition time in Python 3.8, so importing this module under Python 3.8 raises TypeError: 'type' object is not subscriptable.
While install.sh enforces Python ≥ 3.9, litellm/setup_wizard.py is part of the installable package and may be imported in 3.8 environments. Use from typing import Tuple (or from __future__ import annotations) for backward compatibility:
| def _proxy_settings() -> tuple[int, str]: | |
| def _proxy_settings() -> "tuple[int, str]": |
or, for broader compatibility:
from typing import Tuple
def _proxy_settings() -> Tuple[int, str]:
litellm/setup_wizard.py
Outdated
| def _run_wizard() -> None: | ||
| _print_welcome() | ||
|
|
||
| print(f" {bold('Lets get started.')}") |
There was a problem hiding this comment.
Missing apostrophe in welcome text
'Lets get started.' is missing the apostrophe in the contraction "Let's".
| print(f" {bold('Lets get started.')}") | |
| print(f" {bold('Let\\'s get started.')}") |
litellm/setup_wizard.py
Outdated
| "description": "Claude Opus, Sonnet, Haiku", | ||
| "env_key": "ANTHROPIC_API_KEY", | ||
| "key_hint": "sk-ant-...", | ||
| "models": ["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001"], |
There was a problem hiding this comment.
Hardcoded model IDs violate project convention and will go stale
"claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5-20251001" (line 36) and "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0" (line 63) are hardcoded directly in the wizard. Per the project's convention (see the model_prices_and_context_window.json + get_model_info pattern), model-specific identifiers must not be hardcoded in the codebase. Every new Claude or Bedrock release will require a code change and a LiteLLM upgrade before it surfaces in the wizard.
Consider querying the model registry at wizard startup to populate the model lists dynamically:
import litellm
def _get_models_for_provider(prefix: str) -> list:
return [m for m in litellm.model_list if m.startswith(prefix)]The same issue also applies to "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0" on line 63.
Rule Used: What: Do not hardcode model-specific flags in the ... (source)
litellm/setup_wizard.py
Outdated
| import termios | ||
| import tty |
There was a problem hiding this comment.
UNIX-only imports at module level break Windows
termios and tty are POSIX-only standard library modules. Placing them at the top-level import means that import litellm.setup_wizard — or, more concretely, litellm --setup — will immediately raise ModuleNotFoundError: No module named 'termios' on any Windows machine, with no graceful fallback to the number-entry path that was already written to handle this case.
The fix is to move both imports inside the one function that actually uses them (_read_key), so that _select_providers can still catch OSError / AttributeError and fall back to _select_providers_fallback on Windows:
def _read_key() -> str:
"""Read one keypress from /dev/tty in raw mode."""
import termios # POSIX-only — imported here to avoid breaking Windows at import time
import tty
with open("/dev/tty", "rb") as tty_fh:
...
litellm/setup_wizard.py
Outdated
| scripts_dir = sysconfig.get_path("scripts") | ||
| litellm_bin = os.path.join(scripts_dir, "litellm") |
There was a problem hiding this comment.
sysconfig.get_path() can return None, causing TypeError
sysconfig.get_path("scripts") returns None when the scheme doesn't include that path key (e.g., on some custom or embedded Python builds). Passing None directly to os.path.join raises:
TypeError: expected str, bytes or os.PathLike object, not NoneType
This would surface as an unhandled exception right after the user has completed all wizard prompts — the worst possible moment. The try/except OSError noted in the previous discussion about os.execlp would not catch a TypeError.
A guard is needed before os.path.join:
scripts_dir = sysconfig.get_path("scripts")
if not scripts_dir:
print(f"\n Error: could not locate Python scripts directory.")
print(f" Run manually: litellm --config {config_path} --port {port}")
return
litellm_bin = os.path.join(scripts_dir, "litellm")| ProxyInitializationHelpers._maybe_setup_prometheus_multiproc_dir( | ||
| num_workers=num_workers, | ||
| litellm_settings=litellm_settings if config else None, | ||
| litellm_settings=litellm_settings if config else None, # type: ignore[possibly-unbound] |
There was a problem hiding this comment.
type: ignore[possibly-unbound] masks a real NameError risk
The # type: ignore[possibly-unbound] comment suppresses Pyright's warning that litellm_settings may not be assigned before this call. This variable is conditionally populated by the config-loading branch earlier in run_server; if config is falsy the conditional litellm_settings if config else None guards against using an unbound name at runtime, but only in this one specific call site.
If the code path is ever refactored and this guard is removed (or a second call site is introduced without it), a real NameError could reach production. The safer fix is to ensure litellm_settings has a well-typed default at the point of assignment, which would remove the need for the suppression comment entirely:
litellm_settings: Optional[dict] = None
# ... (existing config-loading block conditionally assigns litellm_settings)
ProxyInitializationHelpers._maybe_setup_prometheus_multiproc_dir(
num_workers=num_workers,
litellm_settings=litellm_settings, # no type: ignore needed
)
litellm/setup_wizard.py
Outdated
| if retry == "y": | ||
| del env_vars[p["env_key"]] | ||
| # Re-prompt by looping — restart key collection for this provider | ||
| while True: | ||
| key = input(f" {blue('❯')} {bold(p['name'])} API key {hint}: ").strip() | ||
| if not key: | ||
| break | ||
| env_vars[p["env_key"]] = key | ||
| print(f" {grey('Testing credentials…')}", end="", flush=True) | ||
| error = _test_provider_key(p, env_vars) | ||
| if error is None: | ||
| print(f"\r {green(_CHECK + ' ' + p['name'])} credentials valid ") | ||
| break | ||
| print(f"\r {_c(_BOLD, '✘')} {bold(p['name'])} {grey(error)}") |
There was a problem hiding this comment.
Orphaned extra credentials after retry abandonment
When credential validation fails and the user chooses to retry (retry == "y"), the primary key is deleted from env_vars (line 402), but any needs_extra keys that were already collected (e.g., AWS_SECRET_ACCESS_KEY, AWS_REGION_NAME for Bedrock) are not cleaned up. If the user then abandons the retry by pressing Enter (empty key, line 406–407), env_vars retains those extra keys without the primary key.
These orphaned keys — which are not filtered by the _LITELLM_ prefix guard — will be written into the environment_variables: block of the generated YAML. The resulting config will contain AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME without AWS_ACCESS_KEY_ID, leaking partial credentials and producing a config that fails at runtime with a confusing auth error.
The retry path should clean up all keys collected for the provider before re-prompting:
if retry == "y":
del env_vars[p["env_key"]]
# Also clean up any extra keys collected for this provider
for extra_key in p.get("extra_keys", []):
env_vars.pop(extra_key, None)
# Clean up azure api base sentinel
env_vars.pop(f"_LITELLM_AZURE_API_BASE_{p['id'].upper()}", None)
# Re-prompt by looping — restart key collection for this provider
while True:
...
…bound variable error
…led litellm binary
…terminal, not exhausted pipe
…o re-runs are faster
…t-steps after proxy start
…y from litellm.utils
…name collisions, tests - setup_wizard.py: add _yaml_escape() for safe YAML embedding of API keys - setup_wizard.py: add _styled_input() with readline ANSI ignore markers - setup_wizard.py: change DIVIDER to _divider() fn to avoid import-time color capture - setup_wizard.py: validate port range 1-65535, initialize before loop - setup_wizard.py: qualify azure display names (azure-gpt-4o) to avoid collision with openai - setup_wizard.py: work on env_copy in _build_config to avoid mutating caller's dict - setup_wizard.py: skip model_list entries for providers with no credentials - setup_wizard.py: prompt for azure deployment name - setup_wizard.py: wrap os.execlp in try/except with friendly fallback - setup_wizard.py: wrap config write in try/except OSError - setup_wizard.py: fix _validate_and_report to use two print lines (no \r overwrite) - setup_wizard.py: add .gitignore tip next to key storage notice - setup_wizard.py: fix run_setup_wizard() return type annotation to None - scripts/install.sh: drop pipefail (not supported by dash on Ubuntu when invoked as sh) - scripts/install.sh: use litellm[proxy] from PyPI (not hardcoded dev branch) - scripts/install.sh: guard /dev/tty read with -r check for Docker/CI compat - scripts/install.sh: remove --force-reinstall to avoid downgrading dependencies - tests/test_litellm/test_setup_wizard.py: 13 unit tests for _build_config and _yaml_escape
…, credential flow - guard termios/tty imports with try/except ImportError for Windows compat - quote master_key as YAML double-quoted scalar (same as env vars) - remove unused port param from _build_config signature - _validate_and_report now returns the final key so re-entered creds are stored - add test for master_key YAML quoting
- _yaml_escape: add control character escaping (\n, \r, \t) - test: fix tautological assertion in test_build_config_azure_no_deployment_skipped - test: add tests for control character escaping in _yaml_escape
ishaan-jaff
force-pushed
the
worktree-dynamic-tickling-journal
branch
from
61c1387 to
0e77077
Compare
* feat(xai): add grok-4.20 beta 2 models with pricing (#23900) Add three grok-4.20 beta 2 model variants from xAI: - grok-4.20-multi-agent-beta-0309 (reasoning + multi-agent) - grok-4.20-beta-0309-reasoning (reasoning) - grok-4.20-beta-0309-non-reasoning Pricing (from https://docs.x.ai/docs/models): - Input: $2.00/1M tokens ($0.20/1M cached) - Output: $6.00/1M tokens - Context: 2M tokens All variants support vision, function calling, tool choice, and web search. Closes LIT-2171 * docs: add Quick Install section for litellm --setup wizard (#23905) * docs: add Quick Install section for litellm --setup wizard * docs: clarify setup wizard is for local/beginner use * feat(setup): interactive setup wizard + install.sh (#23644) * feat(setup): add interactive setup wizard + install.sh Adds `litellm --setup` — a Claude Code-style TUI onboarding wizard that guides users through provider selection, API key entry, and proxy config generation, then optionally starts the proxy immediately. - litellm/setup_wizard.py: wizard with ASCII art, numbered provider menu (OpenAI, Anthropic, Azure, Gemini, Bedrock, Ollama), API key prompts, port/master-key config, and litellm_config.yaml generation - litellm/proxy/proxy_cli.py: adds --setup flag that invokes the wizard - scripts/install.sh: curl-installable script (detect OS/Python, pip install litellm[proxy], launch wizard) Usage: curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh litellm --setup * fix(install.sh): remove orange color, add LITELLM_BRANCH env var for branch installs * fix(install.sh): install from git branch so --setup is available for QA * fix(install.sh): remove stale LITELLM_BRANCH reference that caused unbound variable error * fix(install.sh): force-reinstall from git to bypass cached PyPI version * fix(install.sh): show pip progress bar during install * fix(install.sh): always launch wizard via $PYTHON_BIN -m litellm, not PATH binary * fix(install.sh): use litellm.proxy.proxy_cli module (no __main__.py exists) * fix(install.sh): suppress RuntimeWarning from module invocation * fix(install.sh): use Python bin-dir litellm binary to avoid CWD sys.path shadowing * fix(install.sh): use sysconfig.get_path('scripts') to find pip-installed litellm binary * fix(install.sh): redirect stdin from /dev/tty on exec so wizard gets terminal, not exhausted pipe * fix(install.sh): warn about git clone duration, drop --no-cache-dir so re-runs are faster * feat(setup_wizard): arrow-key selector, updated model names * fix(setup_wizard): use sysconfig binary to start proxy, not python -m litellm * feat(setup_wizard): credential validation after key entry + clear next-steps after proxy start * style(install.sh): show git clone warning in blue * refactor(setup_wizard): class with static methods, use check_valid_key from litellm.utils * address greptile review: fix yaml escaping, port validation, display name collisions, tests - setup_wizard.py: add _yaml_escape() for safe YAML embedding of API keys - setup_wizard.py: add _styled_input() with readline ANSI ignore markers - setup_wizard.py: change DIVIDER to _divider() fn to avoid import-time color capture - setup_wizard.py: validate port range 1-65535, initialize before loop - setup_wizard.py: qualify azure display names (azure-gpt-4o) to avoid collision with openai - setup_wizard.py: work on env_copy in _build_config to avoid mutating caller's dict - setup_wizard.py: skip model_list entries for providers with no credentials - setup_wizard.py: prompt for azure deployment name - setup_wizard.py: wrap os.execlp in try/except with friendly fallback - setup_wizard.py: wrap config write in try/except OSError - setup_wizard.py: fix _validate_and_report to use two print lines (no \r overwrite) - setup_wizard.py: add .gitignore tip next to key storage notice - setup_wizard.py: fix run_setup_wizard() return type annotation to None - scripts/install.sh: drop pipefail (not supported by dash on Ubuntu when invoked as sh) - scripts/install.sh: use litellm[proxy] from PyPI (not hardcoded dev branch) - scripts/install.sh: guard /dev/tty read with -r check for Docker/CI compat - scripts/install.sh: remove --force-reinstall to avoid downgrading dependencies - tests/test_litellm/test_setup_wizard.py: 13 unit tests for _build_config and _yaml_escape * style: black format setup_wizard.py * fix: address remaining greptile issues - Windows compat, YAML quoting, credential flow - guard termios/tty imports with try/except ImportError for Windows compat - quote master_key as YAML double-quoted scalar (same as env vars) - remove unused port param from _build_config signature - _validate_and_report now returns the final key so re-entered creds are stored - add test for master_key YAML quoting * fix: add --port to suggested command, guard /dev/tty exec in install.sh * fix: quote api_base in YAML, skip azure if no deployment, only redraw on state change * fix: address greptile review comments - _yaml_escape: add control character escaping (\n, \r, \t) - test: fix tautological assertion in test_build_config_azure_no_deployment_skipped - test: add tests for control character escaping in _yaml_escape * feat(ui): remove Chat UI page link and banner from sidebar and playground (#23908) * feat(guardrails): MCPJWTSigner - built-in guardrail for zero trust MCP auth (#23897) * Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers * Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present. * Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings. * Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers * Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present. * Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings. * feat(guardrails): add MCPJWTSigner built-in guardrail for zero trust MCP auth Signs outbound MCP tool calls with a LiteLLM-issued RS256 JWT so MCP servers can trust a single signing authority instead of every upstream IdP. Enable in config.yaml: guardrails: - guardrail_name: mcp-jwt-signer litellm_params: guardrail: mcp_jwt_signer mode: pre_mcp_call default_on: true JWT carries sub (user_id), act.sub (team_id, RFC 8693), tool-level scope, iss, aud, iat/exp/nbf. RSA-2048 keypair auto-generated at startup unless MCP_JWT_SIGNING_KEY env var is set. Adds /.well-known/jwks.json endpoint and jwks_uri to /.well-known/openid-configuration so MCP servers can verify LiteLLM-issued tokens via OIDC discovery. * Update MCPServerManager to raise HTTPException with status code 400 for extra headers in OpenAPI-backed servers. Adjust tests to verify the correct status code and exception message. * fix: address P1 issues in MCPJWTSigner - OpenAPI servers: warn + skip header injection instead of 500 - JWKS Cache-Control: 5min for auto-generated keys, 1h for persistent - sub claim: fallback to apikey:{token_hash} for anonymous callers - ttl_seconds: validate > 0 at init time * docs: add MCP zero trust auth guide with architecture diagram * docs: add FastMCP JWT verification guide to zero trust doc * fix: address remaining Greptile review issues (round 2) - mcp_server_manager: warn when hook Authorization overwrites existing header - __init__: remove _mcp_jwt_signer_instance from __all__ (private internal) - discoverable_endpoints: copy dict instead of mutating in-place on OIDC augmentation - test docstring: reflect warn-and-continue behavior for OpenAPI servers - test: update scope assertions for least-privilege (no mcp:tools/list on tool-call JWTs) * fix: address Greptile round 3 feedback - initialize_guardrail: validate mode='pre_mcp_call' at init time — misconfigured mode silently bypasses JWT injection, which is a zero-trust bypass - _build_claims: remove duplicate inline 'import re' (module-level import already present) - _types.py: add TODO comment explaining jwt_claims is forward-compat plumbing for a follow-up PR that will forward upstream IdP claims into outbound MCP JWTs * feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes Addresses all missing pieces from the scoping doc review: FR-5 (Verify + re-sign): MCPJWTSigner now accepts access_token_discovery_uri and token_introspection_endpoint. When set, the incoming Bearer token is extracted from raw_headers (threaded through pre_call_tool_check), verified against the IdP's JWKS (JWT) or introspected (opaque), and only re-signed if valid. Falls back to user_api_key_dict.jwt_claims for LiteLLM JWT-auth mode. FR-12 (Configurable end-user identity mapping): end_user_claim_sources ordered list drives sub resolution — sources: token:<claim>, litellm:user_id, litellm:email, litellm:end_user_id, litellm:team_id. FR-13 (Claim operations): add_claims (insert-if-absent), set_claims (always override), remove_claims (delete) applied in that order. FR-14 (Two-token model): channel_token_audience + channel_token_ttl issue a second JWT injected as x-mcp-channel-token: Bearer <token>. FR-15 (Incoming claim validation): required_claims raises HTTP 403 when any listed claim is absent; optional_claims passes listed claims from verified token into the outbound JWT. FR-9 (Debug headers): debug_headers: true emits x-litellm-mcp-debug with kid, sub, iss, exp, scope. FR-10 (Configurable scopes): allowed_scopes replaces auto-generation. Also fixed: tool-call JWTs no longer grant mcp:tools/list (overpermission). P1 fixes: - proxy/utils.py: _convert_mcp_hook_response_to_kwargs merges rather than replaces extra_headers, preserving headers from prior guardrails. - mcp_server_manager.py: warns when hook injects Authorization alongside a server-configured authentication_token (previously silent). - mcp_server_manager.py: pre_call_tool_check now accepts raw_headers and extracts incoming_bearer_token so FR-5 verification has the raw token. - proxy/utils.py: remove stray inline import inspect inside loop (pre-existing lint error, now cleaned up). Tests: 43 passing (28 new tests covering all FR flags + P1 fixes). * feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes (core) Remaining files from the FR implementation: mcp_jwt_signer.py — full rewrite with all new params: FR-5: access_token_discovery_uri, token_introspection_endpoint, verify_issuer, verify_audience + _verify_incoming_jwt(), _introspect_opaque_token() FR-12: end_user_claim_sources ordered resolution chain FR-13: add_claims, set_claims, remove_claims FR-14: channel_token_audience, channel_token_ttl → x-mcp-channel-token FR-15: required_claims (raises 403), optional_claims (passthrough) FR-9: debug_headers → x-litellm-mcp-debug FR-10: allowed_scopes; tool-call JWTs no longer over-grant tools/list mcp_server_manager.py: - pre_call_tool_check gains raw_headers param to extract incoming_bearer_token - Silent Authorization override warning fixed: now fires when server has authentication_token AND hook injects Authorization tests/test_mcp_jwt_signer.py: 28 new tests covering all FR flags + P1 fixes (43 total, all passing) * fix(mcp_jwt_signer): address pre-landing review issues - Remove stale TODO comment on UserAPIKeyAuth.jwt_claims — the field is already populated and consumed by MCPJWTSigner in the same PR - Fix _get_oidc_discovery to only cache the OIDC discovery doc when jwks_uri is present; a malformed/empty doc now retries on the next request instead of being permanently cached until proxy restart - Add FR-5 test coverage for _fetch_jwks (cache hit/miss), _get_oidc_discovery (cache/no-cache on bad doc), _verify_incoming_jwt (valid token, expired token), _introspect_opaque_token (active, inactive, no endpoint), and the end-to-end 401 hook path — 53 tests total, all passing * docs(mcp_zero_trust): rewrite as use-case guide covering all new JWT signer features Add scenario-driven sections for each new config area: - Verify+re-sign with Okta/Azure AD (access_token_discovery_uri, end_user_claim_sources, token_introspection_endpoint) - Enforcing caller attributes with required_claims / optional_claims - Adding metadata via add_claims / set_claims / remove_claims - Two-token model for AWS Bedrock AgentCore Gateway (channel_token_audience / channel_token_ttl) - Controlling scopes with allowed_scopes - Debugging JWT rejections with debug_headers Update JWT claims table to reflect configurable sub (end_user_claim_sources) * fix(mcp_jwt_signer): wire all config.yaml params through initialize_guardrail The factory was only passing issuer/audience/ttl_seconds to MCPJWTSigner. All FR-5/9/10/12/13/14/15 params (access_token_discovery_uri, end_user_claim_sources, add/set/remove_claims, channel_token_audience, required/optional_claims, debug_headers, allowed_scopes, etc.) were silently dropped, making every advertised advanced feature non-functional when loaded from config.yaml. Add regression test that asserts every param is wired through correctly. * docs(mcp_zero_trust): add hero image * docs(mcp_zero_trust): apply Linear-style edits - Lead with the problem (unsigned direct calls bypass access controls) - Shorter statement section headers instead of question-form headers - Move diagram/OIDC discovery block after the reader is bought in - Add 'read further only if you need to' callout after basic setup - Two-token section now opens from the user problem not product jargon - Add concrete 403 error response example in required_claims section - Debug section opens from the symptom (MCP server returning 401) - Lowercase claims reference header for consistency * fix(mcp_jwt_signer): fix algorithm confusion attack + add OIDC discovery 24h TTL - Remove alg from unverified JWT header; use signing_jwk.algorithm_name from JWKS key instead. Reading alg from attacker-controlled headers enables alg:none / HS256 confusion attacks. - Add _oidc_discovery_fetched_at timestamp and _OIDC_DISCOVERY_TTL = 86400 (24h). Without a TTL the cached discovery doc never refreshes, so IdP key rotation is invisible. --------- Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com> * fix(ci): stabilize CI - formatting, type errors, test polling, security CVEs, router bug, batch resolution Fix 1: Run Black formatter on 35 files Fix 2: Fix MyPy type errors: - setup_wizard.py: add type annotation for 'selected' set variable - user_api_key_auth.py: remove redundant type annotation on jwt_claims reassignment Fix 3: Fix spend accuracy test burst 2 polling to wait for expected total spend instead of just 'any increase' from burst 2 Fix 4: Bump Next.js 16.1.6 -> 16.1.7 to fix CVE-2026-27978, CVE-2026-27979, CVE-2026-27980, CVE-2026-29057 Fix 5: Fix router _pre_call_checks model variable being overwritten inside loop, causing wrong model lookups on subsequent deployments. Use local _deployment_model variable instead. Fix 6: Add missing resolve_output_file_ids_to_unified call in batch retrieve non-terminal-to-terminal path (matching the terminal path behavior) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * chore: regenerate poetry.lock to sync with pyproject.toml Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix: format merged files from main and regenerate poetry.lock Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(mypy): annotate jwt_claims as Optional[dict] to fix type incompatibility Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): update router region test to use gpt-4.1-mini (fix flaky model lookup) Replace deprecated gpt-3.5-turbo-1106 with gpt-4.1-mini + mock_response in test_router_region_pre_call_check, following the same pattern used in commit 717d37c for test_router_context_window_check_pre_call_check_out_group. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * ci: retry flaky logging_testing (async event loop race condition) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): aggregate all mock calls in langfuse e2e test to fix race condition The _verify_langfuse_call helper only inspected the last mock call (mock_post.call_args), but the Langfuse SDK may split trace-create and generation-create events across separate HTTP flush cycles. This caused an IndexError when the last call's batch contained only one event type. Fix: iterate over mock_post.call_args_list to collect batch items from ALL calls. Also add a safety assertion after filtering by trace_id and mark all langfuse e2e tests with @pytest.mark.flaky(retries=3) as an extra safety net for any residual timing issues. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): black formatting + update OpenAPI compliance tests for spec changes - Apply Black 26.x formatting to litellm_logging.py (parenthesized style) - Update test_input_types_match_spec to follow $ref to InteractionsInput schema (Google updated their OpenAPI spec to use $ref instead of inline oneOf) - Update test_content_schema_uses_discriminator to handle discriminator without explicit mapping (Google removed the mapping key from Content discriminator) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * revert: undo incorrect Black 26.x formatting on litellm_logging.py The file was correctly formatted for Black 23.12.1 (the version pinned in pyproject.toml). The previous commit applied Black 26.x formatting which was incompatible with the CI's Black version. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): deduplicate and sort langfuse batch events after aggregation The Langfuse SDK may send the same event (e.g., trace-create) in multiple flush cycles, causing duplicates when we aggregate from all mock calls. After filtering by trace_id, deduplicate by keeping only the first event of each type, then sort to ensure trace-create is at index 0 and generation-create at index 1. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> --------- Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
- Add xai/grok-4.20-beta-0309-reasoning (3rd xAI model, was missing) - Update New Model count 11 → 12 - Fix supports_minimal_reasoning_effort description (full gpt-5.x series) - Add Akto guardrail integration (BerriAI#23250) - Add MCP JWT Signer guardrail (BerriAI#23897) - Add pre_mcp_call header mutation (BerriAI#23889) - Add litellm --setup wizard (BerriAI#23644) - Fix ### Bug Fixes → #### Bugs under New Models - Add missing Documentation Updates section - Rename Diff Summary "AI Integrations" → "Logging / Guardrail / Prompt Management Integrations" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 participant
What
Adds
litellm --setup— a Claude Code-style TUI onboarding wizard + a curl-installableinstall.sh.Demo flow:
Curl install:
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | shChanges
litellm/setup_wizard.py— TUI wizard (pure stdlib + ANSI, no extra deps)litellm/proxy/proxy_cli.py— adds--setupflagscripts/install.sh— detects OS/Python,pip install litellm[proxy], launches wizardTesting
Tested: wizard runs end-to-end, config generates valid YAML, proxy starts with
"I'm alive!"health, models load from config (gpt-4o,gpt-4o-mini).Pre-Submission Checklist
make test-unitcompatible (no new test deps)Type
CI (LiteLLM team)