feat(cli): add lite autoroute to QA complexity-based auto-routing against a real proxy - #33249
Merged
Merged
Conversation
…inst a real proxy Lets a customer try litellm's complexity_router against models they already have on their existing, unmodified production proxy, with no config.yaml edits and no new infra. lite autoroute configure discovers accessible models via /model_group/info and walks through tier assignment (plus optional LLM classifier / semantic matching / adaptive selection); every referenced model becomes its own litellm_proxy/<name> deployment forwarding back to the real proxy with the real key, so every actual call, routed completions, classifier calls, embedding calls, still lands on their real proxy. lite autoroute up launches that generated config as an ephemeral local proxy, patches ~/.claude/settings.json to point Claude Code at it, and streams routing decisions live; Ctrl-C/SIGTERM (or lite autoroute down after an unclean exit) restores everything. Also adds lite model-groups list (a thin CLI wrapper over the existing ModelGroupsManagementClient), and generalizes up.py's settings-backup/restore helpers to take explicit paths so this feature can reuse them instead of duplicating the logic. Depends on litellm_lite_up_down (#33231) for that generalization.
|
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 60d3d05704
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
Greptile SummaryThis PR adds
Confidence Score: 5/5Safe to merge; no blocking defects found in the changed code paths. Previously flagged issues (stale backup overwrite, uncaught ValidationError on empty config, orphaned subprocess after health-check timeout) are all addressed in the final state. The remaining findings are both non-blocking quality suggestions: the proxy log file is created with default permissions rather than 0600, and process.py — log file permissions and port allocation.
|
| Filename | Overview |
|---|---|
| litellm/proxy/client/cli/commands/autoroute/commands.py | Core up/down/configure command orchestration. Stale-backup guard, try/except around YAML load, and orphaned-process cleanup are all addressed. _mint_and_embed_master_key correctly targets general_settings.master_key. |
| litellm/proxy/client/cli/commands/autoroute/process.py | Process lifecycle helpers. Two issues: log file created with default permissions (potential info leak) and TOCTOU race between port allocation and proxy bind. Both are P2. |
| litellm/proxy/client/cli/commands/autoroute/config.py | Config model and generation logic. Handles mode: null from real proxies, deduplicates model deployments, correctly places master_key under general_settings. Clean. |
| litellm/proxy/client/cli/commands/autoroute/wizard.py | Configure wizard using InquirerPy fuzzy picker. Properly guards non-list API responses with ClickException, checks interactivity upfront, and writes config with secure_create (0600). |
| litellm/proxy/client/cli/commands/autoroute/settings.py | Claude Code settings patching. Clears apiKeyHelper, sets ANTHROPIC_AUTH_TOKEN + BASE_URL, forces all default model tier env vars to 'autorouter'. Correct and well-tested. |
| litellm/proxy/client/cli/commands/up.py | Refactored write_backup/read_backup/restore_claude_settings to accept explicit paths; added secure_create context manager; load_json_or_empty now handles empty files. Non-breaking for existing lite up/lite down. |
| litellm/proxy/client/cli/commands/model_groups.py | Thin lite model-groups list command. Guards non-list response with ClickException (replacing the previous bare assert). Table and JSON output modes work correctly. |
| scripts/install-cli.sh | Adds LITELLM_CLI_REF opt-in for source installs. Unset defaults to PyPI release unchanged. The variable is properly double-quoted when passed to uv, preventing word-splitting issues. |
| tests/test_litellm/proxy/client/cli/autoroute/test_commands.py | Good coverage: refuses on missing config, live PID, stale backup, health-check failure; happy path validates settings patching, permissions, backup/restore lifecycle, and master-key embedding. |
| tests/test_litellm/proxy/client/cli/autoroute/test_wizard.py | Tests both the wizard orchestration (mocking the picker) and the real InquirerPy widget via prompt_toolkit pipe input. Timing-based async tests could be fragile on very slow CI but are reasonable for widget testing. |
Reviews (7): Last reviewed commit: "fix(cli): bind the ephemeral autoroute p..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| def _teardown() -> None: | ||
| if not restored.acquire(blocking=False): | ||
| return | ||
| terminate(process.pid) |
Contributor
There was a problem hiding this comment.
Medium: Released proxy port can be impersonated
A Claude process that started during up retains the loopback URL after teardown. Once this process is terminated, another local account can bind the released port and receive subsequent prompts and the authorization token from that still-running Claude session. The endpoint needs a lifecycle that remains bound until its clients exit, such as launching Claude as a managed child or retaining a non-forwarding guard listener; restoring the settings file alone does not update existing processes.
Contributor
Author
There was a problem hiding this comment.
Accepting this as a known, documented tradeoff rather than fixing in code for this PR: a real fix (e.g. a guard listener that keeps the port bound and inert until lite autoroute up/down explicitly reclaims it) is a meaningfully larger change than this PR's scope, and the underlying risk (a Claude Code session that outlives up keeps sending to a now-unbound port) is the same one-time-patch tradeoff lite up already ships with, just with a static token instead of a re-resolved one. This is now called out explicitly in the README's Caveats section and in the up/teardown CLI output itself, so users see it at the point of risk. Leaving this open rather than silently dismissing it.
Contributor
PR overviewThis pull request adds a CLI “lite autoroute” workflow for QA of complexity-based routing through a real LiteLLM proxy. The touched autoroute command code manages starting, configuring, and tearing down local proxy-backed Claude sessions for that flow. There is one open security concern remaining after three prior issues were addressed. The remaining issue is a local lifecycle problem: after teardown, a still-running client can keep using a loopback endpoint whose port may be rebound by another local user, exposing subsequent prompts and an authorization token. The risk is meaningful on shared machines but depends on local access and the teardown/client timing, so the blast radius is limited. Open issues (1)
Fixed/addressed: 3 · PR risk: 6/10 |
complexity_router already supports a pool of models per tier (randomly picked per request; adaptive mode specifically needs a pool to choose within), but the configure wizard only ever let you assign one. Tiers are now a tuple of model names; the wizard prompt accepts comma-separated indices to pick more than one per tier.
Numbered-index selection didn't scale past a handful of models, so switch
the tier picker to InquirerPy's fzf-style fuzzy search. Also set
ANTHROPIC_DEFAULT_{SONNET,HAIKU,OPUS}_MODEL to "autorouter" in Claude
Code's settings, since Router resolves auto-router deployments by literal
model name with no wildcard support, so a "*" catch-all model_name would
never match real traffic.
Lets testers try an unreleased branch's CLI changes with the same curl-piped installer, instead of waiting for a PyPI release.
Clears osv-scan findings for PYSEC-2026-3444 and PYSEC-2026-3447.
commands.py wrote config.yaml (embeds the real proxy key) and Claude Code's settings.json (embeds the ephemeral proxy's master key) with plain open(), landing at the umask-derived default (commonly 0644) until a later chmod call caught up. That window, and the missed case where settings.json already exists (chmod never ran at all there), left a credential-bearing file readable by another local account. secure_create() fixes the mode via fchmod on the fd before any content is written, covering both the brand-new-file and already-exists cases, and commands.py/wizard.py now route their sensitive writes through it.
Contributor
Author
|
@greptile review |
…ed port lite autoroute up's master key is embedded statically (unlike lite up's apiKeyHelper, resolved per request), so a Claude Code session still running after teardown keeps sending it, along with prompt content, to a now-unbound loopback port that another local account can bind. This is the same one-time-patch tradeoff lite up already accepts, just with a static secret instead of a re-resolved one -- document it in the README's Caveats section and surface it in the teardown message itself.
- terminate the ephemeral proxy child process when its health check fails, instead of leaking an orphaned, unrecoverable process bound to the port - replace bare assert isinstance checks (no-ops under python -O) with click.ClickException in the model-groups list and configure wizard code paths - close launch_proxy's log file handle once the child process has inherited its fd, instead of leaking it - add build_generated_proxy_config to config.py's __all__
Contributor
Author
|
@greptile review |
write_backup wrote the backup (which can embed the original apiKeyHelper/settings content) with plain open() + a chmod call after the fact -- the same permissive-until-corrected window already fixed for autoroute's config.yaml and Claude settings writes, and missed entirely when the backup file already exists with broader permissions. Moves secure_create (atomic-enough 0600 via fchmod before any content is written) to up.py, the module both lite up and lite autoroute share, and has autoroute/process.py import it from there instead of keeping its own copy.
Contributor
Author
|
@greptile review |
ishaan-berri
requested changes
Jul 15, 2026
| LITELLM_PACKAGE="litellm[cli]" | ||
| # Defaults to the PyPI release; LITELLM_CLI_REF opts into installing from source instead. | ||
| if [ -n "${LITELLM_CLI_REF:-}" ]; then | ||
| LITELLM_PACKAGE="litellm[cli] @ git+https://github.com/BerriAI/litellm.git@${LITELLM_CLI_REF}" |
Contributor
There was a problem hiding this comment.
isn't this a security issue, why are we now installing from git vs using the package ?
Contributor
Author
There was a problem hiding this comment.
not a security issue, in my opinion, as user is specifying the version they want to install.
the goal is to allow a user to test the autorouter without waiting on a litellm release (which can take longer)
Contributor
Author
There was a problem hiding this comment.
if you want i can have it be a notice to them on startup or on the doc - that this is an experimental, fast method
The pid-record check only catches a still-live duplicate process; a SIGKILL'd `up` leaves no live pid but does leave AUTOROUTE_BACKUP_PATH behind. Without this guard, a fresh `up` overwrote that backup with the currently-patched Claude settings instead of the true originals, so `down`/Ctrl-C would restore the wrong content permanently. up.py's `lite up` already guards the analogous case; mirror it here.
proxy_cli.py defaults --host to 0.0.0.0 when not passed explicitly. launch_proxy never passed it, so the ephemeral proxy -- despite every base_url in this module being built from 127.0.0.1 -- was actually reachable from other hosts on the network, including its unauthenticated-until-config-lands routes before the master key is wired in.
Contributor
Author
|
@greptile review |
krrish-berri-2
merged commit Jul 15, 2026
52591ba
into
litellm_lite_up_down
101 of 102 checks passed
Contributor
Author
|
@greptile review |
Contributor
Author
|
@greptile review |
krrish-berri-2
added a commit
that referenced
this pull request
Jul 16, 2026
…e proxy (#33231) * feat(cli): add `lite up`/`lite down` to ambiently route Claude Code through the proxy Patches ~/.claude/settings.json in place (env.ANTHROPIC_BASE_URL + apiKeyHelper via `lite auth print-token`) so any `claude` session started afterward, from any terminal, routes through the local LiteLLM proxy with no wrapper command needed, unlike the existing `lite claude` subprocess-exec approach. Backs up the original file first and restores it on Ctrl-C/SIGTERM, or via `lite down` after an unclean exit. Cursor is not supported: no equivalent file-based config to patch. * feat(cli): add lite autoroute to QA complexity-based auto-routing against a real proxy (#33249) * feat(cli): add lite autoroute to QA complexity-based auto-routing against a real proxy Lets a customer try litellm's complexity_router against models they already have on their existing, unmodified production proxy, with no config.yaml edits and no new infra. lite autoroute configure discovers accessible models via /model_group/info and walks through tier assignment (plus optional LLM classifier / semantic matching / adaptive selection); every referenced model becomes its own litellm_proxy/<name> deployment forwarding back to the real proxy with the real key, so every actual call, routed completions, classifier calls, embedding calls, still lands on their real proxy. lite autoroute up launches that generated config as an ephemeral local proxy, patches ~/.claude/settings.json to point Claude Code at it, and streams routing decisions live; Ctrl-C/SIGTERM (or lite autoroute down after an unclean exit) restores everything. Also adds lite model-groups list (a thin CLI wrapper over the existing ModelGroupsManagementClient), and generalizes up.py's settings-backup/restore helpers to take explicit paths so this feature can reuse them instead of duplicating the logic. Depends on litellm_lite_up_down (#33231) for that generalization. * feat(cli): allow multiple models per autoroute tier complexity_router already supports a pool of models per tier (randomly picked per request; adaptive mode specifically needs a pool to choose within), but the configure wizard only ever let you assign one. Tiers are now a tuple of model names; the wizard prompt accepts comma-separated indices to pick more than one per tier. * feat(cli): fuzzy model picker and auto-route Claude Code to autorouter Numbered-index selection didn't scale past a handful of models, so switch the tier picker to InquirerPy's fzf-style fuzzy search. Also set ANTHROPIC_DEFAULT_{SONNET,HAIKU,OPUS}_MODEL to "autorouter" in Claude Code's settings, since Router resolves auto-router deployments by literal model name with no wildcard support, so a "*" catch-all model_name would never match real traffic. * feat(cli): allow installing lite CLI from source via LITELLM_CLI_REF Lets testers try an unreleased branch's CLI changes with the same curl-piped installer, instead of waiting for a PyPI release. * fix(ci): modernize type hints to clear ruff strict-rule budget * fix(ci): bump httplib2 and setuptools to patched versions Clears osv-scan findings for PYSEC-2026-3444 and PYSEC-2026-3447. * fix(cli): write autoroute's secret-bearing files with mode 0600 commands.py wrote config.yaml (embeds the real proxy key) and Claude Code's settings.json (embeds the ephemeral proxy's master key) with plain open(), landing at the umask-derived default (commonly 0644) until a later chmod call caught up. That window, and the missed case where settings.json already exists (chmod never ran at all there), left a credential-bearing file readable by another local account. secure_create() fixes the mode via fchmod on the fd before any content is written, covering both the brand-new-file and already-exists cases, and commands.py/wizard.py now route their sensitive writes through it. * docs(cli): warn that a stale Claude Code session can leak to a squatted port lite autoroute up's master key is embedded statically (unlike lite up's apiKeyHelper, resolved per request), so a Claude Code session still running after teardown keeps sending it, along with prompt content, to a now-unbound loopback port that another local account can bind. This is the same one-time-patch tradeoff lite up already accepts, just with a static secret instead of a re-resolved one -- document it in the README's Caveats section and surface it in the teardown message itself. * fix(cli): address greptile review feedback on autoroute PR - terminate the ephemeral proxy child process when its health check fails, instead of leaking an orphaned, unrecoverable process bound to the port - replace bare assert isinstance checks (no-ops under python -O) with click.ClickException in the model-groups list and configure wizard code paths - close launch_proxy's log file handle once the child process has inherited its fd, instead of leaking it - add build_generated_proxy_config to config.py's __all__ * fix(cli): close TOCTOU window in lite up's settings backup write write_backup wrote the backup (which can embed the original apiKeyHelper/settings content) with plain open() + a chmod call after the fact -- the same permissive-until-corrected window already fixed for autoroute's config.yaml and Claude settings writes, and missed entirely when the backup file already exists with broader permissions. Moves secure_create (atomic-enough 0600 via fchmod before any content is written) to up.py, the module both lite up and lite autoroute share, and has autoroute/process.py import it from there instead of keeping its own copy. * fix(cli): refuse autoroute up when a stale backup exists from a crash The pid-record check only catches a still-live duplicate process; a SIGKILL'd `up` leaves no live pid but does leave AUTOROUTE_BACKUP_PATH behind. Without this guard, a fresh `up` overwrote that backup with the currently-patched Claude settings instead of the true originals, so `down`/Ctrl-C would restore the wrong content permanently. up.py's `lite up` already guards the analogous case; mirror it here. * fix(cli): bind the ephemeral autoroute proxy to loopback only proxy_cli.py defaults --host to 0.0.0.0 when not passed explicitly. launch_proxy never passed it, so the ephemeral proxy -- despite every base_url in this module being built from 127.0.0.1 -- was actually reachable from other hosts on the network, including its unauthenticated-until-config-lands routes before the master key is wired in. * docs(cli): show curl install for the autoroute QA flow Points readers at scripts/install-cli.sh's curl one-liner instead of assuming uv/pip is already set up, and documents the LITELLM_CLI_REF override for trying an unreleased branch or commit. * fix(cli): surface a clean error on an empty or corrupt autoroute config A configure run killed between secure_create's O_TRUNC and the write completing leaves an empty config.yaml on disk. The next up read that via yaml.safe_load (None) into the generated-config TypeAdapter uncaught, surfacing a raw pydantic.ValidationError instead of pointing the user back at `lite autoroute configure`. * fix(cli): bind lite up's apiKeyHelper to the proxy it was started against _ensure_fresh_login only checked token freshness, not which proxy the cached token belonged to, and resolve_api_key_helper built a bare `lite auth print-token` command with no --base-url. A user logged into proxy A who ran `up --base-url proxy-b` (or LITELLM_PROXY_URL=proxy-b) would silently get proxy A's real token wired into Claude Code's apiKeyHelper; since apiKeyHelper is invoked bare, print-token's existing origin check never engaged, so proxy B -- attacker-controlled or not -- received every subsequent request's Authorization header carrying proxy A's credential. _ensure_fresh_login now requires the cached token's base_url to match before treating it as usable, forcing a fresh login for the selected proxy otherwise. resolve_api_key_helper now takes that base_url and threads it through as an explicit --base-url, so print-token's existing (but previously unreachable in the apiKeyHelper flow) base_url_explicit check actually enforces the match at request time too. * fix(cli): surface clean errors instead of raw tracebacks in lite up/down load_json_or_empty and read_backup both delegate to pydantic's validate_json, which raises ValidationError on invalid JSON or a non-object root -- neither up() nor down() caught it, so a corrupt settings or backup file surfaced an unformatted Python traceback instead of a clean CLI error. Both now convert to UpError, and down() (previously uncaught entirely) and up()'s teardown path now handle it. restore_claude_settings also gained a parent.mkdir guard before rewriting CLAUDE_SETTINGS_PATH: if ~/.claude/ was removed while `lite up` was running, the restore would crash before deleting the backup file, permanently stranding it and breaking every future `lite down`. * docs(cli): call out env-var auth for autoroute commands * fix(cli): clean up leaked proxy and surface clean errors in autoroute Three related gaps, all following an UpError getting raised somewhere that wasn't catching it yet: - up() left the just-launched ephemeral proxy running with no pid record if load_json_or_empty/write_backup/secure_create raised after the health check passed, mirroring the existing ProcessLaunchError cleanup for the health-check-failure branch. - _teardown() didn't catch restore_claude_settings raising UpError (e.g. a corrupt backup at stop time), which would otherwise escape to Click as an unhandled error in the normal-exit path, or print "Error in atexit" in the atexit path. up.py's own _restore_once handles the identical case the same way. - read_pid_record let a corrupt PID file surface a raw pydantic.ValidationError instead of a clean message, and did so in down(), the command specifically meant for crash recovery. down() now clears an unreadable pid record and continues cleanup instead of aborting, since a corrupt pid file must never block the one command meant to recover from exactly this kind of crash. * docs(cli): warn against running lite up and lite autoroute up together
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
This depends on #33231 (
litellm_lite_up_down), which is why it's based on that branch instead oflitellm_internal_stagingdirectly; it reuses that PR's settings-backup/restore helpers.Try it yourself
Install the CLI straight from this PR's commit, no PyPI release needed (swap in whatever this branch's latest commit is by the time you read this):
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/851b38fc4a7f80fd9b8c82daa6275a561a30fe84/scripts/install-cli.sh | \ LITELLM_CLI_REF=851b38fc4a7f80fd9b8c82daa6275a561a30fe84 shStart your real proxy (or point at one you already run):
Point the CLI at your proxy and key, and see what models it can reach:
Assign models to the four complexity tiers with the fuzzy picker (type to filter, tab to multi-select, enter to confirm):
Launch the ephemeral auto-router proxy. This also points Claude Code at it, including its Sonnet/Haiku/Opus defaults, so no
/modeljuggling is needed:In another terminal, use Claude Code as normal:
Send a trivial one-line question, then a long multi-step request, and watch the
lite autoroute upterminal: each one gets classified and routed to a different tier's model, forwarded to your real proxy, which shows the same calls in its own logs (http://localhost:4000/ui/?page=logs).Press Ctrl-C on
lite autoroute upwhen done; it restores your original Claude Code settings. If it ever dies uncleanly instead (kill -9, a crash), recover with:What I verified
Ran this against two live, unmodified litellm proxy processes with real Anthropic/OpenAI keys, no mocks: a request to the ephemeral proxy's
autoroutermodel got classified byauto_router/complexity_router, routed to the SIMPLE tier's model, forwarded over HTTP to the real proxy, which called Anthropic's real API and returned a genuine completion, all traceable in both proxies' logs. Also confirmedconfigure's generated config has the right deployments,uppatches and restores~/.claude/settings.jsoncorrectly on both SIGINT andkill -9, and a seconduprefuses to start while one is already running.That pass caught two real bugs, both fixed here: the ephemeral proxy's freshly-minted key was written to
litellm_settings.master_key, which the proxy server never reads, so it had no real authentication; and model discovery crashed on a real proxy's/model_group/inforesponse whenever a model had an explicit"mode": null, which is common for embedding models registered without a mode.Type
🆕 New Feature
✅ Test
📖 Documentation
Changes
Adds
lite model-groups list, a thin wrapper overModelGroupsManagementClient.info()(/model_group/info), andlite autoroute configure/up/downunderlitellm/proxy/client/cli/commands/autoroute/.configurediscovers the models a key can access, walks through assigning them to SIMPLE/MEDIUM/COMPLEX/REASONING tiers via a fuzzy type-to-filter picker, with optional LLM-classifier, semantic-matching, and adaptive selection, and writes a generated config.yaml where every referenced model is its ownlitellm_proxy/<name>deployment forwarding back to the real proxy with the real key.uplaunches that config as an ephemeral local proxy with a freshly minted key, patches Claude Code's settings (including its default Sonnet/Haiku/Opus models, since Router resolves auto-router deployments by literal model name with no wildcard support) to point at it, streams its routing-decision logs live, and restores everything on Ctrl-C/SIGTERM;downis the manual recovery path after an unclean exit.up.py'swrite_backup/read_backup/restore_claude_settingsare generalized to take explicit paths (defaulting to the existing constants, solite up/lite downare unaffected) so this feature reuses them instead of duplicating the backup/restore logic.scripts/install-cli.shgains an opt-inLITELLM_CLI_REFoverride so a branch (or tag, or commit) can be installed straight from source instead of the latest PyPI release; unset, it installslitellm[cli]from PyPI exactly as before.QA runbook
Final Attestation