feat(lens): coordinate worker releases and bundled installs

[moe-berri]

TLDR

Problem this solves:

• Lens setup installs a stale worker image
• Gateway upgrades can break deployed workers

How it solves it:

• Select the worker image from the running LiteLLM release
• Reject incompatible workers before they take an investigation
• Install matching releases through Compose or Helm

Intentional product change: worker setup selects the matching release and exposes a private token for Compose and Helm. Existing Lens workers must upgrade before they can claim work

User Flow

Before: the setup command installs a pinned worker that can lag behind LiteLLM

1. Open http://localhost:4410/lens/?tab=investigations and choose Connect worker
2. Use Advanced options to select the existing analysis key, then Get install command
3. Copy a command containing the old fixed image digest and forced amd64 platform
4. Compose and Helm users cannot copy just the token from this screen

After: the command selects the matching release, with a token option for managed deployments

1. Open http://localhost:4410/lens/?tab=investigations and choose Connect worker
2. Use Advanced options to select the same analysis key, then Get install command
3. Copy a command containing the matching worker image for either supported architecture
4. Expand Using Docker Compose or Helm? and copy the private worker token

Pre-Submission checklist

• Added meaningful regression tests
• Targeted backend and UI tests pass locally
• All current-tip CI and required automated reviews pass
• Scope is coordinated worker installation and upgrades
• Fresh bot reviews for 61703efa3e; previous Greptile confidence was 5/5

Screenshots / Proof of Fix

Before (3defa90)

1. Load the gateway's release selector from a source checkout without LITELLM_RELEASE_TAG
2. It recommends ghcr.io/berriai/litellm-lens-worker:v1.105.0, using the installed package version even though the checkout can contain different source

After (61703ef)

1. Start the real gateway from this checkout with an empty LITELLM_RELEASE_TAG and a private admin key
2. GET http://127.0.0.1:4419/health/liveliness returns HTTP 200
3. POST http://127.0.0.1:4419/lens/workers/register with a valid registration body returns HTTP 503 and This LiteLLM build has no release identity. Use a published release, make lens-dev, or build the gateway and worker from the same commit with the same LITELLM_RELEASE_TAG. No worker credential is created

The latest source revision stops unlabelled containers, editable checkouts, and direct source installs from guessing a published worker version. Published package-index installs retain version-based selection. make lens-dev gives the proxy and worker the same source identity, overriding unrelated inherited release settings

The production Compose check now also recreates the gateway without its identity, verifies setup and claims are refused, then restores the original identity and checks that the worker and trace recover. The hosted production Compose check passes, including refusal and recovery on an unversioned gateway (run)

The paired dashboard screenshots in User Flow were captured at fd14e51ecf and cbd403778a using the same URL, registration, and real database. The frontend is unchanged between the captured revision and this tip. Clipboard checks verified the matching image and separately copied private token; no token appears in the screenshots

Validation

The earlier full Compose test at cbd403778a builds the production image and starts LiteLLM, PostgreSQL, ClickHouse, and the worker. It writes and reads a real trace, verifies the recommended image and authenticated heartbeat, rejects a wrong release with HTTP 409, restarts the services, and checks that the trace, worker identity, token, and billing assignment remain intact. The runtime, gateway, backend, UI, and migration image jobs pass, along with Buildkite end-to-end tests

On isolated Render services, upgraded the monolith and standalone worker from test release v0.0.1-lens-test to v0.0.2-lens-test using the same databases, keys, and worker token. A normal paid POST /v1/chat/completions returned HTTP 200, answer 4, and 22 tokens both before and after the upgrade. The existing dashboard session and saved findings survived

Against the component backend at cbd403778a, a real Helm-installed worker completed POST /lens/{id}/runs with {"lookback_hours":1}. GET /lens/{id} returned completed, two screened and investigated traces, two findings including brief, and cost $0.0480805. The original saved finding was retained. The worker used a non-root user, read-only root filesystem, memory-backed temporary storage, and a limited token from the existing Kubernetes Secret. The Secret survived the worker image upgrade and scale-to-zero/resume cycle

The upgraded standalone Render worker recovered an interrupted investigation after its five-minute lease expired, using the original worker token. The same job completed on its second attempt with two reviewed traces and cost $0.0747985. Separately, copied a newly registered worker token through the dashboard at cbd403778a and deployed it to Render. That investigation completed on its first attempt with two reviewed traces, two saved findings containing brief, and cost $0.0485835. Both runs retained the existing finding

An old protocol and wrong release each returned HTTP 409 without claiming a queued investigation or increasing its attempt count. Production worker imports, release metadata, and writable temporary storage pass on both amd64 and arm64. The full-storage recovery smoke passes

make check passes. Focused backend tests pass 84 cases, UI tests pass 19, Helm lint/render and 166 chart tests pass. The combined Lens suite passes 200 tests and the PostgreSQL lifecycle suite passes eight cases. Devin's merged budget fix #44431 is included through the main merge. The duplicate #44430 is closed

Revision 3defa90b09 also normalizes numeric Helm backend tags, rejects unversioned worker Docker builds, and isolates Compose storage with generated passwords. Compose URL-encodes those passwords before the original production entrypoint runs. Local make check and 166 chart tests pass. The hosted full-stack smoke passes at this tip using URL-reserved password characters and verifying that the worker cannot reach either datastore directly. All production image builds, the image vulnerability scan, and the hosted PostgreSQL migration suite pass

QA runbook

• tests/e2e/migrations/lens_compose_smoke.sh: real bundled startup and restart persistence
  • Build the production gateway image and set LITELLM_IMAGE to its local tag; Docker Compose, curl, jq, and OpenSSL are required
  • Start deploy/lens/stack.yaml with a private master key, salt key, version, and port; wait for GET /health/liveliness
  • Send a valid OTLP trace to POST /v1/traces; read the same trace and span IDs from GET /v1/traces/{trace_id}
  • Generate a limited analysis key, register a worker at POST /lens/workers/register, and verify the returned image matches the chosen release
  • Start the worker with its returned token; GET /lens must show a new authenticated heartbeat
  • Send a worker claim with worker_release=v0.0.0-old; expect HTTP 409 with upgrade guidance
  • From inside the worker, try PostgreSQL and ClickHouse by service name and IP; both connections must fail while authenticated proxy heartbeats continue
  • Restart LiteLLM and the worker with the same volumes and credentials; verify another heartbeat, unchanged billing assignment, and the saved trace
  • Sanity check: assertions verify returned IDs and persisted content, and cleanup removes only the test project's volumes

Companion changes

project-releaser #291 publishes matching GHCR and Docker Hub worker releases. litellm-ops #219 adds the worker to hourly selected-commit builds and dev chart rollouts. Documentation #2055 covers new installations, existing deployments, standalone hosting, and upgrades

Caveats

Severe

• Existing Lens workers require matching images before resuming
• Pause investigations and finish the LiteLLM rollout before resuming

Medium

• Verify registry permissions, then stage a fresh coordinated build
• Shared hourly dev rollout requires its repository and token Secret

Low

• CodeQL still exceeds its 2 GiB query limit
• At 61703efa3e, production Compose, image builds/scan, migrations, and all three bot reviews pass; broader Buildkite chore(deps): bump svgo from 3.3.2 to 3.3.3 in /docs/my-website #22840 passes in 33 minutes, 21 seconds

The same CodeQL LogInjection.ql result-size failure occurs on main and on this PR at 3defa90b09. This is an incomplete analysis, not a passing security check. The PR author accepts this existing tooling failure; scanner configuration is unchanged. At 3defa90b09, patch coverage is 94.87%, above the 81.11% target; Greptile gives 5/5, Veria passes, Bugbot reports no new issues, and Buildkite #22806 passes. The 61703efa3e follow-up passes local make check, 205 Lens cases, and 65 focused identity, endpoint, and development-script cases; its production Compose check, image builds/scan, migrations, and fresh reviews now pass (Greptile 5/5, Veria pass, Bugbot clean). Buildkite #22840 passes in 33 minutes, 21 seconds. No production registries or shared deployments were modified by this agent

Type

Bug fix, deployment support, documentation

---

Note

Medium Risk
Changes Lens worker claiming and registration (protocol 4, release matching) and deployment defaults, so existing workers must upgrade images; misconfigured builds without LITELLM_RELEASE_TAG will refuse setup until fixed.

Overview
Lens gateway and worker releases are now tied together so setup no longer pins a stale worker digest and upgrades cannot silently mix versions.

Gateway and worker images carry LITELLM_RELEASE_TAG (build-arg on main Dockerfiles and the Lens worker image, which fails the build if the tag is missing). litellm/proxy/lens/release.py derives the recommended worker image from that tag (with optional LENS_WORKER_IMAGE override). Worker registration returns the matching image; /lens/worker/claim bumps to protocol 4 and requires worker_release to match—otherwise HTTP 409 with upgrade text, or HTTP 503 when the gateway has no release identity (no guessing from the installed package version).

Installation paths: new deploy/lens/stack.yaml + config.yaml for a full Compose stack; worker-only Compose switches from digest pins to LITELLM_VERSION tags. Helm adds an opt-in lensWorker deployment and sets LENS_WORKER_IMAGE on the backend. The dashboard install command uses the API image, drops forced amd64, and adds Compose/Helm token copy. make lens-dev and CI smoke tests align proxy and worker on the same release tag.

^{Reviewed by Cursor Bugbot for commit 61703ef. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lens_release (61703ef) with main (5f96998)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[moe-berri]

@greptileai @veria-ai Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[moe-berri]

bugbot run

Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[greptile-apps]

Confidence Score: 5/5

[High risk] Coordinates worker and gateway releases through build-time versioning.

The PR appears safe to merge based on the reviewed changes and the resolved prior findings.

Summary

The PR coordinates Lens gateway and worker releases and adds matching installation paths.

• Worker registration recommends a release-matched image, while claims reject incompatible workers.
• Compose, Helm, and dashboard setup expose corresponding worker installation options.
• The latest changes refuse setup and claims when a gateway has no release identity and align source-development identities.

_{Reviews (6) · Last reviewed commit: "fix(lens): refuse worker recommendations..."}

[moe-berri]

@greptileai @veria-ai Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[moe-berri]

bugbot run

Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[moe-berri]

@greptileai @veria-ai Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[moe-berri]

bugbot run

Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

@greptileai @veria-ai Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[moe-berri]

bugbot run

Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

@greptileai @veria-ai Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[moe-berri]

bugbot run

Please review the current tip for release regressions, incompatible worker versions, unsafe upgrades, and missed failure paths

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

@greptileai @veria-ai Please review the latest build-identity guard, source development setup, and recovery checks for release regressions and missing failure paths.

[moe-berri]

bugbot run

Please review the latest build-identity guard, source development setup, and recovery checks for release regressions and missing failure paths.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 61703ef. Configure here.}