feat(lens): show end-user feedback on traces, stored in ClickHouse by ishaan-berri · Pull Request #45171 · BerriAI/litellm

ishaan-berri · 2026-10-07T21:44:00Z

TLDR

Problem this solves:

Apps collect end-user ratings on agent runs but Lens has nowhere to store them
A developer opening a run cannot see what the user thought of it

How it solves it:

New typed /lens/feedback API: an app posts its end user's 0 to 10 score and comment
Stored in a new ClickHouse lens_feedback table next to the traces
Trace list shows each run's score, and a low score tints the whole row red
Opening a run shows the user feedback first, above the steps

User Flow

Before: an app's users rate the agent's answers, but the developer debugging in Lens never sees those ratings

The app's user rates a run 2/10 and writes "it ran the wrong command first", and the app has nowhere in LiteLLM to send it
The developer opens http://localhost:4000/ui/?page=lens, Traces tab, and sees no feedback on any run
They open the run and see only its steps, with no idea a user complained

After: the rating flows into Lens and is the first thing the developer sees on that run

The app sends PUT /lens/feedback with its own key, the trace_id (or Claude Code session_id), the end user, score 2 and the comment
The developer opens http://localhost:4000/ui/?page=lens, Traces tab: the run shows a red ★ 2/10 badge and its whole row is tinted red
They open the run: right under the header a "User feedback" card shows 2/10, the user's quote, who said it and when
A run several users rated shows each one plus the average, on a neutral card when nobody scored it low

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Setup: local proxy on :4000 with tracing: on, ClickHouse on :18123, Postgres on :15432, and real Lens traces already ingested for team 9319c223…. $APP_KEY is an ordinary virtual key minted for that team with POST /key/generate, standing in for the customer's app. The proxy creates lens_feedback itself at boot

Before (`46d440a`)

API

There is no feedback route, so /lens/feedback only matches the existing /lens/{lens_id} investigation routes and an app has nowhere to send a rating

UI

See the Before screenshot under User Flow

After (`18fd204`)

API

The app posts its end user's low rating on its own team's trace:
curl -s -X PUT http://localhost:4000/lens/feedback -H "Authorization: Bearer $APP_KEY" -H 'Content-Type: application/json' -d '{"trace_id":"a2b2ab7f16eaaec662eebdea0a2b63be","user":"customer-1042","score":2,"comment":"I asked for the sum of 1 to 100 and it ran the wrong command first. Took two tries."}'
```
{"trace_id":"a2b2ab7f16eaaec662eebdea0a2b63be","trace_ref":"482C5101…","score":2,"comment":"I asked for the sum of 1 to 100 and it ran the wrong command first. Took two tries.","author":"customer-1042","created_at":"2026-10-07T21:57:01.414000Z","updated_at":"2026-10-07T21:57:01.414000Z"}
```
HTTP 200
Two more users rate a different run (9 "Quick and correct." from customer-2210, 7 "Right answer, a bit slow." from customer-3077): both HTTP 200
The same app key tries a trace owned by another tenant:
curl -s -X PUT http://localhost:4000/lens/feedback -H "Authorization: Bearer $APP_KEY" ... -d '{"trace_id":"94816cc030f363944412a825c8c991b7","user":"customer-1042","score":5,"comment":"other tenant trace"}'
```
{"detail":"Trace not found"}
```
HTTP 404
A score of 11 returns 422

ClickHouse table

SHOW CREATE TABLE litellm.lens_feedback after proxy boot

ENGINE = ReplacingMergeTree(UpdatedAt, IsDeleted)
ORDER BY (TeamId, ApiKeyHash, TraceId, Author)
TTL toDateTime(CreatedAt) + toIntervalDay(14)
-- INDEX idx_trace_id TraceId TYPE bloom_filter(0.001), CONSTRAINT score_range CHECK Score <= 10

UI

See the After screenshots under User Flow, taken against the same proxy and data

Type

🆕 New Feature

Caveats (if any)

Medium

Any key can write feedback, scoped to traces its team (or, without a team, the key itself) sent
- Reading it in Lens stays proxy admin and admin viewer only
user is whatever the app sends; LiteLLM does not verify end-user identity
Feedback attaches to a trace (trace_id + trace_ref), Lens's unit for a session
- A Claude Code session_id maps to the same id ingest derives, pinned by matching Rust and Python tests

Low

"Low" means any score of 4 or below; it is one constant in the UI
Feedback expires with traces, using the same retention setting
Shape mirrors Langfuse scores and LangSmith feedback, but nothing exports to them yet

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

ReplacingMergeTree(UpdatedAt, IsDeleted) keyed like agent_traces_by_key so feedback joins traces in sort order, with a bloom filter on TraceId, a score CHECK, and the same retention as traces Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feedback_target resolves a visible trace's team, key and trace_ref before a write; feedback lists the latest live entry per author; feedback_summary returns count, average and lowest score for a batch of traces Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

PUT saves or replaces the caller's score and comment, GET lists every entry, DELETE removes the caller's, POST /summary feeds the trace list. Callers can target a trace_id or a Claude Code session_id Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A Feedback column shows each run's average score and rating count, marked red when any score is 4 or below, and a filter narrows to rated or low-score runs through the URL Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CLAassistant · 2026-10-07T21:44:07Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

greptile-apps · 2026-10-07T21:52:00Z

Confidence Score: 4/5

[Critical risk] Adds database schema and API endpoints for storing user feedback on traces.

The repository’s comment and comprehension requirements must be satisfied before merging; the remaining feedback issues are non-blocking

Findings

Read-only users get save controls ▶
Failed feedback looks empty ▶
Save and remove can overlap ▶
Feedback outlives its trace ▶
Routine comments break repository rules ▶
Comprehensions exceed the allowed limit ▶

Summary

Adds per-person 0–10 scores and comments to Lens traces, stored in ClickHouse. The dashboard adds summaries, feedback filters, and an edit panel

Read-only users are offered edits that always fail
Failed summary requests appear as empty filtered results
Save and remove can overlap, and feedback can outlive its trace
The new code needs to satisfy the repository’s comment and comprehension requirements

_{Reviews (1) · Last reviewed commit: "feat(lens-ui): show and edit human feedb..." · Reviewed by Greptile}

greptile-apps · 2026-10-07T21:52:04Z

+            <FeedbackForm
+              key={view.mine?.updated_at ?? "new"}
+              mine={view.mine}
+              saving={save.isPending}
+              removing={remove.isPending}
+              onSave={(score, comment) => save.mutate({ score, comment })}
+              onRemove={() => remove.mutate()}


Read-only users get save controls

Admin viewers and demo users get an editable FeedbackForm, but every save is rejected. They can enter a score and comment only to get an error. Pass the read-only state into the panel and hide the form for these users

greptile-apps · 2026-10-07T21:52:05Z

+
+export function matchesFeedback(state: TraceFeedbackState | undefined, filter: FeedbackFilter): boolean {
+  if (filter === "all") return true;
+  if (state?.status !== "ready" || state.summary.count === 0) return false;


Failed feedback looks empty

Under “Has feedback” or “Low score,” a failed summary request hides every affected run as though none matched. The table only receives errors from the trace request, so the reviewer sees an empty result instead of the feedback error. Show the failed lookup and a retry action

greptile-apps · 2026-10-07T21:52:05Z

+          <Button type="button" variant="ghost" size="xs" disabled={removing} onClick={onRemove}>
+            Remove
+          </Button>
+        )}
+        <Button type="submit" size="xs" disabled={score === null || saving}>


Save and remove can overlap

Remove stays enabled during an update, and Update stays enabled during removal. A reviewer can send conflicting changes to the same entry, leaving a result that differs from the last button they clicked. Disable both buttons while either request is pending

greptile-apps · 2026-10-07T21:52:05Z

    ("otel_traces", "toDateTime(Timestamp)"),
    ("agent_traces_by_key", "toDateTime(StartTs)"),
    ("spend_logs", "toDateTime(start_time)"),
+    ("lens_feedback", "toDateTime(CreatedAt)"),


Feedback outlives its trace

Feedback expires from CreatedAt, while traces expire from StartTs. A comment left near a trace’s expiry can remain stored for almost another retention period. Once the trace disappears, the API cannot find the target to remove that comment. Align their expiry times

greptile-apps · 2026-10-07T21:52:05Z

+  );
+}
+
+/** Human 0–10 scores and comments on one run, plus the caller's own editable entry. */


Routine comments break repository rules

This new comment repeats the function’s purpose. AGENTS.md prohibits routine source comments unless they meet a listed exception. Remove it before merging. The same issue appears in feedbackView and stubPost

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

greptile-apps · 2026-10-07T21:52:05Z

+                **access_parameters(scope).model_dump(), trace_ids=sorted({t.trace_id for t in traces})
+            ),
+        )
+        return tuple(summary for trace in traces for summary in _summaries(trace, rows))


Comprehensions exceed the allowed limit

This comprehension uses two for clauses. AGENTS.md allows only one per comprehension, so split it into a helper before merging. The fake summary query in test_feedback_endpoints.py repeats the same violation

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

codspeed · 2026-10-07T21:54:25Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_lens_session_feedback (79cd41e) with main (4f49f88)¹}

No successful run was found on main (e4d9a59) during the generation of this report, so 4f49f88 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

codecov · 2026-10-07T22:00:44Z

Codecov Report

❌ Patch coverage is 96.55172% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/lens/feedback_endpoints.py	90.54%	7 Missing ⚠️
litellm/proxy/lens/feedback_repository.py	98.64%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Any key can now PUT/DELETE feedback on traces its team or key sent, naming the end user in an optional user field. Reading feedback stays with Lens admins Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…red rows Replace the edit popover and feedback filter with a read-only view: the list shows each run's score in a fixed-width badge and tints the row red when a user scored it 4 or below, and opening a run shows what users said with their score right under the header Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

veria-ai · 2026-10-07T22:14:08Z

+        return Scope(all_teams=True)
+    if auth.user_role == LitellmUserRoles.PROXY_ADMIN_VIEW_ONLY:
+        raise HTTPException(403, "Admin viewers cannot write feedback")
+    return Scope(team_id=auth.team_id or "", api_key_hash="" if auth.team_id else auth.token or "")


Medium: Wildcard feedback access for teamless JWTs

JWTAuthManager.user_api_key_auth_from_result() leaves auth.token unset, so a caller without a resolved team receives Scope(team_id="", api_key_hash=""); lens_feedback_target.sql treats the empty key hash as a wildcard. A non-admin JWT user can therefore overwrite or delete another key's feedback by supplying a known teamless trace/session ID and the victim's user; reject callers lacking both a team and a key hash.

Suggested change

return Scope(team_id=auth.team_id or "", api_key_hash="" if auth.team_id else auth.token or "")

if not auth.team_id and not auth.token:

raise HTTPException(403, "Feedback requires a team or API key")

return Scope(team_id=auth.team_id or "", api_key_hash="" if auth.team_id else auth.token or "")

veria-ai · 2026-10-07T22:14:19Z

PR overview

Adds end-user feedback to trace views, backed by ClickHouse storage and endpoints for managing feedback.

One access-control issue remains: a non-admin JWT caller without a resolved team or API-key hash can overwrite or delete another key’s feedback by supplying a known teamless trace or session ID and the victim’s user value. The demonstrated impact is limited to unauthorized modification or removal of feedback; no issues have yet been addressed.

Open issues (1)

Medium: Wildcard feedback access for teamless JWTs — litellm/proxy/lens/feedback_endpoints.py:61

Fixed/addressed: 0 · PR risk: 6/10

… a list A proxy without the feedback route or an error body made the list crash on query.data.map; treat it as unavailable feedback instead Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The worker image copies lens/models.py without the litellm package, so the litellm.constants import broke it; move the proxy-only feedback models into feedback_models.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… failed one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…was recorded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…n_feedback # Conflicts: # litellm-rust/crates/traces-clickhouse/src/query/lens.rs # litellm/rust_bridge/trace/generated/types.py # ui/litellm-dashboard/src/components/lens/data/demo/createLensDemo.ts # ui/litellm-dashboard/src/components/lens/traces/api.ts # ui/litellm-dashboard/src/components/lens/traces/list/AgentTracesSection.integration.test.tsx # ui/litellm-dashboard/src/lib/http/schema.d.ts

…n_feedback # Conflicts: # ui/litellm-dashboard/src/components/lens/traces/list/AgentTracesSection.integration.test.tsx

ishaan-berri and others added 9 commits October 7, 2026 14:37

test(lens): pin the Claude session to trace id hash shared with feedback

edf4bdc

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(lens): expose feedback queries to the Python trace bridge

399386c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(lens): add typed feedback models and ClickHouse feedback store

5477825

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(lens-ui): add typed feedback calls to the traces API

1c9393d

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(lens-ui): show and edit human feedback inside a trace

357840f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ishaan-berri requested a review from a team October 7, 2026 21:44

ishaan-berri had a problem deploying to e2e-changed October 7, 2026 21:44 — with GitHub Actions Error

greptile-apps Bot reviewed Oct 7, 2026

View reviewed changes

ishaan-berri and others added 2 commits October 7, 2026 15:03

feat(lens): let app keys post end-user feedback on their own traces

0a65209

Any key can now PUT/DELETE feedback on traces its team or key sent, naming the end user in an optional user field. Reading feedback stays with Lens admins Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ishaan-berri had a problem deploying to e2e-changed October 7, 2026 22:04 — with GitHub Actions Error

ishaan-berri changed the title ~~feat(lens): human 0-10 feedback on traces, stored in ClickHouse~~ feat(lens): show end-user feedback on traces, stored in ClickHouse Oct 7, 2026

ishaan-berri enabled auto-merge (squash) October 7, 2026 22:08

veria-ai Bot reviewed Oct 7, 2026

View reviewed changes

ishaan-berri and others added 2 commits October 7, 2026 15:16

fix(lens-ui): keep the trace list up when the feedback summary is not…

25bc5e1

… a list A proxy without the feedback route or an error body made the list crash on query.data.map; treat it as unavailable feedback instead Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

style(lens-ui): prettier and keep the trace table at main's complexity

90253b5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ishaan-berri had a problem deploying to e2e-changed October 7, 2026 22:16 — with GitHub Actions Error

ishaan-berri had a problem deploying to e2e-changed October 7, 2026 22:22 — with GitHub Actions Error

ishaan-berri and others added 2 commits October 7, 2026 19:03

fix(lens-ui): open a trace on its top-level step instead of the first…

e3fc442

… failed one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(lens-ui): pin that failed runs open on the root step

9dba4e5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ishaan-berri and others added 4 commits October 7, 2026 19:03

refactor(lens-ui): drop the unused firstErrorSpan helper

7a27d53

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(lens-ui): stop testing the removed firstErrorSpan helper

ea9ccba

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(lens-ui): stop showing the model name as a run's input when none …

259e762

…was recorded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(lens-ui): cover the trace list row with no recorded input

2848bf7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ishaan-berri force-pushed the litellm_lens_session_feedback branch from 2848bf7 to 466f42b Compare October 8, 2026 02:21

ishaan-berri added 7 commits October 7, 2026 19:29

feat(lens): accept feedback rows on the lens service internal route

7b942ac

test(lens): cover feedback read and write through the lens service

8e663fa

fix(lens): route feedback writes to the remote lens service

e72965e

test(lens): cover remote feedback inserts

34e77eb

fix(lens): point feedback setup error at the lens service config

70d85e3

Merge remote-tracking branch 'upstream/main' into litellm_lens_sessio…

79cd41e

…n_feedback # Conflicts: # ui/litellm-dashboard/src/components/lens/traces/list/AgentTracesSection.integration.test.tsx

ishaan-berri requested a deployment to e2e-changed October 8, 2026 02:32 — with GitHub Actions Waiting

moe-berri approved these changes Oct 8, 2026

View reviewed changes

ishaan-berri merged commit 03ba79d into main Oct 8, 2026
121 of 128 checks passed

ishaan-berri deleted the litellm_lens_session_feedback branch October 8, 2026 03:14

ishaan-berri mentioned this pull request Oct 8, 2026

fix(lens): block teamless feedback writes and fix retention test #45266

Merged

3 of 4 tasks

devin-ai-integration Bot mentioned this pull request Oct 8, 2026

fix(ci): trim rust-test debuginfo so the job fits on the runner disk #45318

Merged

yucheng-berri mentioned this pull request Oct 8, 2026

feat(guardrails): per-mode stream_scope with bedrock stream and pass-through fixes #43801

Merged

4 of 5 tasks

martin-collado-dvt mentioned this pull request Oct 8, 2026

fix(ci): restore Rust retention and feature-matrix checks #45364

Open

6 tasks done

amaan-ai20 mentioned this pull request Oct 8, 2026

feat(providers): add ZeroGPU as an OpenAI-compatible provider #43874

Open

4 of 6 tasks

Navigation Menu

Repository navigation

feat(lens): show end-user feedback on traces, stored in ClickHouse - #45171

feat(lens): show end-user feedback on traces, stored in ClickHouse#45171
ishaan-berri merged 27 commits into
mainfrom
litellm_lens_session_feedback

ishaan-berri commented Oct 7, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Oct 7, 2026

Uh oh!

greptile-apps Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Oct 7, 2026

Uh oh!

greptile-apps Bot Oct 7, 2026

Uh oh!

greptile-apps Bot Oct 7, 2026

Uh oh!

greptile-apps Bot Oct 7, 2026

Uh oh!

greptile-apps Bot Oct 7, 2026

Uh oh!

greptile-apps Bot Oct 7, 2026

Uh oh!

codspeed Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

veria-ai Bot Oct 7, 2026

Uh oh!

veria-ai Bot commented Oct 7, 2026

Uh oh!

Uh oh!

This branch is waiting to be deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ishaan-berri commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (46d440a)

API

UI

After (18fd204)

API

ClickHouse table

UI

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

CLAassistant commented Oct 7, 2026

Uh oh!

greptile-apps Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

greptile-apps Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

codecov Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

veria-ai Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

veria-ai Bot commented Oct 7, 2026

PR overview

Open issues (1)

Uh oh!

Uh oh!

This branch is waiting to be deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ishaan-berri commented Oct 7, 2026 •

edited

Loading

Before (`46d440a`)

After (`18fd204`)

greptile-apps Bot commented Oct 7, 2026 •

edited

Loading

codspeed Bot commented Oct 7, 2026 •

edited

Loading

codecov Bot commented Oct 7, 2026 •

edited

Loading