feat(tracing): correlate spend logs with agent traces by yujonglee-berri · Pull Request #43865 · BerriAI/litellm

yujonglee-berri · 2026-09-30T14:43:57Z

TLDR

Problem this solves:

Agent traces lack request costs and request-log links
Python owned enriched ClickHouse SQL outside the Rust storage boundary

How it solves it:

Write spend rows through the Rust ClickHouse transport
Keep named, tenant-scoped read queries with the Rust schema
Show costs and request details in the Logs page
Reject unknown insert columns and bound insert size and time

Intentional product change: Trace rows show cost and request details when a spend row matches. Missing spend remains unavailable rather than appearing as $0

User Flow

Before: an operator can inspect an agent trace, but cannot see the LiteLLM request or cost behind its LLM spans

They enable agent tracing and open http://localhost:4000/ui/?page=logs
Their agent calls the proxy and exports spans to POST http://localhost:4000/v1/traces
They open the trace drawer and see spans and token counts without request costs or links

After: the same trace shows the matching request and cost

They enable agent tracing and the clickhouse callback, then open http://localhost:4000/ui/?page=logs
Their agent calls the proxy and exports spans to POST http://localhost:4000/v1/traces
They open the trace drawer and see costs, request details, and a link to the request log

Pre-Submission checklist

I have added meaningful tests
Focused Python, native bridge, ClickHouse, and UI tests passed locally
My PR passes all required CI/CD checks
My PR's scope is as isolated as possible
I have received a Greptile Confidence Score of at least 4/5

Type

New Feature

Caveats (if any)

Medium

Cost enrichment requires the clickhouse callback and a matching response id

Low

Live proxy proof, UI screenshots, and reviews have not been rerun at this tip

Final Attestation

The tests check the right things, including edge cases and real-world regressions

codecov · 2026-09-30T14:51:28Z

Codecov Report

❌ Patch coverage is 92.77108% with 18 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...integrations/clickhouse/clickhouse_spend_logger.py	84.33%	13 Missing ⚠️
litellm/rust_bridge/traces.py	60.00%	4 Missing ⚠️
litellm/tracing/store.py	90.90%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

codspeed · 2026-09-30T16:23:22Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_trace_spend_enrichment (0841d91) with litellm_agent_tracing_integration (b7ab950)}

devin-ai-integration

Devin Review found 7 potential issues.

3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-09-30T18:39:37Z

+litellm_settings:
+  success_callback:
+    - clickhouse


🔴 Failed requests lose spend rows

When configured through success_callback, ClickHouseSpendLogger never receives failed requests. Its failure handler runs only for callbacks registered for failures, leaving failed calls absent from spend logs

Suggested change

litellm_settings:

success_callback:

- clickhouse

litellm_settings:

callbacks:

- clickhouse

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:39:39Z

trace_from_rows

+           sum(SpanCount) AS SpanCount, sum(AgentCount) AS AgentCount, sum(LlmCount) AS LlmCount,
+           sum(ToolCount) AS ToolCount, sum(ErrorCount) AS ErrorCount, sum(InputTokens) AS InputTokens,
+           sum(OutputTokens) AS OutputTokens, groupUniqArrayArray(Models) AS Models,
+           groupUniqArrayArray(AgentNames) AS AgentNames, groupArrayArray(RequestIds) AS RequestIds


🔴 Repeated span IDs inflate trace cost

When multiple spans carry one LiteLLMRequestId, RequestIds repeats that ID. The array join sums its spend repeatedly, so the trace list overstates cost

Learn more

The agent trace materialized view stores one request ID per span, including repeated IDs across partial rows. The list query expands this array into one row per ID before summing joined spend. Duplicate IDs thus count the same request multiple times, even though trace_from_rows derives the detail cost separately from spans

Example: Two spans both carry response-1 and its spend log costs $0.05. The list reports $0.10 instead of $0.05

Recommended fix: Deduplicate request IDs within each trace before the array join, and add a ClickHouse integration test with repeated span response IDs

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:39:40Z

TraceReceiver.start

+            _clickhouse_spend_logger: Final = ClickHouseSpendLogger()
+            _in_memory_loggers.append(_clickhouse_spend_logger)


🔴 Spend-only setup loses every batch

When clickhouse runs without tracing enabled, ClickHouseSpendLogger never creates its spend_logs table. Fresh installations reject every batch and eventually drop all spend rows

Learn more

The callback constructs a ClickHouse storage client but never initializes its table. TraceReceiver.start is the only production caller of schema initialization, and startup skips it unless general_settings.tracing.store is clickhouse. ClickHouseBatchLogger._insert retries failed inserts and then drops the batch

Example: Enable only litellm_settings.callbacks: [clickhouse] with a new ClickHouse database. The first insert fails because spend_logs does not exist, and three attempts later its rows disappear

Recommended fix: Initialize the spend schema when starting the callback independently of trace ingestion, or reject a callback-only configuration explicitly until its table is ready

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:39:41Z

ClickHouseBatchLogger.init

+            _clickhouse_spend_logger: Final = ClickHouseSpendLogger()
+            _in_memory_loggers.append(_clickhouse_spend_logger)


🔴 Synchronous setup leaves spend queue idle

When ClickHouseSpendLogger initializes outside an event loop, its periodic flush never starts. Calls below the batch threshold remain queued indefinitely and vanish on exit

Learn more

ClickHouseBatchLogger.init suppresses the missing-loop error without scheduling a later start. Its start method exists but no production caller invokes it, and enqueue triggers a flush only at the batch threshold

Example: An SDK process configures the callback synchronously and makes 10 requests in an event loop. With the default 10,000-row batch, those rows never reach ClickHouse

Recommended fix: Schedule the periodic flush on the first async event or explicitly start the callback when the application loop becomes available, with idempotent task lifecycle handling

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:39:43Z

NativeTraceStorage

@@ -70,7 +70,6 @@ def from_env(cls) -> "TraceReceiver":
                TraceStorage(
                    database=os.getenv("CLICKHOUSE_DATABASE", "litellm"),
                    url=os.environ["CLICKHOUSE_URL"],


🟡 Configured trace reader is ignored

When CLICKHOUSE_READER_URL points to a separate reader, TraceReceiver uses CLICKHOUSE_URL for every read instead. Read traffic shifts to the writer, bypassing the configured reader

Learn more

Previously NativeTraceStorage accepted a separate reader URL, while this call now passes only the writer. Its constructor creates a reader connection using that same writer URL. Existing installations with CLICKHOUSE_READER_URL no longer direct traffic to their reader

Example: Set CLICKHOUSE_URL to the primary and CLICKHOUSE_READER_URL to a replica. All trace list and detail requests now query the primary

Recommended fix: Keep an optional separate reader endpoint in the native constructor and use the reader environment variable when present, falling back to the writer URL only if absent

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:39:44Z

+      AND (empty({team_ids:Array(String)}) OR team_id IN {team_ids:Array(String)}) AND ({api_key_hash:String} = '' OR api_key = {api_key_hash:String})
+) AS s ON s.response_id = o.LiteLLMRequestId


🟥 Shared response IDs leak cross-team spend

When two teams share a response ID, trace_spans joins spend without matching s.team_id to o.TeamId. A team member can see another team's request details and cost

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:39:45Z

+      LITELLM_MASTER_KEY: test
+      LITELLM_SALT_KEY: sk-local-tracing-salt-key
+      DATABASE_URL: postgresql://litellm:litellm@db:5432/litellm
+      STORE_MODEL_IN_DB: "True"
+      CLICKHOUSE_URL: http://default:local-tracing-write@clickhouse:8123
+      CLICKHOUSE_DATABASE: default


🟨 Local tracing stack uses fixed credentials

The compose stack starts with fixed database credentials and a short proxy master key. Running it outside a local test environment exposes privileged access with known credentials

Was this helpful? React with 👍 or 👎 to provide feedback.

greptile-apps · 2026-09-30T18:44:50Z

Confidence Score: 2/5

[High risk] Adds tracing query execution and ClickHouse spend logging.

The PR should not merge until spend logging, trace-read compatibility, and cache-hit cost attribution are corrected

Findings

Spend rows lack schema setup ▶
Separate trace readers stop working ▶
Cache hits distort trace costs ▶
Unmatched request IDs disappear ▶
Banned type suppression added ▶
Setup example mixes formats ▶
Spend joins lack database tests ▶

Summary

The PR adds ClickHouse spend logging, moves trace reads into named Rust-owned SQL queries, and displays request costs and details in the Logs UI

Spend-row initialization and trace-read credential compatibility need attention
Cache-hit response IDs can produce incorrect trace cost totals

_{Reviews (1) · Last reviewed commit: 8d323d9}

greptile-apps · 2026-09-30T18:44:53Z

+class ClickHouseSpendLogger(ClickHouseBatchLogger):
+    table = SPEND_LOGS_TABLE


Spend rows lack schema setup If the clickhouse callback is enabled without agent tracing, nothing creates spend_logs. Inserts fail, retry, and eventually drop the spend rows.

greptile-apps · 2026-09-30T18:44:53Z

-                .map(|value| Connection::reader(value, &database))
-                .transpose()
-                .map_err(map_error)?,
+            reader: Connection::reader(url, &database).map_err(map_error)?,


Separate trace readers stop working If a deployment uses CLICKHOUSE_READER_URL for read credentials or a replica, this now ignores it and sends reads to the writer. A writer without SELECT permission cannot serve trace reads. The repository requires a user-controlled flag for this backwards-incompatible change.

Rule Used: What: avoid backwards-incompatible changes without user-controlled flags Why: This breaks current behaviour for users using existing functionality Example of BAD: this PR (#22164) introduced run_post_custom... (source)

greptile-apps · 2026-09-30T18:44:53Z

+) AS t
+LEFT ARRAY JOIN t.RequestIds AS request_id
+LEFT JOIN (
+    SELECT response_id, any(spend) AS spend FROM spend_logs FINAL WHERE (empty({team_ids:Array(String)}) OR team_id IN {team_ids:Array(String)}) AND ({api_key_hash:String} = '' OR api_key = {api_key_hash:String}) GROUP BY response_id


Cache hits distort trace costs Cache hits can produce distinct spend rows with the same stripped response_id. Grouping them with any(spend) picks an arbitrary cost, so matching trace requests can show incorrect totals.

greptile-apps · 2026-09-30T18:44:54Z

        input_tokens=int(row["input_tokens"]),
        output_tokens=int(row["output_tokens"]),
-        litellm_request_id=row["litellm_request_id"] or None,
+        litellm=_litellm_request(row),


Unmatched request IDs disappear A span with a LiteLLM request ID loses that ID from the response when no spend row matches. This makes the request harder to identify while enrichment is delayed or unavailable.

greptile-apps · 2026-09-30T18:44:54Z

+
+
+def test_success_row_mapping():
+    row = spend_log_row_from_payload(_payload(), {})  # type: ignore[arg-type]


Banned type suppression added This test uses # type: ignore, which the repository guide bans because pyright does not honor it. The changed trace-store tests also use it. This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

greptile-apps · 2026-09-30T18:44:54Z

  tracing:
-    store: clickhouse   # + CLICKHOUSE_URL env var
+    store: clickhouse
+export CLICKHOUSE_URL=<ClickHouse URL>


Setup example mixes formats The setup card puts a shell export inside the proxy YAML example. Copying the block as config produces invalid YAML; separate the environment command from the config.

greptile-apps · 2026-09-30T18:44:54Z

+read_queries! {
+    ListTraces => "list_traces" => "list_traces.sql",
+    TraceSpans => "trace_spans" => "trace_spans.sql",
+    SpanDetail => "span_detail" => "span_detail.sql",


Spend joins lack database tests The database-backed named-read test runs only SpanDetail, not the new list or span spend joins. A query-planning or join regression could therefore reach deployment without a failing test.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

yujonglee-berri added this pull request to stack #43866 September 30, 2026 14:44

yujonglee-berri mentioned this pull request Sep 30, 2026

feat(tracing): add ingestion and UI on Rust foundation #43864

Closed

3 of 6 tasks

yujonglee-berri force-pushed the litellm_agent_tracing_integration branch from 0abf1e4 to c7482fd Compare September 30, 2026 16:56

yujonglee-berri force-pushed the litellm_trace_spend_enrichment branch 3 times, most recently from b0393e3 to 263c404 Compare September 30, 2026 17:20

yujonglee-berri had a problem deploying to e2e-changed September 30, 2026 18:13 — with GitHub Actions Error

yujonglee-berri had a problem deploying to e2e-changed September 30, 2026 18:34 — with GitHub Actions Error

yujonglee-berri marked this pull request as ready for review September 30, 2026 18:35

yujonglee-berri requested a review from a team September 30, 2026 18:35

ishaan-berri approved these changes Sep 30, 2026

View reviewed changes

moe-berri mentioned this pull request Sep 30, 2026

feat(lens): analyze agent activity with a separate worker #43889

Merged

4 of 6 tasks

yujonglee-berri force-pushed the litellm_trace_spend_enrichment branch from 8d323d9 to 5e4f8be Compare September 30, 2026 18:37

yujonglee-berri had a problem deploying to e2e-changed September 30, 2026 18:37 — with GitHub Actions Error

devin-ai-integration Bot reviewed Sep 30, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

yujonglee-berri and others added 4 commits September 30, 2026 12:00

feat(tracing): add tenant scoped agent tracing integration

a3455fd

fix(tracing): include new tests in CI and meet UI lint budgets

473faec

style(tracing): format Rust bridge facade

7b04935

fix(tracing): restore Python 3.10 compatibility

b7ab950

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

yujonglee-berri force-pushed the litellm_agent_tracing_integration branch from df529d4 to b7ab950 Compare September 30, 2026 19:00

yujonglee-berri added 6 commits September 30, 2026 19:01

wip

2f0bcc7

feat(traces): establish shared Rust storage foundation

b88ab3e

feat(tracing): enrich agent traces with spend logs

3e74da0

test(tracing): verify strict inserts and named native reads

338702e

fix(tracing): preserve UI lint headroom after spend enrichment

1fd9e84

feat: add local tracing stack and shared query registry

f099316

yujonglee-berri and others added 2 commits September 30, 2026 19:02

refactor(tracing): use one ClickHouse URL for reads and writes

914450a

fix(tracing): satisfy stack checks

0841d91

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot force-pushed the litellm_trace_spend_enrichment branch from 55edb12 to 0841d91 Compare September 30, 2026 19:03

devin-ai-integration Bot requested a deployment to e2e-changed September 30, 2026 19:03 Waiting

devin-ai-integration Bot force-pushed the litellm_agent_tracing_integration branch from b7ab950 to 4a5f738 Compare September 30, 2026 19:45

devin-ai-integration Bot removed this pull request from stack #43866 September 30, 2026 19:45

yujonglee-berri added this pull request to stack #43912 September 30, 2026 19:49

moe-berri removed this pull request from stack #43912 September 30, 2026 21:25

yujonglee-berri closed this Sep 30, 2026

yujonglee-berri deleted the litellm_trace_spend_enrichment branch September 30, 2026 21:37

		_clickhouse_spend_logger: Final = ClickHouseSpendLogger()
		_in_memory_loggers.append(_clickhouse_spend_logger)

		AND (empty({team_ids:Array(String)}) OR team_id IN {team_ids:Array(String)}) AND ({api_key_hash:String} = '' OR api_key = {api_key_hash:String})
		) AS s ON s.response_id = o.LiteLLMRequestId

		class ClickHouseSpendLogger(ClickHouseBatchLogger):
		table = SPEND_LOGS_TABLE



		def test_success_row_mapping():
		row = spend_log_row_from_payload(_payload(), {}) # type: ignore[arg-type]

Conversation

yujonglee-berri commented Sep 30, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Pre-Submission checklist

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

codecov Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 2/5

Findings

Summary

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

This branch is waiting to be deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yujonglee-berri commented Sep 30, 2026 •

edited by devin-ai-integration Bot

Loading

codecov Bot commented Sep 30, 2026 •

edited

Loading

codspeed Bot commented Sep 30, 2026 •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading