feat(traces): add Rust storage foundation by yujonglee-berri · Pull Request #43819 · BerriAI/litellm

yujonglee-berri · 2026-09-30T06:36:02Z

TLDR

Problem this solves:

Agent tracing needs one ClickHouse schema and bounded storage path

How it solves it:

Keep schema, row encoding, and read transport in Rust
Use the current Python bridge execution facade from refactor(rust): centralize Python bridge execution wrappers #43871
Replace the crate README with concise boundary rules

Pre-Submission checklist

I have added meaningful tests
Focused Rust and bridge checks passed locally
My PR passes all required CI/CD checks
My PR's scope is as isolated as possible
I have received a Greptile Confidence Score of at least 4/5

Type

New Feature

Caveats (if any)

Medium

Existing incompatible ClickHouse tables require a manual migration

Low

This foundation has no direct user flow until feat(tracing): add ingestion and UI on Rust foundation #43864
Live proof and reviews have not been rerun at this tip

Final Attestation

The tests check the right things, including edge cases and real-world regressions

codspeed · 2026-09-30T06:44:02Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_trace_query_api (b0518f2) with main (264b09a)}

codecov · 2026-09-30T06:48:38Z

Codecov Report

❌ Patch coverage is 0% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/rust_bridge/traces.py	0.00%	20 Missing ⚠️

📢 Thoughts on this report? Let us know!

devin-ai-integration

Devin Review found 7 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

devin-ai-integration · 2026-09-30T18:38:38Z

schema_statements

+                <query>GRANT SELECT ON default.otel_traces</query>
+                <query>GRANT SELECT ON default.agent_traces</query>
+                <query>GRANT SELECT ON default.spend_logs</query>


🔴 Reader cannot query custom trace databases

When ensure_schema creates a nondefault database, litellm_traces_reader retains grants only on default. Queries against the configured trace tables fail with access denied

Learn more

The schema creator accepts any valid database identifier in schema_statements. The reader profile grants SELECT on tables in default only. When trace_query selects another database through Connection::configured, SELECT reads tables on which that user has no grants.

Example: Creating tables in trace_test succeeds. Querying trace_test.otel_traces as litellm_traces_reader fails with access denied.

Recommended fix: Match reader grants to the configured schema database during deployment or provisioning, and test a nondefault database with the restricted reader.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:38:39Z

agent_traces_mv

+PARTITION BY toDate(StartTs)
+ORDER BY (TeamId, TraceId)


🔴 Trace rollups split across calendar days

When one trace spans midnight, StartTs puts its batches into different partitions. AggregatingMergeTree cannot merge the same (TeamId, TraceId) across partitions, leaving multiple incomplete rollup rows

Learn more

The materialized view agent_traces_mv emits one aggregate row per trace and insert batch, with StartTs equal to the minimum span timestamp in that batch. AggregatingMergeTree only merges rows with the same key within one partition. A trace with batches on different dates has two distinct aggregate rows even after merges. A read that expects one row per trace receives split counts and incompatible root fields.

Example: Team A inserts the root at 23:59 and an LLM span at 00:01 the next day with the same TraceId. The agent and LLM counts remain in separate rows instead of forming one trace.

Recommended fix: Choose a partition expression stable for all batches of a trace, or explicitly aggregate across partitions on every read and ensure retention matches the intended whole-trace lifetime.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:38:40Z

+ENGINE = ReplacingMergeTree(end_time)
+PARTITION BY toYYYYMM(start_time)
+ORDER BY (team_id, toDateTime(start_time), request_id)


🔴 Same-second spend logs collapse into one

When distinct spend logs share a team, second, and request ID, ReplacingMergeTree treats them as one row. A later merge silently discards one spend entry

Learn more

ReplacingMergeTree identifies duplicates using the ORDER BY tuple, not the version column. Converting start_time to DateTime truncates the DateTime64(3) milliseconds. Distinct request logs that reuse an ID within a second therefore have the same replacing key, and ClickHouse retains only the highest end_time on merge.

Example: Two entries for request req-1 from the same team start at 12:00:00.100 and 12:00:00.800. They share the key (team, 12:00:00, req-1) and one spend record disappears.

Recommended fix: Preserve millisecond precision in the ordering key and determine whether request ID is unique across all spend-log events. Use a key that identifies actual retry duplicates rather than unrelated events.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:38:42Z

+        let response = client
+            .post(connection.url().clone())
+            .body(statement)
+            .send()
+            .await


🔴 Schema setup can wait indefinitely

If ClickHouse accepts a schema request but never responds, ensure_schema has no request timeout. Python's ensure_schema future remains pending indefinitely, blocking startup paths awaiting it

Suggested change

let response = client

.post(connection.url().clone())

.body(statement)

.send()

.await

let response = client

.post(connection.url().clone())

.timeout(std::time::Duration::from_secs(15))

.body(statement)

.send()

.await

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:38:43Z

RootName fields

+    anyLastIf(SpanName, ParentSpanId = '')                 AS RootName,
+    anyLastIf(InputPreview, ParentSpanId = '')             AS RootInput,
+    anyLastIf(StatusCode, ParentSpanId = '')               AS RootStatus,


🟡 Missing-root batches erase trace root fields

When later batches contain no root span, anyLastIf produces empty root values for their rollup rows. anyLast merges those rows with the root batch and can replace the root name, input, and status with empties

Learn more

The view groups spans from each incoming insert into one row per trace. For a batch with no root span, anyLastIf returns the default empty string. The destination's RootName fields use SimpleAggregateFunction(anyLast, String), which can retain the empty value during part merges rather than preserve the original root.

Example: Insert a root span named agent in batch one, then an LLM child alone in batch two. The second rollup supplies an empty RootName, and the merged trace can lose agent.

Recommended fix: Store a nullable or explicitly conditional root aggregate that ignores absent root spans when combining batches, and test separate root and child inserts followed by merges.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:38:45Z

+    if sql.trim().is_empty() {
+        return Err(Error::EmptySql);


🟥 Untrusted SQL bypasses reader-only intent

With a privileged ClickHouse account, execute_read accepts arbitrary SQL and relies on a request setting for read-only behavior. The account retains its underlying privileges, so a query that bypasses the setting can modify data.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-30T18:38:46Z

+        if !matches!(url.scheme(), "http" | "https") || url.host().is_none() {
+            return Err(Error::InvalidUrl);


🟨 ClickHouse credentials travel over plaintext HTTP

When Connection::configured or Connection::writer receives an http:// URL, requests send credentials without transport encryption. Anyone observing that network path can read them.

Was this helpful? React with 👍 or 👎 to provide feedback.

greptile-apps · 2026-09-30T18:41:23Z

Confidence Score: 1/5

[High risk] Adds database schema and storage layer for traces.

This PR is not ready to merge because trace root metadata can be lost and schema setup can silently retain the wrong policy or stall startup.

Findings

Root fields can disappear ▶
Retention changes are ignored ▶
Schema setup can stall startup ▶
Reader grants exclude default database ▶
Row encoding has no limit ▶

Summary

This PR adds the Rust ClickHouse trace schema, row encoder and bounded read transport, plus Python bridge exports and tests.

Partial rollups can lose root metadata.
Schema setup does not apply changed retention settings and can wait indefinitely on a stalled server.
The supplied reader grants do not cover the tracing caller's default database.
Row encoding remains unbounded.

_{Reviews (1) · Last reviewed commit: 535c3ad}

greptile-apps · 2026-09-30T18:41:30Z

+    RootName      SimpleAggregateFunction(anyLast, String),
+    RootInput     SimpleAggregateFunction(anyLast, String),
+    RootStatus    SimpleAggregateFunction(anyLast, String),


Root fields can disappear When a root span arrives in one batch and child spans arrive in a later batch, the later batch has empty root fields. Merging those batches with anyLast can overwrite the real values, leaving the trace list in PR #43864 with a blank name, input preview, or status.

greptile-apps · 2026-09-30T18:41:30Z

+    spend_log_retention_days: u32,
+) -> Result<(), Error> {
+    for statement in schema_statements(database, trace_retention_days, spend_log_retention_days)? {
+        let response = client


Retention changes are ignored If an operator changes either retention period after the tables exist, ensure_schema runs CREATE TABLE IF NOT EXISTS again but never updates their TTLs. Setup succeeds while trace and spend data continue to be retained or deleted under the old policy.

greptile-apps · 2026-09-30T18:41:30Z

+    for statement in schema_statements(database, trace_retention_days, spend_log_retention_days)? {
+        let response = client
+            .post(connection.url().clone())
+            .body(statement)
+            .send()


Schema setup can stall startup If ClickHouse accepts a connection but stalls while handling a schema statement, this POST has no total request timeout. The proxy startup path in PR #43864 awaits tracing setup, so startup can remain blocked indefinitely.

greptile-apps · 2026-09-30T18:41:30Z

+                <query>GRANT SELECT ON default.otel_traces</query>
+                <query>GRANT SELECT ON default.agent_traces</query>
+                <query>GRANT SELECT ON default.spend_logs</query>


Reader grants exclude default database If this reader profile is installed with the tracing caller's default configuration from PR #43864, the tables are created in litellm but these grants cover only default. Valid trace reads then fail with access denied.

greptile-apps · 2026-09-30T18:41:30Z

+pub fn encode_rows(rows: Vec<BTreeMap<String, Value>>) -> Result<String, Error> {
+    rows.into_iter()
+        .map(|row| {
+            let encoded = row
+                .into_iter()
+                .map(|(name, value)| insert_value(&name, value).map(|value| (name, value)))
+                .collect::<Result<BTreeMap<_, _>, _>>()?;
+            serde_json::to_string(&encoded).map_err(|_| Error::InvalidRow)
+        })
+        .collect::<Result<Vec<_>, _>>()
+        .map(|rows| rows.join("\n"))
+}


Row encoding has no limit This exported function collects every serialized row and then joins them into another string without checking the total byte count. A large valid batch can consume substantial additional memory before its caller can enforce a write limit; the bounded insert path added in PR #43864 does not change this function.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

yujonglee-berri had a problem deploying to e2e-changed September 30, 2026 06:36 — with GitHub Actions Error

yujonglee-berri changed the title ~~feat(traces): add Rust-backed trace query scaffolding~~ feat(traces): add Rust storage foundation Sep 30, 2026

yujonglee-berri mentioned this pull request Sep 30, 2026

feat: agent tracing - OTLP ingest, ClickHouse store, Logs trace view #43816

Merged

yujonglee-berri added this pull request to stack #43860 September 30, 2026 14:13

yujonglee-berri requested a deployment to e2e-changed September 30, 2026 14:29 — with GitHub Actions Waiting

yujonglee-berri force-pushed the litellm_trace_query_api branch from 2259b10 to ed2e507 Compare September 30, 2026 14:29

yujonglee-berri mentioned this pull request Sep 30, 2026

feat(tracing): add ingestion and UI on Rust foundation #43864

Closed

3 of 6 tasks

yujonglee-berri removed this pull request from stack #43860 September 30, 2026 14:44

yujonglee-berri added this pull request to stack #43866 September 30, 2026 14:44

yujonglee-berri force-pushed the litellm_trace_query_api branch from 13ff79d to 535c3ad Compare September 30, 2026 16:56

yujonglee-berri marked this pull request as ready for review September 30, 2026 18:35

yujonglee-berri requested a review from a team September 30, 2026 18:35

moe-berri mentioned this pull request Sep 30, 2026

feat(lens): analyze agent activity with a separate worker #43889

Merged

4 of 6 tasks

yujonglee-berri added 11 commits September 30, 2026 11:37

wip

e4b893a

feat(traces): establish shared Rust storage foundation

28e2407

fix(traces): escape ClickHouse text parameters

50dc095

test(traces): exercise response cap with bounded strings

835118f

fix(traces): remove unnecessary lint expectation

c120adb

fix(traces): encode ClickHouse timestamp units in Rust

350d06a

test(traces): mark exception match as a regex

1da1437

refactor(traces): execute schema setup in Rust

829e9a7

refactor(traces): use shared logging execution wrapper

1f1c985

docs(traces): replace foundation README with boundary rules

c3f3b14

fix(traces): use current bridge execution facade

1f3eb17

yujonglee-berri force-pushed the litellm_trace_query_api branch from 535c3ad to 1f3eb17 Compare September 30, 2026 18:37

devin-ai-integration Bot reviewed Sep 30, 2026

View reviewed changes

ishaan-berri approved these changes Sep 30, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

fix(traces): account for protocol cast in lint budget

b0518f2

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

ishaan-berri mentioned this pull request Sep 30, 2026

feat(ui): agent traces tab on logs with timeline and otel setup guide #43891

Merged

3 of 5 tasks

yujonglee-berri merged commit 41df8cf into main Sep 30, 2026
98 of 100 checks passed

yujonglee-berri deleted the litellm_trace_query_api branch September 30, 2026 19:00

devin-ai-integration Bot mentioned this pull request Sep 30, 2026

fix(traces): correct ClickHouse rollup partitioning, dedupe keys, and retention changes #43901

Merged

3 of 5 tasks

ishaan-berri mentioned this pull request Sep 30, 2026

fix(tracing): let virtual keys open traces and export them as markdown #43932

Open

3 of 5 tasks

		if !matches!(url.scheme(), "http" \| "https") \|\| url.host().is_none() {
		return Err(Error::InvalidUrl);

Conversation

yujonglee-berri commented Sep 30, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

Pre-Submission checklist

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

codspeed Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

codecov Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 1/5

Findings

Summary

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yujonglee-berri commented Sep 30, 2026 •

edited by devin-ai-integration Bot

Loading

codspeed Bot commented Sep 30, 2026 •

edited

Loading

codecov Bot commented Sep 30, 2026 •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading