feat(tracing): add scoped SQL queries and schema-aware help

[yujonglee-berri]

TLDR

Problem this solves:

• Trace SQL queries lack a public endpoint
• Team access needs enforcement beyond readonly query settings
• Custom metadata fields are hard to discover

How it solves it:

• Adds SQL queries and schema-aware help endpoints
• Maps authenticated teams and keys to managed ClickHouse readers
• Enforces scope through restrictive policies on all three tables
• Provisions readers automatically using the configured ClickHouse connection
• Keeps query execution, discovery, and guidance in Rust

Intentional product change: team users can run SQL and discover fields within their authenticated scope. Proxy admins and view-only admins can read all LiteLLM trace data. Project-bound keys remain restricted to their own key's rows until project-wide scope is implemented

The broader configuration work remains in #43941

User Flow

Before: a team developer cannot explore their traces through SQL

1. Export spans with a team key to POST http://localhost:4000/v1/traces
2. GET http://localhost:4000/v1/traces/query/help returns 404
3. POST http://localhost:4000/v1/traces/query with a SQL body returns 405

After: the same developer can explore their team's traces

1. Export spans with a team key to POST http://localhost:4000/v1/traces
2. GET http://localhost:4000/v1/traces/query/help returns schema, examples, and scoped field discovery
3. POST http://localhost:4000/v1/traces/query returns rows only within the key's authenticated scope
4. Explicitly querying another team's rows returns an empty result

Setup and enforcement

Set CLICKHOUSE_URL and, optionally, CLICKHOUSE_DATABASE. Trace startup no longer requires CLICKHOUSE_READER_URL. SQL queries and help use automatically provisioned readers on the configured writer connection's server. The configured account needs its existing schema/ingestion privileges, CREATE USER, ALTER USER, CREATE ROW POLICY on the three trace tables, and permission to grant SELECT on those tables

Each database and authorization scope gets a stable reader name, and the proxy's master key derives its password. Provisioning resets that password with ALTER USER, so rotating the master key revokes credentials derived from the previous one. Customers do not supply per-team credentials. ClickHouse stores password hashes. Provisioning creates locked settings and both permissive and restrictive row policies before granting table access. Failures reject the request and log setup guidance; execution never falls back to the writer identity

Team policies cover otel_traces, agent_traces_by_key, and spend_logs. Project-bound keys and teamless keys additionally require their authenticated key hash. Scope comes from proxy authentication, not the request body or SQL. Help uses the same reader, so its metadata discovery cannot sample another team's rows

Readers have only explicit SELECT grants and enforce readonly mode, 10 seconds, 1,000 rows, 4 MiB results, 256 MiB memory, two threads, and eight concurrent queries per user. Each storage instance also admits at most eight SQL/help requests. Reader provisioning is coalesced and cached with a 1,024-entry bound

Validation

• 17 Rust unit tests, 36 migration tests, and 7 live ClickHouse query-access tests pass
• 42 Python endpoint tests and 21 fresh-extension bridge tests pass
• All 60 new executable Python lines are covered locally
• Rust Clippy and make check pass without changing budgets
• Removing the team predicate makes the isolation regression fail
• Restoring the predicate makes the regression pass
• Removing the ALTER USER step makes the rotation regression fail

Live tests cover raw and qualified table names, CTEs, unions, joins, merge-table reads, summary rows, spend rows, help discovery, quoted team IDs, repeated provisioning, master-key rotation, resource settings, denied privileges, and provisioning failures. Discovery explicitly describes its bounded seven-day sample as unordered rather than promising the newest rows

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally
• My PR passes all required CI/CD checks
• My PR's scope is as isolated as possible
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

CI, Codecov, and bot reviews must be evaluated for e287f3495f6902faab11d774514b21b2474346ae. Earlier results do not approve this tip. No maintainer review is being requested while those checks are pending or unavailable

Latest-tip review evidence: Greptile re-review requested after the review fixes in e287f34. Required CI and Codecov results remain outstanding

Screenshots / Proof of Fix

Two local proxy processes used an isolated Postgres database and ClickHouse database named trace_scope_proof. Team keys were created through /team/new and /key/generate. Each exported one span through /v1/traces, including a forged resource team label that the proxy replaced from authentication. The after process had only CLICKHOUSE_URL, with CLICKHOUSE_READER_URL unset

The local curl files contain authorization and JSON content-type headers. Response excerpts below preserve data or detail; ClickHouse schema and timing fields are omitted for readability. These checks query stored spans and do not invoke an LLM

Before (c4f506a)

Team SQL

1. curl -sS -K /tmp/litellm-scope-proof/a.curl http://127.0.0.1:4104/v1/traces/query -d '{"sql":"SELECT TeamId, SpanName FROM otel_traces ORDER BY TeamId"}'
2. HTTP 405: {"detail":"Method Not Allowed"}

Explicit other-team read

1. curl -sS -K /tmp/litellm-scope-proof/a.curl http://127.0.0.1:4104/v1/traces/query --data-binary @/tmp/litellm-scope-proof/other-team-query.request.json, containing {"sql":"SELECT TeamId, SpanName FROM trace_scope_proof.otel_traces WHERE TeamId = 'scope-proof-b'"}
2. HTTP 405: {"detail":"Method Not Allowed"}

Admin SQL

1. curl -sS -K /tmp/litellm-scope-proof/admin.curl http://127.0.0.1:4104/v1/traces/query -d '{"sql":"SELECT TeamId, SpanName FROM otel_traces ORDER BY TeamId"}'
2. HTTP 405: {"detail":"Method Not Allowed"}

Scoped query help

1. curl -sS -K /tmp/litellm-scope-proof/a.curl http://127.0.0.1:4104/v1/traces/query/help
2. HTTP 404: {"detail":"Not Found"}

After (6d1cf91)

Team SQL

1. curl -sS -K /tmp/litellm-scope-proof/a.curl http://127.0.0.1:4105/v1/traces/query -d '{"sql":"SELECT TeamId, SpanName FROM otel_traces ORDER BY TeamId"}'
2. HTTP 200: {"data":[{"TeamId":"scope-proof-a","SpanName":"team-a-span"}]}

Explicit other-team read

1. curl -sS -K /tmp/litellm-scope-proof/a.curl http://127.0.0.1:4105/v1/traces/query --data-binary @/tmp/litellm-scope-proof/other-team-query.request.json, containing {"sql":"SELECT TeamId, SpanName FROM trace_scope_proof.otel_traces WHERE TeamId = 'scope-proof-b'"}
2. HTTP 200: {"data":[]}

Admin SQL

1. curl -sS -K /tmp/litellm-scope-proof/admin.curl http://127.0.0.1:4105/v1/traces/query -d '{"sql":"SELECT TeamId, SpanName FROM otel_traces ORDER BY TeamId"}'
2. HTTP 200: {"data":[{"TeamId":"scope-proof-a","SpanName":"team-a-span"},{"TeamId":"scope-proof-b","SpanName":"team-b-span"}]}

Scoped query help

1. curl -sS -K /tmp/litellm-scope-proof/a.curl http://127.0.0.1:4105/v1/traces/query/help
2. HTTP 200, including "access":"Authenticated team scope enforced by ClickHouse row policies; proxy admins can read all teams, while project-bound and teamless keys can read only their own rows"

Caveats

Severe

• SQL access now requires automatic reader-provisioning permissions

Medium

• Project-wide access is deferred; project keys see their own rows
• Readers for scopes that are no longer used persist until removed
• During a rolling master-key rotation, processes still on the old key fail trace queries
• Provisioning targets the configured server, without cluster-wide account orchestration

Low

• Metadata discovery samples unordered rows from the last seven days

Type

New Feature

Final Attestation

• Tests verify authenticated scope, denied bypasses, and provisioning failure behavior

Link to Devin session: https://app.devin.ai/sessions/6e2de59712e541e19e0537343bca6290
Open in Devin Desktop: https://app.devin.ai/desktop/session/6e2de59712e541e19e0537343bca6290?variant=devin
Requested by: @yujonglee-berri

[codecov]

Codecov Report

❌ Patch coverage is 96.66667% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/rust_bridge/traces.py	85.71%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yujonglee-berri]

@greptile-apps wdyt

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds SQL query execution with access control to the trace storage system.

The PR appears safe to merge based on this review.

Findings

1.  Recent metadata can be missed ▶
2.  Rotation disrupts active readers ▶
3.  Slow help requests occupy permits ▶

Summary

The PR adds authenticated, scope-restricted trace SQL and schema-aware help backed by managed ClickHouse readers. The latest tip updates Python bridge fixtures to account for the additional ALTER USER provisioning request.

• The revised request totals match the provisioning and read sequences.
• No new actionable issue was identified since the previous review.

_{Reviews (6) · Last reviewed commit: "test(tracing): update reader setup reque..."}

[yujonglee-berri]

@greptileai Please review the latest commit again, checking discovery failure recovery, current metadata sampling, SQL error responses, and resolved review threads

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_trace_sql_query (e287f34) with main (b9e71e9)}

[yujonglee-berri]

@greptileai Please review the latest tip for managed reader provisioning, authenticated team scope, privilege bypasses, and scoped metadata discovery

[devin-ai-integration]

@greptileai Please review the latest tip 8c2013f, which derives reader names from scope alone, resets reader passwords with ALTER USER so master-key rotation revokes old credentials, and removes local mutation from query help

[devin-ai-integration]

@greptileai Please review the latest tip 89394d5, which runs query help discovery sequentially again

[devin-ai-integration]

@greptileai Please review the latest tip e287f34, which updates the Python bridge test fixtures for the extra ALTER USER provisioning request