feat: export agent traces as markdown for claude and codex by ishaan-berri · Pull Request #43873 · BerriAI/litellm

ishaan-berri · 2026-09-30T16:08:09Z

TLDR

Problem this solves:

No simple way to hand an agent trace to Claude / Codex to debug it
Virtual keys got 401 on GET /v1/traces/{trace_id} (the templated route never matched)

How it solves it:

GET /v1/traces/{id}?format=md returns the run as Markdown: input, each step nested, output
&span_id= exports only that step and its children
Templated litellm_native_routes (e.g. /v1/traces/{trace_id}) now match like OpenAI routes do

Stacked on #43864.

User Flow

Before: a developer wants Claude to explain why their agent run failed, but has to piece the trace together by hand

They open the run and copy the steps from the UI one by one
curl -H "Authorization: Bearer sk-…" https://litellm-domain/v1/traces/<id> with their virtual key returns 401 Only proxy admin can be used…

After: the same developer pastes one line into Claude Code or Codex

curl -H "Authorization: Bearer sk-…" "https://litellm-domain/v1/traces/<id>?format=md" returns 200 with Markdown
The Markdown covers the input, every LLM call (model, tokens, LiteLLM request id), every tool call with its args and result, nested subagents, the first line of each error, and the output
Claude reads it and says the tool get_customer_plan failed, and the model call didn't

A key only gets traces in its own team scope. format=md uses the same scope_for() as the JSON read.

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Local proxy on this branch, ClickHouse 25.8, real LangChain agent traces (a tool raised ValueError). Requests used the research-bot virtual key, not the master key.

Before (litellm_agent_tracing_integration)

curl -H "Authorization: Bearer $KEY" localhost:4011/v1/traces/f78f6df3… returns 401 Authentication Error, Only proxy admin can be used to generate, delete, update info for new keys/users/teams. Route=/v1/traces/f78f6df3…

After (this branch)

curl -H "Authorization: Bearer $KEY" "localhost:4011/v1/traces/f78f6df3…?format=md" returns 200:

# Agent trace: support_triage_agent

- trace_id: `f78f6df35480060fafadac887e234241` · service: `research-agent` · 2026-09-30T06:43:52.928000+00:00
- 1.3s · 5 spans · 1 agents · 1 LLM calls · 1 tool calls · 2 errors

## Input

**user**: Customer acme-404 says billing is wrong. What plan are they on?

## Steps

- **llm** `ChatOpenAI` · 1.30s · claude-sonnet-4-5 · 659→60 tok · request `chatcmpl-3bf9cef4-…`
  - output: **assistant** → `get_customer_plan({"customer_id": "acme-404"})`
- **tool** `get_customer_plan` · 0.00s · **FAILED**
  - error: `ValueError('customer acme-404 not found in billing DB')`
  - input: {"customer_id":"acme-404"}

Tests:

uv run pytest tests/test_litellm/tracing tests/test_litellm/proxy/test_tracing_endpoints.py tests/test_litellm/proxy/auth/test_route_checks.py
635 passed

Type

🆕 New Feature
🐛 Bug Fix
✅ Test

Caveats (if any)

Low

Long fields are cut at AGENT_TRACING_MARKDOWN_MAX_FIELD_CHARS (default 2000) so big prompts don't flood the paste
There's no cost in the Markdown yet; it arrives with the spend-enrichment PR stacked on this one

🤖 Generated with Claude Code

CLAassistant · 2026-09-30T16:08:18Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

codecov · 2026-09-30T16:13:11Z

Codecov Report

❌ Patch coverage is 92.66055% with 48 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...integrations/clickhouse/clickhouse_batch_logger.py	83.63%	9 Missing ⚠️
litellm/tracing/receiver.py	84.00%	8 Missing ⚠️
litellm/rust_bridge/traces.py	87.75%	6 Missing ⚠️
litellm/tracing/decode.py	95.62%	6 Missing ⚠️
litellm/tracing/store.py	94.25%	5 Missing ⚠️
litellm/proxy/proxy_server.py	69.23%	4 Missing ⚠️
litellm/proxy/tracing_endpoints.py	93.44%	4 Missing ⚠️
litellm/tracing/markdown.py	95.06%	4 Missing ⚠️
litellm/integrations/clickhouse/schema.py	85.71%	1 Missing ⚠️
litellm/proxy/common_utils/http_parsing_utils.py	91.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

veria-ai · 2026-09-30T16:15:01Z

+
+    def visible_parent(span: Span) -> str | None:
+        parent = by_id.get(span["parent_span_id"] or "")
+        while parent is not None and _hidden(parent):


Medium: Cyclic spans block proxy workers

With tracing enabled, a normal virtual-key holder can submit a root span, a framework span whose parent is itself, and a visible child of that framework span through POST /v1/traces, then request GET /v1/traces/{id}?format=md. Normalization and JSON trace loading accept this graph, but this loop never terminates; since the renderer executes synchronously inside the async endpoint, the export blocks that worker from serving other requests.

Track visited span IDs and reject or cut cyclic parent chains before following the next parent.

veria-ai · 2026-09-30T16:15:08Z

PR overview

The PR adds Markdown export of agent traces for Claude and Codex, including rendering trace spans through the trace retrieval endpoint.

One issue remains open, with none addressed so far. When tracing is enabled, a virtual-key holder can submit a cyclic span graph and request its Markdown export, causing the renderer to loop indefinitely. Because rendering runs synchronously inside the async endpoint, this blocks the affected proxy worker from serving other requests.

Open issues (1)

Medium: Cyclic spans block proxy workers — litellm/tracing/markdown.py:82

Fixed/addressed: 0 · PR risk: 6/10

greptile-apps · 2026-09-30T16:15:33Z

Confidence Score: 0/5

[Medium risk] Adds markdown export for agent traces via API.

The PR is not safe to merge until trace access and the incorrect or failing Markdown export paths are fixed.

Findings

Admin viewer trace reads blocked ▶
Hidden span children disappear ▶
Missing span returns success ▶
Large trace exports fail ▶
Traceback-only errors break export ▶
Repeated spans get mismatched IO ▶
Tool arguments bypass length limit ▶

Summary

The PR adds scoped Markdown and subtree exports for agent traces and makes templated native trace routes accessible to virtual keys.

The route-classification change also blocks proxy-admin-view-only trace reads.
Subtree selection, large IO reads, error formatting, and duplicate-span IO selection need correction.
Tool-call arguments bypass the configured Markdown field limit.

_{Reviews (1) · Last reviewed commit: "test(auth): trace routes are callable by..."}

greptile-apps · 2026-09-30T16:15:37Z

+        # native routes with placeholders, e.g. "/v1/traces/{trace_id}"
+        for native_route in LiteLLMRoutes.litellm_native_routes.value:
+            if "{" in native_route and RouteChecks._route_matches_pattern(route=route, pattern=native_route):
+                return True


Admin viewer trace reads blocked

When a proxy-admin-view-only user requests an individual trace, this new match classifies the URL as an LLM API route. The admin-viewer access check rejects those routes before it allows safe GET requests, so both JSON and Markdown trace reads now return 403 even though scope_for() grants that role trace access.

greptile-apps · 2026-09-30T16:15:37Z

+        header += [f"## Subtree of `{root['name']}`", ""]
+        lines.append(_span_block(root, (root_input, root_output), 0))
+        walk(root["span_id"], 1)


Hidden span children disappear

When span_id names a framework span or a hidden model or tools chain, the export starts walking from that span. Its children have already been moved under a visible ancestor, so the requested subtree silently omits them.

greptile-apps · 2026-09-30T16:15:37Z

+    if root is None:
+        return "\n".join([*header, "_no spans_"])


Missing span returns success

When the trace exists but the requested span_id does not, this branch renders _no spans_. The Markdown endpoint then returns 200, misleading callers instead of returning 404 as the existing span-detail endpoint does.

greptile-apps · 2026-09-30T16:15:37Z

+TRACE_IO_SQL: Final = f"""
+SELECT SpanId AS span_id, Input AS input, Output AS output
+FROM {OTEL_TRACES_TABLE}
+WHERE TraceId = {{trace_id:String}} AND {_SCOPE_OTEL}
+"""


Large trace exports fail

When a trace's stored input and output exceed the reader's 4 MiB response limit, this query fails before Markdown rendering can apply the 2,000-character field limit. A valid trace whose metadata-only JSON response is readable can therefore fail to export as Markdown rather than producing clipped text.

greptile-apps · 2026-09-30T16:15:37Z

+
+def _first_error_line(error: str) -> str:
+    """LangSmith records `repr(exc)` + traceback with no separator; keep just the exception."""
+    return error.split("Traceback (most recent call last):", 1)[0].splitlines()[0].strip()


Traceback-only errors break export

If an OTLP error status begins with Traceback (most recent call last):, splitting before that marker leaves no first line to index. This raises IndexError, so one failed span prevents the whole Markdown trace from exporting.

greptile-apps · 2026-09-30T16:15:37Z

+TRACE_IO_SQL: Final = f"""
+SELECT SpanId AS span_id, Input AS input, Output AS output
+FROM {OTEL_TRACES_TABLE}
+WHERE TraceId = {{trace_id:String}} AND {_SCOPE_OTEL}
+"""


Repeated spans get mismatched IO

When a span has repeated stored rows, get_trace() selects its earliest timestamped row, but this unordered IO query returns every row and get_span_io() keeps whichever appears last. The Markdown export can pair the displayed span with input or output from a different row, and the result can vary between reads.

greptile-apps · 2026-09-30T16:15:37Z

+        for call in m.get("tool_calls") or []:
+            lines.append(f"**{m['role']}** → `{call.get('name')}({json.dumps(call.get('args'), default=str)})`")


Tool arguments bypass length limit

Tool-call arguments are serialized directly into the Markdown line without the clipping applied to message content. A large arguments object can dominate the paste despite the configured maximum field length, making the export harder to use.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

ishaan-berri · 2026-09-30T22:03:13Z

Superseded by #43932, which rebases this onto main now that #43915 merged.

ishaan-berri had a problem deploying to e2e-changed September 30, 2026 16:08 — with GitHub Actions Error

veria-ai Bot reviewed Sep 30, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

yujonglee-berri added 12 commits September 30, 2026 09:36

wip

7a5cd81

feat(traces): establish shared Rust storage foundation

fdbd54e

fix(traces): escape ClickHouse text parameters

ed9ac76

test(traces): exercise response cap with bounded strings

194b88a

fix(traces): remove unnecessary lint expectation

bf4150f

fix(traces): encode ClickHouse timestamp units in Rust

98711e7

test(traces): mark exception match as a regex

6c67eb5

refactor(traces): execute schema setup in Rust

573c0ac

refactor(traces): use shared logging execution wrapper

874742b

docs(traces): replace foundation README with boundary rules

62aaaf4

fix(traces): use current bridge execution facade

535c3ad

feat(tracing): add tenant scoped agent tracing integration

c7482fd

yujonglee-berri force-pushed the litellm_agent_tracing_integration branch from 0abf1e4 to c7482fd Compare September 30, 2026 16:56

yujonglee-berri and others added 10 commits September 30, 2026 10:09

fix(tracing): include new tests in CI and meet UI lint budgets

799ba8c

style(tracing): format Rust bridge facade

b0c1c56

feat(tracing): render traces as Markdown for Claude / Codex

495f52a

feat(tracing): fetch all span input/output in one query for exports

af2bbc3

feat(tracing): add TraceReceiver.get_trace_markdown

fa57efa

feat(proxy): support format=md on GET /v1/traces/{trace_id}

8ed0c6e

fix(auth): match placeholder native routes like /v1/traces/{trace_id}

bd80952

feat(tracing): add markdown export field limit constant

1d1a5d8

test(tracing): cover Markdown trace export

baaee82

test(auth): trace routes are callable by virtual keys

e3f5f65

ishaan-berri force-pushed the litellm_trace-markdown-export branch from a6e29f1 to e3f5f65 Compare September 30, 2026 18:29

ishaan-berri requested a deployment to e2e-changed September 30, 2026 18:29 — with GitHub Actions Waiting

ishaan-berri mentioned this pull request Sep 30, 2026

feat(ui): agent traces tab on logs with timeline and otel setup guide #43886

Closed

3 of 5 tasks

yujonglee-berri force-pushed the litellm_agent_tracing_integration branch 2 times, most recently from df529d4 to b7ab950 Compare September 30, 2026 19:00

devin-ai-integration Bot force-pushed the litellm_agent_tracing_integration branch from b7ab950 to 4a5f738 Compare September 30, 2026 19:45

ishaan-berri mentioned this pull request Sep 30, 2026

fix(tracing): let virtual keys open traces and export them as markdown #43932

Open

3 of 5 tasks

ishaan-berri closed this Sep 30, 2026

		for call in m.get("tool_calls") or []:
		lines.append(f"{m['role']} → `{call.get('name')}({json.dumps(call.get('args'), default=str)})`")

Conversation

ishaan-berri commented Sep 30, 2026

TLDR

User Flow

Pre-Submission checklist

Screenshots / Proof of Fix

Before (litellm_agent_tracing_integration)

After (this branch)

Type

Caveats (if any)

Low

Uh oh!

CLAassistant commented Sep 30, 2026

Uh oh!

codecov Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

veria-ai Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

veria-ai Bot commented Sep 30, 2026

PR overview

Open issues (1)

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 0/5

Findings

Summary

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

ishaan-berri commented Sep 30, 2026

Uh oh!

This branch is waiting to be deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Sep 30, 2026 •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading