feat(traces): tracing development seed by devin-ai-integration[bot] · Pull Request #44363 · BerriAI/litellm

devin-ai-integration · 2026-10-03T08:22:40Z

TLDR

Problem this solves:

Long-running tracing development branch needs review and CI coverage
Traces, ClickHouse export, and dev fixtures were not on main

How it solves it:

Lands the litellm_tracing_dev_seed branch as one reviewable PR
Rust traces and traces-clickhouse crates with generated wire contracts
Span normalization rules isolated per format with behavior tests
Dev fixture seeder produces linked tracing and spend rows locally
ClickHouse spend rows redact credential-like custom metadata
Custom metadata is dropped entirely when message logging is off
Success rows with no calculated cost keep spend as null
Clearing a tag's max budget in the UI now removes it

Caveats (if any)

Severe

Development seed branch: contains wip and checkpoint commits, roughly 40k added lines
No live-proxy QA run captured yet; the proof section stays empty until one is run

Medium

Custom metadata like prompt is still stored when message logging is on
- Only credential-named keys are redacted in that mode

Type

🆕 New Feature
🐛 Bug Fix

Link to Devin session: https://app.devin.ai/sessions/f4624b86e1e241aba359c4274f653d25
Open in Devin Desktop: https://app.devin.ai/desktop/session/f4624b86e1e241aba359c4274f653d25?variant=devin
Requested by: @yujonglee-berri

Each span format (Claude Code, LangSmith, OpenInference, gen_ai) now lives under normalize/convention/ as a unit struct implementing Convention, owning both its detection and its extraction. Precedence is one ordered registry instead of an if-chain in mod.rs that reached into each module differently. The modules now share one way to read attributes: present() for the first non-empty key and Payload for a text that also reports the key it consumed, replacing three different idioms and the &mut Vec threaded through payload readers. Instrumentation::adjust returns a new Extraction instead of mutating one, with each SDK rule as its own function, and the LangChain middleware suffix list exists once.

…ator

… native boundary

devin-ai-integration · 2026-10-03T08:22:43Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

greptile-apps · 2026-10-03T08:27:41Z

Confidence Score: 1/5

[High risk] Database schema migrations and trace storage configuration.

The PR is not ready to merge because interrupted or restarted background settlements can lose billing, and pending settlement rows can retain sensitive reservation metadata.

Summary

This PR adds tracing storage and normalization, generated trace contracts, and development fixtures. Changes since the previous review also add durable background-interaction settlement, GitLab ROI support, and dashboard and proxy updates.

Background settlement needs recovery for interrupted claims and the creating deployment’s retrieval configuration.
Its persisted context also needs protection for sensitive reservation metadata.

_{Reviews (2) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."}

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

codspeed · 2026-10-03T08:41:46Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_tracing_dev_seed (d999a07) with main (d260765)}

codecov · 2026-10-03T08:48:20Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-10-03T09:22:41Z

@greptileai

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

yujonglee-berri added 30 commits October 2, 2026 18:49

feat(dev): seed linked tracing and spend fixtures

b399485

chore(dev): use OpenAI model in tracing config

5240d80

chore(dev): align tracing credentials with UI E2E

f6316fc

fix(dev): update fixture seeder query scope

ec71b92

feat(dev): seed linked tracing and spend fixtures

5801d2d

chore(dev): use OpenAI model in tracing config

42469c2

chore(dev): align tracing credentials with UI E2E

7c61378

fix(dev): update fixture seeder query scope

874ec79

wip

fec1404

wip

1430943

wip

50b9a83

chore(trace): checkpoint ongoing Rust migration

ca971d1

chore: merge upstream tracing development branch

df7a202

refactor(trace): group Python bridge under trace package

ce4ac00

feat(trace): export Rust-owned wire schemas and enforce contract bounds

6a6e2cb

fix(trace): bound quoted counts in ClickHouse wire schemas

222466e

feat(trace): generate Python wire contracts with datamodel-code-gener…

7709fe9

…ator

test(trace): validate migrated callers and generated contracts at the…

dcd82f9

… native boundary

refactor(traces): rename normalization convention to format

9271432

fix(traces): reconcile spend evidence and preserve unknown costs

7c181c0

feat(traces): normalize additional telemetry formats

46a6598

test(traces): cover captured normalization fixtures

f6686fb

refactor(traces): isolate SDK normalization rules

e0f87a6

feat(tracing): seed all trace exports for local dashboard

6098bd0

fix(clickhouse): preserve custom LiteLLM request metadata

8ca1a05

docs(traces): define normalization module boundaries

badb840

docs(traces): define resolution and OTLP boundaries

d992695

fix(ui): normalize nullable trace message names

498dc7c

refactor(traces): split resolver modules and cover resolution behavior

d94442e

yujonglee-berri added 6 commits October 3, 2026 00:04

test(traces): replace normalization snapshots with behavior assertions

7e3b121

fix(ui): align dashboard API contracts with generated types

88786c8

refactor(traces): type normalization and storage boundaries

aa76678

fix(traces): seed captured SDK spend and preserve provider identities

4835753

wip

4decb23

test(traces): verify guide discovery and content ordering

cf57506

devin-ai-integration Bot requested a review from a team October 3, 2026 08:22

devin-ai-integration Bot assigned yujonglee-berri Oct 3, 2026

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/integrations/clickhouse/clickhouse_spend_logger.py Outdated

Comment thread ui/litellm-dashboard/src/app/(dashboard)/tag-management/_components/tag_info.tsx Outdated

Comment thread litellm/integrations/clickhouse/clickhouse_spend_logger.py Outdated

Merge branch 'main' into litellm_tracing_dev_seed

cd35674

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

yujonglee-berri enabled auto-merge (squash) October 3, 2026 09:10

yujonglee-berri approved these changes Oct 3, 2026

View reviewed changes

yujonglee-berri and others added 6 commits October 3, 2026 09:22

fix(clickhouse): redact custom metadata and keep unknown spend null

9a60517

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(tags): clear explicitly nulled tag budget fields

5baa68f

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(ui): import zod directly and track Next type references

66a8ab1

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

test(traces): fix ClickHouse migration fixture parameters

c15cdd8

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(ci): package Lens prompts and defer Prisma imports in fixture seeder

07a4937

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into litellm_tracing_dev_seed

5aaaacc

yujonglee-berri and others added 3 commits October 3, 2026 09:51

style(clickhouse): format spend logger

65b328e

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(traces): import Prisma row type at runtime

9798c23

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

test(tracing): stabilize gzip test collection

d999a07

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

yujonglee-berri merged commit e340e54 into main Oct 3, 2026
96 of 97 checks passed

yujonglee-berri deleted the litellm_tracing_dev_seed branch October 3, 2026 09:59

shudonglin mentioned this pull request Oct 4, 2026

chore: sync upstream 2026-10-03 rayward-external/litellm#293

Merged

Nikhi00718 mentioned this pull request Oct 8, 2026

fix(tracing): preserve message parts and embedded tool calls #44078

Closed

4 of 6 tasks

Navigation Menu

Repository navigation

feat(traces): tracing development seed - #44363

feat(traces): tracing development seed#44363
yujonglee-berri merged 46 commits into
mainfrom
litellm_tracing_dev_seed

devin-ai-integration Bot commented Oct 3, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented Oct 3, 2026

Uh oh!

greptile-apps Bot commented Oct 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codspeed Bot commented Oct 3, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 3, 2026

Uh oh!

devin-ai-integration Bot commented Oct 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

devin-ai-integration Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

Caveats (if any)

Severe

Medium

Type

Uh oh!

devin-ai-integration Bot commented Oct 3, 2026

Uh oh!

greptile-apps Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 1/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codspeed Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

codecov Bot commented Oct 3, 2026

Codecov Report

Uh oh!

devin-ai-integration Bot commented Oct 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented Oct 3, 2026 •

edited

Loading

greptile-apps Bot commented Oct 3, 2026 •

edited

Loading

codspeed Bot commented Oct 3, 2026 •

edited

Loading