Add otel tracing instrumentation by sallyom · Pull Request #506 · llm-d/llm-d-router

sallyom · 2025-12-05T13:20:16Z

This PR satisfies the llm-d distributed tracing proposal llm-d/llm-d#119 implementation plan.

github-actions · 2025-12-29T01:38:40Z

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

sallyom · 2026-01-07T04:27:41Z

The GAIE entry request span PR is kubernetes-sigs/gateway-api-inference-extension#2057

zdtsw · 2026-01-07T14:12:16Z

 func (s *Server) chatCompletionsHandler(w http.ResponseWriter, r *http.Request) {
+	requestStart := time.Now()
+	tracer := telemetry.Tracer()
+	ctx, span := tracer.Start(r.Context(), "llm_d.pd_proxy.request",


read the proposal part seem iit has decided to use llm_d.pd_proxy.

not for this PR, but should we change the folder to pkg/pd/proxy than pkg/sidecar/proxy for better naming?

Down the road we may have the sidecar proxy handle more than P/D disagg so I would suggest not renaming.

ok leaving as/is for this PR

Add centralized telemetry package and custom spans following the llm-d distributed tracing proposal. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com>

Signed-off-by: sallyom <somalley@redhat.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com>

Signed-off-by: sallyom <somalley@redhat.com>

sallyom · 2026-02-23T21:34:24Z

Usually I see a 'Verified' tag on the signed commits:

Commits show as verified now - I had to add a new signing key.

wrt the linting error, the test is complaining about the package name common but this PR isn't introducing that package name, it's pre-existing.

kfswain · 2026-02-23T23:17:43Z

/approve

elevran · 2026-02-24T15:46:41Z

@sallyom thanks for following through on the feedback.

wrt the linting error, the test is complaining about the package name common but this PR isn't introducing that package name, it's pre-existing.

The PR adds the _test.go file. Unlike the existing file, it lacks a linter bypass comment (//revive:disable:var-naming) so it is flagged.

I'm looking at the signature requirement:

the github UI shows commits as signed
the github cli shows them as unsigned

$ git fetch upstream pull/506/head:pr-506
$ git checkout pr-506
$ git log --pretty="%H %s" --show-signature 2>/dev/null
No signature
693a0294eb5801e2715e815f7b15483dd68c33d1 remove extra comments from sidecar spans
No signature
9aedcc978cb9c85200eb73217985c6746b56cd53 test: add edge case tests for StripScheme
No signature
5ddef7a5831272d6c906bb92693f68d807676eff fix: address review nits for tracing PR
No signature
104f3c8cc7c689c750a165adabf2587d9c8753c2 fix: avoid os.Exit bypassing defer in main
No signature
da3c41640860e0d1b47f6947fd9bf3830bc87f14 tracing: remove extra success results & startup spans and cleanup
No signature
3a31c9d5999e263f814a477a52548d182fadcdb0 update Dockerfile.sidecar
No signature
4cf00716d55c55bc5286481a2ba2474bd66b28bc Add opentelemetry tracing
gpg: Signature made Thu Feb 19 19:28:16 2026 IST
gpg:                using RSA key B5690EEEBB952194
gpg: Can't check signature: No public key
...

elevran · 2026-02-24T16:06:41Z

worked around the signing verification be reruning the test after deleting "the found unsigned commits" comment (perhaps there was some left over state - who knows :-)).
Anyway - still need to fix the lint error and then we are good to go!

Signed-off-by: sallyom <somalley@redhat.com>

Signed-off-by: greg pereira <grpereir@redhat.com>

protect against segfault on tests

* Add opentelemetry tracing Add centralized telemetry package and custom spans following the llm-d distributed tracing proposal. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * update Dockerfile.sidecar Signed-off-by: sallyom <somalley@redhat.com> * tracing: remove extra success results & startup spans and cleanup Signed-off-by: sallyom <somalley@redhat.com> * fix: avoid os.Exit bypassing defer in main Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * fix: address review nits for tracing PR Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * test: add edge case tests for StripScheme Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * remove extra comments from sidecar spans Signed-off-by: sallyom <somalley@redhat.com> * fix lint error Signed-off-by: sallyom <somalley@redhat.com> * protect against segfault on tests Signed-off-by: greg pereira <grpereir@redhat.com> --------- Signed-off-by: sallyom <somalley@redhat.com> Signed-off-by: greg pereira <grpereir@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: greg pereira <grpereir@redhat.com> (cherry picked from commit 1519a28)

* Add opentelemetry tracing Add centralized telemetry package and custom spans following the llm-d distributed tracing proposal. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * update Dockerfile.sidecar Signed-off-by: sallyom <somalley@redhat.com> * tracing: remove extra success results & startup spans and cleanup Signed-off-by: sallyom <somalley@redhat.com> * fix: avoid os.Exit bypassing defer in main Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * fix: address review nits for tracing PR Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * test: add edge case tests for StripScheme Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * remove extra comments from sidecar spans Signed-off-by: sallyom <somalley@redhat.com> * fix lint error Signed-off-by: sallyom <somalley@redhat.com> * protect against segfault on tests Signed-off-by: greg pereira <grpereir@redhat.com> --------- Signed-off-by: sallyom <somalley@redhat.com> Signed-off-by: greg pereira <grpereir@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: greg pereira <grpereir@redhat.com> (cherry picked from commit 1519a28) (cherry picked from commit 215658e)

* Add opentelemetry tracing Add centralized telemetry package and custom spans following the llm-d distributed tracing proposal. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * update Dockerfile.sidecar Signed-off-by: sallyom <somalley@redhat.com> * tracing: remove extra success results & startup spans and cleanup Signed-off-by: sallyom <somalley@redhat.com> * fix: avoid os.Exit bypassing defer in main Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * fix: address review nits for tracing PR Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * test: add edge case tests for StripScheme Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com> * remove extra comments from sidecar spans Signed-off-by: sallyom <somalley@redhat.com> * fix lint error Signed-off-by: sallyom <somalley@redhat.com> * protect against segfault on tests Signed-off-by: greg pereira <grpereir@redhat.com> --------- Signed-off-by: sallyom <somalley@redhat.com> Signed-off-by: greg pereira <grpereir@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: greg pereira <grpereir@redhat.com>

github-project-automation Bot added this to llm-d-router Dec 5, 2025

sallyom force-pushed the tracing branch from 662eba9 to 60d4150 Compare December 7, 2025 15:24

github-actions Bot added the lifecycle/stale label Dec 29, 2025

github-actions Bot added lifecycle/rotten and removed lifecycle/stale labels Jan 5, 2026

sallyom force-pushed the tracing branch 4 times, most recently from 9c56950 to c55c5a6 Compare January 5, 2026 20:14

github-actions Bot removed the lifecycle/rotten label Jan 6, 2026

sallyom force-pushed the tracing branch 3 times, most recently from 884c096 to 8c4a03d Compare January 7, 2026 04:20

sallyom changed the title ~~[WIP] Tracing~~ Add otel tracing instrumentation Jan 7, 2026

sallyom force-pushed the tracing branch 7 times, most recently from 24dc5b9 to 15eb137 Compare January 7, 2026 05:44

wseaton reviewed Jan 7, 2026

View reviewed changes

Comment thread pkg/plugins/scorer/precise_prefix_cache.go Outdated

zdtsw reviewed Jan 7, 2026

View reviewed changes

sallyom force-pushed the tracing branch from 15eb137 to ff87a59 Compare January 15, 2026 02:10

github-actions Bot requested review from elevran and nilig January 15, 2026 02:11

sallyom force-pushed the tracing branch from ff87a59 to fca8980 Compare January 15, 2026 02:45

github-actions Bot requested a review from kfswain January 15, 2026 02:45

github-actions Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 12, 2026

sallyom force-pushed the tracing branch from e09b7df to b7a4a40 Compare February 23, 2026 18:34

sallyom and others added 7 commits February 23, 2026 16:27

Add opentelemetry tracing

4cf0071

Add centralized telemetry package and custom spans following the llm-d distributed tracing proposal. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com>

update Dockerfile.sidecar

3a31c9d

Signed-off-by: sallyom <somalley@redhat.com>

tracing: remove extra success results & startup spans and cleanup

da3c416

Signed-off-by: sallyom <somalley@redhat.com>

fix: avoid os.Exit bypassing defer in main

104f3c8

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com>

fix: address review nits for tracing PR

5ddef7a

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com>

test: add edge case tests for StripScheme

9aedcc9

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sallyom <somalley@redhat.com>

remove extra comments from sidecar spans

693a029

Signed-off-by: sallyom <somalley@redhat.com>

sallyom force-pushed the tracing branch from b7a4a40 to 693a029 Compare February 23, 2026 21:27

github-actions Bot previously approved these changes Feb 23, 2026

View reviewed changes

github-project-automation Bot moved this to In review in llm-d-router Feb 23, 2026

llm-d deleted a comment from github-actions Bot Feb 24, 2026

fix lint error

e7b7d29

Signed-off-by: sallyom <somalley@redhat.com>

sallyom dismissed github-actions[bot]’s stale review via e7b7d29 February 24, 2026 20:17

elevran reviewed Feb 24, 2026

View reviewed changes

Comment thread pkg/sidecar/proxy/chat_completions.go Outdated

Gregory-Pereira and others added 2 commits February 24, 2026 13:11

protect against segfault on tests

d747153

Signed-off-by: greg pereira <grpereir@redhat.com>

Merge pull request #1 from Gregory-Pereira/tracing

9099259

protect against segfault on tests

Gregory-Pereira enabled auto-merge (squash) February 24, 2026 21:59

Gregory-Pereira disabled auto-merge February 24, 2026 22:45

kfswain approved these changes Feb 25, 2026

View reviewed changes

github-actions Bot merged commit 1519a28 into llm-d:main Feb 25, 2026
8 checks passed

github-project-automation Bot moved this from In review to Done in llm-d-router Feb 25, 2026

Conversation

sallyom commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Dec 29, 2025

Uh oh!

sallyom commented Jan 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zdtsw Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

elevran Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

sallyom Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sallyom commented Feb 23, 2026

Uh oh!

kfswain commented Feb 23, 2026

Uh oh!

elevran commented Feb 24, 2026

Uh oh!

elevran commented Feb 24, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sallyom commented Dec 5, 2025 •

edited

Loading