Support emitting session.id on LLM spans in Envoy AI Gateway’s OpenTelemetry export for Phoenix session grouping

[using-system]

Hello,

Is session.id support for trace OpenTelemetry export is already available behind a flag or in a newer build ? (https://opentelemetry.io/docs/specs/semconv/general/session/)

Or it's possible to include the session.id semantic attribute
on LLM-related spans emitted by Envoy AI Gateway’s OpenTelemetry exporter, so Arize Phoenix can interpret and group traces by session in its “Session” tab.

Proposed implementation sketch

Accept a configurable source for session identity (priority order), for example: HTTP header (e.g., x-session-id)
OR
Get it from BaggageItem key "session.id" if exists on the current span (https://opentelemetry.io/docs/specs/otel/baggage/api/) ?

Many thanks.

Matthieu

[mathetake]

sounds reasonable request. thoughts? @codefromthecrypt

[codefromthecrypt]

It feels better to have less configuration. If there is an existing convention we should just add the attribute. I was also looking at envoys x-request-id for a similar reason.

Can you be clear about what it is you are using that emits this header? It is always best to be concrete in use case as otherwise it feels like chasing spec and that could displace other work.

What is the framework you are using that is doing sessions and what header is it sending?

[codefromthecrypt]

@anuraaga I think i know the answer but do you know of any otel http instrumentation that have a config for mapping http headers to span attributes?

[anuraaga]

AFAIK it's pretty common. In Java, any instrumentation should support it thanks to the instrumentation-api. Other languages will be less consistent.

https://opentelemetry.io/docs/zero-code/java/agent/instrumentation/http/

[codefromthecrypt]

Cool it looks like that link has an env list so would capture exact header names as attributes. Strongly prefer to use an existing env variable approach vs add more config.

[codefromthecrypt]

Also it looks like session.id is natively handled in Phoenix https://arize.com/docs/phoenix/tracing/llm-traces/sessions

[using-system]

Also it looks like session.id is natively handled in Phoenix https://arize.com/docs/phoenix/tracing/llm-traces/sessions

Yes. From my understanding they rely on the session.id attribute on the current span (see OpenTelemetry semconv: https://opentelemetry.io/docs/specs/semconv/general/session/).

We should retrieve session.id and set it as an attribute on the current span before making the call (e.g., as described here: https://arize.com/docs/ax/observe/sessions-and-users).

The open question is how to retrieve that session.id in the gtw. The most transparent approach i think is to read the baggage item session.id if it exists in the baggage header (https://opentelemetry.io/docs/specs/otel/baggage/api/
), and then copy it onto the span. The same pattern can be used for user.id ?

const bag = propagation.getBaggage(extractedCtx); const sessionId = bag?.getEntry("session.id")?.value; const userId = bag?.getEntry("user.id")?.value; ... if (sessionId) span.setAttribute("session.id", sessionId); if (userId) span.setAttribute("user.id", userId);

1. extract session.id / user.id from incoming baggage (or other solution)
2. set them as span attributes,.

Prerequisite: the only requirement is that the caller uses OpenTelemetry HTTP instrumentation while following the W3C standards (common across SDKs) and sets the baggage items session.id and user.id when needed.

Alternative (no OTel/Baggage dependency at caller): carry X-Request-ID and X-User-ID; the gtw then reads those headers and sets session.id, and user.id as span attributes for OpenInference.

[codefromthecrypt]

@using-system I know I asked about otel norms, but I think we should think first about use case, second about spec. I don't think otel has a solution for capturing the session.id, and I am also not convinced there is large practice in using baggage for this. So, let's anchor on practice first and if the specs help, that's great, but so far as I can tell, they only get you half way.

e.g. OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS="x-session-id" -> "http.request.header.x_session_id" attribute, not session.id
https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/wsgi/wsgi.html#request-headers

For example, spring boot you would have to set your own code to add a session.id header. or something to rederive one.

So, I want to re-iterate the question from before:

Are you currently using any approach or concrete genai library? or are you looking for a spec to follow? I'm wondering how you got to this session.id requirement, or if you are solving for someone else. Context is helpful (pun intended)

We need to understand the urgency and context of a requirement as it is very likely to do this, we need a custom solution, and end use is the much preferred way. We can start asking existing users which practices they follow, and how it would work for them. Designing without users is a really bad way to move forward.

[using-system]

@codefromthecrypt

I’m using two stacks:

.NET 6–10, kestrel server, with Semantic Kernel (GenAI metrics and tracing support). For Otel Export i use Microsoft nuget packages OpenTelemetry.*

Python, fastapi server, with LangChain and the arize-phoenix package for GenAI traces. For Otel Export i use opentelemetry-instrumentation-*

It's work fine for all span kind agent, chain tool... and all theses spans have session.id attribute. But same problem with .NET, llm span generated by the ai gtw has no session.id attribute.

All theses stacks (+ envoy ai gtw) export to a grpc endpoint Otel Collector contrib disto (then export to grafana/phoenix)

I also try to add a extra env var OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST on the extProc envoy ai gtw (for capture the header then rewrite it on the good format with otlp collector) but it's seem doesn't working with go exporter.

I also try telemetry. tracing. customTags.RequestHeader https://gateway.envoyproxy.io/contributions/design/proxy-tracing/. It's works fine but only for the envoy gateway spans not the llm span generated by extProc.

It's seems the limitation with Phoenix is that the llm-kind span (i.e., the one generated by the Envoy AI Gateway) must have the session.id attribute for the conversation to appear in the Sessions tab. It isn’t smart enough to pull all spans by trace_id and correlate them with the root span, which does contain session.id, for example. A “dirty” workaround would be to reconstruct an llm-kind span with the gateway response from the agent. Other solution is the send an issue on Phoenix to see if it's possible to the backend to correlate span of the trace.

[codefromthecrypt]

Thanks, and it's a good overview. So, in your frameworks, what is actually propagating the session ID? (Is it doing this now? Or are you flexible on how to do this) Because adding to span is hop by hop. I think you mentioned earlier both a header and a baggage (multi-header) approach.

As there is no current otel env to remap a header key as an attribute key, or even to pass a header as-is as an attribute.. I think we would have to make a new ENV to do one of..

Map a given header to the session.id attribute (very clear but only handles this use case)
E.g. OTEL_AIGW_SESSION_ID_HEADER=x-session-id

Map a pair list of header/attribute names (can do more than just this)
E.g. OTEL_AIGW_HEADER_ATTRIBUTE_MAP=x-session-id=session.id

The exact convention might vary a bit, but I think those are the two main patterns an existing framework is likely to be reuse as they don't require adding an otel dep (baggage is obscure and implies a library where many can add an extra header)

[using-system]

For propagating the InProc or dependencies span attribute, I use a Baggage item that is propagated via a listener into every span as an attribute.

I also have the ability, via LangChain or Semantic Kernel, to send a custom header when calling the inference model.

I fully agree that handling this via a Baggage item would be obscure, and that an OTEL_AIGW_HEADER_ATTRIBUTE_MAP, where we can map one or more headers to a span attribute (similar to telemetry.tracing.customTags.RequestHeader https://gateway.envoyproxy.io/contributions/design/proxy-tracing/
but just by an env var), would be ideal—and, above all, generic enough to cover all use cases (not just Phoenix).

[codefromthecrypt]

@using-system I have a couple open PRs, and want to raise one for #1231 and once they file in, I'm happy to help with this. If you can help by trying the ones I've opened it might get me free faster on this part!

[using-system]

Thanks @codefromthecrypt for all your work on this project.
AI gateways are becoming indispensable. I’ll keep advocating for this project and will continue to highlight it in my posts and Medium articles.
I’m a bit short on time at the moment and not relly a go dev, but I’ll try to run through the PRs you opened when I can and share feedback. When the feature raised, I’ll take a quick look as well. I can’t promise thorough testing right now, but I’ll help where possible.

[codefromthecrypt]

@using-system p.s. https://github.com/elastic/observability-examples/tree/main/genai-function-calling shows how to use various auto-instrumentation, and as long as outbound http works from the client side, you might be able to work around lack of session.id for a bit. That said, I'll run through the session.id capture change here as proposed. no worries on reviewing go code!