OTEP: Context-scoped Attributes

[carlosalberto]

Add Context-scoped telemetry attributes which typically apply to all signals associated with a trace as it crosses a single service.

Resurrecting open-telemetry/oteps#207 while updating some changes that have happened since (e.g. changes in InstrumentationScope being a runtime concept rather than a compile-time one). Originally proposed by @Oberon00

Additionally:

1. A Processor alternative being discarded as we would need to add an additional pre OnStart call in order to support Samplers.
2. The open question whether Baggage should be included as part of stamping attributes to individual telemetry items (similar enough functionality).
3. We disable this functionality by default. Users have to opt-in to use this.

Java prototype for traces/logs: https://github.com/open-telemetry/opentelemetry-java/compare/main...carlosalberto:context-scoped-attrs-proto?expand=1

Summary

This OTEP aims to address various related demands that have been brought up in the past, where the scope of Resource attributes is too broad, but the scope of Span attributes is too narrow. For example, this happens where there is a mismatch between the OpenTelemetry SDK’s (and thus TracerProvider’s, MeterProvider’s) process-wide initialization and the semantic scope of a (sub)service.

The context-scoped attributes allows you to attach attributes to all telemetry signals emitted within a Context. Context-scoped attributes are normal attributes, which means you can use strings, integers, floating point numbers, booleans or arrays thereof, just like for Span or Resource attributes. Context-scoped attributes are associated with all telemetry signals emitted while the Context containing the Context-scoped attributes is active and are available to telemetry exporters. For spans, the context within which the span is started applies. Like other telemetry APIs, Context-scoped attributes are write-only for applications. You cannot query the currently set Context-scoped attributes, they are only available on the SDK level (e.g. to Samplers and SpanProcessors and exporters).

Context-scoped attributes should be thought of equivalent to adding the attribute directly to each single telemetry item it applies to.

[cijothomas]

resource cannot be used - this seems like a very specific limitation for the jvm example? Isn't it possible to create multiple Meter/Tracer/Logger providers, each with its own resource, for each independent application within the same process?

[carlosalberto]

Included this as an alternative mentioned in the Motivation section (albeit being a prohibitive one, given you would need to duplicate the pipeline).

[jsuereth]

We have a separate OTEP for that which was submitted - the ability to have multiple meter/tracer/logger providers. That has big limitations in terms of cardinality you need to watch out for. I think this limitation here is a good one, even in context of that other OTEP.

[cijothomas]

exporters? That would be incorrect as a batching_processor can call exporter in its background thread, losing the original context.

[carlosalberto]

Clarified this. Context-scoped attributes only apply to telemetry at creation time.

[jmacd]

Samplers are able to capture context-scoped attributes and apply them for traces, logs have the context, and in metrics we discussed #4318. I see this context-scoped attributes as generic for MeasurementProcessor, in other words.

[cijothomas]

lets make it clear what is meant by disabled here. I guess you are referring to "Context-scoped attributes are not automatically added to spans/logs/metrics unless user has explicitly opted into it. Opt-in mechanism are in [link to the section where we talk about it.]"

[jack-berg]

I didn't see any text describing what an opt-in mechanism would look like. I expected it to take the form of config params on SDK TracerProvider, LoggerProvider, MeterProvider, or new built-in SpanProcessor, LogRecordProcessors, MeasurementProcessor (rejected 🙁) with corresponding config handles.

Did I miss something or is the opt-in mechanism TBD?

[cijothomas]

app-level data vs telemetry annotations - not sure of the differences between them.. In my view, the only difference between Baggage is that Baggage is out-of-proc+in-proc, whereas Context-scope Attributes is in-proc only.
(I can think of calling this as InProcOnlyBagagge as well)

What kind of data users put inside it - totally upto them. They can put data to affect control flow, telemetry attributes, feature-flags etc. OTel don't put any restrictions.

[carlosalberto]

Baggage only accepts strings. Also, if we were to go any Baggage way, we should offer automatic attachment of Baggage values at the SDK level, like the one defined by this OTEP.

[jsuereth]

I would still love some kind of configuration or capability (can be outside the scope of this OTEP) where I can say "these baggage attributes should automatically promote to be context-scoped as well".

I do believe we should NOT add bagggage to all signals "by default" but force users to opt-in to things they wish to trust. This is more a 'secure by design' philosophy of requiring users to opt-in to things from "untrusted" (i.e. not within the app) sources.

[cijothomas]

I am not so sure about "write-only" aspect. Why restrict it, when Context itself supports reading:
specification/tree/main/specification/context#get-value

[carlosalberto]

I'd like to reduce the area if possible. If we get cases were it is valuable to expose this to the public, we can go ahead and do that.

[cijothomas]

this API section is merely talking about adding ScopedAttribtues to Context - so it is better to avoid the mentioning of Span case here (and span attributes)

[carlosalberto]

Actually I think examples are useful in order to be super clear. Changed this to "e.g. Span or LogRecordItem".

[dashpole]

Go has had a "labeler" concept specific to http that is similar to this proposal: https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp#ContextWithLabeler. I would love to see a more cohesive and generic design for this!

[dashpole]

I think context-scoped attributes should not be added as metric attributes by default, given the impact of new attributes on cardinality. I would prefer if the default behavior was for the metrics SDK to treat them the same as default-disabled attributes provided to the Metrics API:

• These attributes are added as filtered_attributes on exemplars by default.
• These attributes can be opted-in to metric attributes using Views.

There are definitely cases where a user will specifically want the contextual attribute stored as a metric attribute by default. I would prefer if this was an optional parameter that could be passed when storing the attribute in context.

[carlosalberto]

I would prefer if this was an optional parameter that could be passed when storing the attribute in context.

I think we could add this with a flags attribute as a metric-only feature (as spans nor logs have such opt-in functionality, and hence a simple boolean wouldn't be so descriptive):

// API
void SetContextScopedAttributes(
  Context context,
  Attributes attributes,
  ContextScopeFlags flags);

enum ContextScopeFlags {
  NONE = 0,
  METRICS_OPT_IN = 1
}

// Usage from the metrics SDK
contextAttributes, flags = GetContextScopedAttributes(context);
if (flags & ContextFlags.METRICS_OPT_IN != 0) {
  // Conditional setting.
} else {
  // set them on metrics unconditionally.
}

Is this something close to what you have in mind?

[dashpole]

Yeah, more or less. In Go I was thinking it would just use the option pattern (like most of our parameters do):

// User code
ctx := context.Background()
attrs := attribute.NewSet(attribute.String("foo", "bar"))
ctx = contextual.ContextWithAttributes(ctx, attrs, contextual.WithAddToMetrics(true))

// Metrics SDK Code
attrs, config := contextual.AttributesFromContext(ctx)
if config.AddToMetrics() {
    // Add attrs directly to metric attributes
} else {
    // Add attrs to filtered attributes passed to exemplarReservoir.Offer.
}

[jack-berg]

I think context-scoped attributes should not be added as metric attributes by default, given the impact of new attributes on cardinality.

The key question for me is about configuration: just how granular / expressive should the configuration tools be. More granular mean more verbose, but a tool capable of solving more problems. Less granular means easier to configure, but less capable.

The PR currently proposes the least granular option:

file_format: "1.0"
# Add all context scoped attributes to all metrics, logs, traces within scope
context_scoped_attributes: true

On the other end of the spectrum, we might have something like:

file_format: "1.0"
tracer_provider:
  # include all context scoped attributes on spans
  context_scoped_attributes:
    included:
      - *
logger_provider:
  # include all context scoped attributes on logs
  context_scoped_attributes:
    included:
      - * 
meter_provider:
  # include all context scoped attributes excepted "foo" and "bar" on metrics
  context_scoped_attributes:
    included:
      - *
    excluded:
      - foo
      - bar   

The upside is that there's a built in mechanism to account for metrics cardinality issues, and any other expressiveness use cases that arise. The downside is that the common case of "enable all context scoped attributes on all signals" is more verbose:

file_format: "1.0"
tracer_provider:
  context_scoped_attributes:
    included:
      - *
logger_provider:
  context_scoped_attributes:
    included:
      - * 
meter_provider:
  context_scoped_attributes:
    included:
      - *

There's also a middle ground of providing a top level and signal specific config option:

file_format: "1.0"
# include all context scoped attributes on all signals
context_scoped_attributes:
  included:
    - *
tracer_provider: ...
logger_provider: ...
meter_provider:
  # override metrics context scoped attributes to include all attributes except foo and bar
  context_scoped_attributes:
    included:
      - *
    excluded:
      - foo
      - bar   

Given that declarative config already makes you repeat things like OTLP exporter configuration for each signal, I lean towards the middle ground approach. I would bet that omitting the additional granularity / expressive now would be temporary and we would end up adding it later anyway. The middle ground strikes a nice balance, allowing for relatively terse config for the common case while still accommodating signal-specific config use cases.

[XSAM]

@dashpole I am not sure metrics should be special-cased by default here.

The main value of this OTEP is a simple mental model: if an attribute is added to the active Context, it applies to telemetry produced in that Context.

Once metrics become the only exception, that model gets harder to explain and easier to misuse. Users will see the attribute on spans and logs and reasonably expect it to also be present on metrics from the same operation.

I agree metric cardinality is a real concern, but I don't think we should hard-code that concern into the default behavior. Cardinality is a consequence of which attributes the user chooses to put into context. That is already something users need to think about when they decide to use this API, just like they do today when adding metric attributes directly.

In other words, I think the right split is:

• semantics: context-scoped attributes apply consistently to all signals
• controls: users can disable/filter them for a specific signal.

That keeps the feature understandable while still giving metrics users the guardrails they need.

Also, for the motivating examples in this PR, http.app is exactly the kind of attribute that seems useful on metrics as well.

---

Besides, this seems like the more extensible design. If we special-case metrics now, then any future signal with similar cost/cardinality concerns may need another exception.

[XSAM]

@dashpole

If we only support contextual attributes on all signals at once, then people just won't be able to use it for things that are high-cardinality.

I am not against disabling some signals when adding attributes to the context. I support offering the option to disable signals.

The semantic model I mentioned is just a way to think about the default behavior.

Are you OK with users being able to mark some attributes as "high cardinality" or "not for metrics" when adding them to the context?

Yes

Are you just opposed to making that the default behavior?

Yes

[dashpole]

Sounds good. I'm OK with the default being to add it to all signals.

[cijothomas]

Sounds good. I'm OK with the default being to add it to all signals.

This seems quite risky to me. Metrics is designed for high-performance and by changing the default behavior to pull in attributes from Context, the performance will be significantly affected. (.NET and Rust implementation relies on incoming read-only-slice, and if we need to add attributes from context by default, then it forces constructing a new read-only-slice and copying incoming one to that, This will kill the performance and also trigger heap allocation likely).

[jsuereth]

@cijothomas Theoretically this was always something you needed to do in metrics, because we prepared to attach baggage labels in the initial implementation. You also need context for trace-id in exemplars.

I hear the concern, but I think we need to find clever ways to solve this - e.g. creating an attributes data structure that optmises to avoid allocations but may not be elegant to interact with. I thoroughly believe we need to find a way to deal with this limitation over time.

[dashpole]

@cijothomas Lets separate a few concerns:

• Users will put attributes in context that have unbounded cardinality, and blow up their memory usage.
• The SDK will need to check for attributes in the context on each call, which has a performance cost.

I've been mostly concerned in this thread about whether users will make the right choice about which signals to opt an attribute into.

if we need to add attributes from context by default, then it forces constructing a new read-only-slice and copying incoming one to that, This will kill the performance and also trigger heap allocation likely).

I don't see a way around this. If a user wants to dynamically get attributes from context and attach them to metrics, then they will have to pay this cost. In Go, we would probably use pooling to try to mitigate this.

Note that the "default" behavior is still performant regardless, since users need to explicitly opt-in to use contextual attributes in their application. I think the question of defaults mostly is about ergonomics and expectations vs protecting users from themselves. Once a user has decided they want an attribute attached to metrics, they assume the risks and performance costs associated with it.

[dashpole]

FYI @open-telemetry/go-maintainers I think whatever comes from this proposal is what should replace our otelhttp.Labeler.

[cijothomas]

This section need to be updated to reflect the wording later in the doc "The API would be extended with two operations to get and set Context-scoped
attributes:"

[cijothomas]

at the time of writing this OTEP there is no specified processor functionality for metrics.

This seems like a weak argument right?. Spec could be adding processor concept. Or Views could be modified to support pulling attributes from context. (Such capability existed in Metrics Views originally, but was removed for scope control)

[carlosalberto]

Yes and no - the related effort and discussion took a whole year and it wasn't accepted/merged. I don't know about the Views path but it could take long time before it's actually discussed/accepted (not to mention stabilized).

[jsuereth]

It does seem like the large argument here over automatic inclusion of metrics merits we take a look at it. IIRC - the original context-scoped attributes failed because of scope creep, so I understand wanting to keep this narrow. Maybe we can just focus the discussion on "What's the right mechanism for context scoped attributes to interact with metrics" - and be a little flexible in thinking for that one.

[cijothomas]

I am not so sure of this argument. OTel could add top-level SpanProcessor and LogRecord processor that can be added by the user, if they wish to get the Context-scope attributes to spans or logs. I am yet to see a strong argument that these attribute should be added by default.

[carlosalberto]

This is mentioned in the alternatives, i.e.

• Samplers have their ShouldSample operation invoked before a processor
  can get their OnStart call. Thus we would need to add a BeforeShouldSample or BeforeStart
  operations to the processor interface, which seems redundant.
• Processors would need to be configured per-signal, which is both redundant but could also
  offer more granularity.

[cijothomas]

Sampler's have access to Context, so they can very well check the context scoped attributes themselves.

[jsuereth]

@cijothomas Programmatic requirements for interacting basically negates the feature for a large portion of users - those leveraging auto-instrumentation capabilities and trying for pure config-driven approaches. E.g. in SRE/Ops personas who are managing an opentelemetry solution on kuberenetes that may not have access to all the developers code. Additionally, you may be adding flags/features for context-scope attributes as a framework but you don't own the SDK (you're not the application), so you cannot enforce those processors are configured correctly and it becomes huge friction for making the thing work.

I think we need to find a way to have context-scoped attributes work by default, otherwise we may just be wasting cycles working on something that's too hard to be adopted.

[cijothomas]

Fully agree that the opt-in should be super easy else this will cause adotption to struggle.

If the spec ships a named built-in processor for this, the operators can enable it right? Would this not work for Logs/Spans which already has a concept of Processor that can read-attributes-from-context and put them inside Logs/Spans.

tracer_provider:
  processors:
    - context_scoped_attributes: {}. -> this is what will inject context attributes to Spans. (Similar to logs)
    - batch: ...

Metrics - Either we need to bring measurement processor, or augment views to have the ability to enrich with attributes from context. (We had this originally, got removed for 1.0, but I guess now is the perfect time to bring it back)

This will keep zero overhead for users who don't opt in, and still allows operators to enable this feature easily (and pay a perf cost, but they can make the decision if attributes give them more value than the perf cost)

[cijothomas]

what is the reason to prevent instrumentation libraries from doing it?

[dashpole]

I view this feature as a way to let the end user bypass instrumentation and communicate directly with the SDK. While instrumentation could also use this API for ergonomic reasons, they don't need it. It prevents instrumentation libraries from needing to add API surface to let users tell them what attributes to pass. E.g. WithAttributesFromRequest(func (req http.Request) []attribute.KeyValue).

By using the OTel APIs directly, instrumentation can pass the same attributes with better performance, and can use Enabled to avoid doing constructing attributes when signals are disabled. Unless we have concrete use-cases that outweigh those drawbacks, I would prefer to start by discouraging use of this API by instrumentation.

[wrisa]

Why is the restriction for only end users? There is a use case apart from end users. In agent.name and workflow.name propagation cases, we were considering context scoped attributes. The genAI's TelemetryHandler (instrumentation library code) has to set gen_ai.workflow.name and gen_ai.agent.name for propagation
to child spans.
cc @lmolkova

[wrisa]

See prototype, workflow name added to context-scope attributes open-telemetry/opentelemetry-python-contrib#4446

[jsuereth]

+1 - I can see instrumentation libraries setting them but having that capability be flag-guarded or config-guarded in some way.

[wrisa]

Adding one more example as a use case, conversation.id or other associated attributes passed to the instrumentation by the user. Instrumentation is setting these as context scoped attributes open-telemetry/opentelemetry-python-contrib#4468

[cijothomas]

Thanks for pushing this forward — context-scoped attributes fills a real gap for OTel. There are a lot of comments in the PR already, so it is hard to tell what is blocking vs nice-to-have suggestions.

Requesting changes to raise my main concern:

I'm opposed to the default behavior of automatically attaching context-scoped attributes to all signals. I'd like to see this be opt-in on a per-signal basis at minimum. Finer granularity (e.g., per-attribute-key filtering as Jack suggested) is even better, but coarser than per-signal is something we should not do. Why? See below:

Metrics: The metrics API is designed for high-performance hot-path recording. Several SDKs (.NET, Rust) accept attributes as a read-only slice/span to avoid heap allocation on every counter.Add() / histogram.Record() call. If context attributes are present and merging is the default behavior, the SDK must construct a new buffer and copy the incoming attributes on every measurement. This is a meaningful and unconditional performance hit for anyone who happens to have context attributes set — which, with default-on behavior, could easily happen without the user realizing it.

Spans: Merging attributes before ShouldSample seems unnecessary too — if the sampler drops the span, the work is thrown away. And since samplers already receive Context as a parameter, they can call GetContextScopedAttributes(context) directly if they want to act on them. No pre-merging needed.

Logs: A LogRecordProcessor that copies context-scoped attributes into log records is straightforward to write, and the spec could easily offer a built-in one. We don't provide a built-in processor to copy Baggage into log records today, and that hasn't been a major issue (Honeycomb's donation of baggage-processor to contrib repos to achieve this is what a lot of users are using).

The right model IMO: context-scoped attributes live in Context and are available to any component that wants them. Signals incorporate them only when explicitly configured to do so.

[jsuereth]

I'm hugely supportive of this OTEP! Thank you so much for making progress on it!

My own approval is blocked on two things I think we need to sort out:

• What's the default interaction with metrics. I understand the concerns folks raise, but I agree with you @carlosalberto that we SHOULD have some default mechanism of interaction here, otherwise I fear this feature is too high friction. Let's sort that out.
• What's the story for instrumentation providing context-scoped attributes? Particularly for GenAI adding an agent-id or HTTP web frameworks attaching some kind of "tenant id" would make sense to me. I do think this would be default-off, with configuration to enable, but I think we need to sort out that story.

Overall - I think this is the right thing to build, just need to tweak some interactions.

[jmacd]

I support disabled-by-default behavior!

[reyang]

I support disabled-by-default behavior!

+1

[jack-berg]

This is a good start!

I think we'll probably end up needing a config experience that is more expressive than just blanket enabled / disabled on a per-signal basis. But that's fine - we can start with broader config and collect feedback, and nothing about that stops us from evolving the config to be more expressive later (i.e. in addition to allow list / deny list style config, config could be at the scope level, or at the view level, rather than the provider level).

I agree with the emerging consensus about disabling by default. If setting context scoped attributes was limited to application authors, perhaps I could be convinced that the application owner is opting in by calling the set API. But since this will be exposed in the API, and there's nothing but convention / guidance stopping instrumentation setting it, IMO we have to disable it by default to mitigate against the excessive resource consumption / denial of service class of advisories that are far too easy to exercise if enabled by default.

For other reviewers: this has been an ongoing issue for as long as I've been involved in this project (5+ years). Please consider that this is an OTEP, and far from the final stable design, and that good design isn't necessarily perfect but is evolvable in the ways we anticipate it might need to evolve, and that perfect is the enemy of pretty darn good.

Thanks @carlosalberto.

[carlosalberto]

Updated to reflect the latest discussion cycles:

1. Disable this feature for all signals by default. Users will have to opt-in explicitly on a per-signal basis.
2. Instrumentation libraries will not be able to set Context-scoped attributes by default - however, as this is needed (see the GenAI conversation in this PR), this can be offered as an opt-in behavior. This means users will also have to explicitly enable this feature at the instrumentation library configuration.
   • Additionally, we want instrumentation libraries to document the Context-scoped attributes they want to use. This can help users decide whether or not to opt-in, e.g. don't set Context-scoped attributes for metrics in case of high cardinality values.