From b0e76d75137a4e8589912f18cf412b7a60949aa2 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 09:44:28 -0400 Subject: [PATCH 01/26] Add strawman OTEP for using entity on instrumentation-scope. --- oteps/entities/0000-scope-and-entities.md | 88 +++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 oteps/entities/0000-scope-and-entities.md diff --git a/oteps/entities/0000-scope-and-entities.md b/oteps/entities/0000-scope-and-entities.md new file mode 100644 index 00000000000..f3c15179505 --- /dev/null +++ b/oteps/entities/0000-scope-and-entities.md @@ -0,0 +1,88 @@ +# Scope and Entities + +Define the relationship between `InstrumentationScope` and `Entity`. + +## Motivation + +OpenTelemetry needs to address two fundamental problems: + +- Reporting data against "mutable" or "changing" entities, where `Resource` + is currently defined as an "immutable" identity in the data model. +- Providing true multi-tenant capabilities, where, e.g. metrics about one tenant + will be implicitly separated from metrics about another tenant. + +The first problem is well outlined in (not accepted) [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316). +Fundamentally, while we need an immutable identity, the reality is that `Resource` +in today's OpenTelemetry usage is not strong enough to support key use cases. For example, +OpenTelemetry JS, in the node.js environment, cannot guarantee that all identifying +attributes for Resource are discovered prior to SDK startup, leading to an "eventual identity" situation +that must be addressed in the Specification. Additionally, our Client/Browser SIG has been +trying to model the notion of "User Session" which has a much shorter lifespan than the SDK itself, so +requiring a single identity that is both immutable and matches the SDK lifetime prevents any good mechanism of reporting user session. + +However, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) explores +relaxing the immutability restriction vs. providing a new mechanism. During prototyping, +initially this seemed to be easily accomplished, but ran into major complications both in interactions +with OpAmp (where a stable idenitity for the SDK is desired), and in desinging a Metrics SDK, where +changes in Resource mean a dynamic and divergent storage strategy, without a priori knowledge of whether these resource mutations are +relevant to the metric or not. + +Additionally, today when reporting data from one "process" about mulitple resources, the only recourse available is to instantiate +multiple SDKs and define different resources in each SDK. This absolute separation can be highly problematic with the notion of +"built-in" instrumentation, where libraries (e.g. gRPC) come with an out-of-the-box OpenTelemetry support and it's unclear how +to ensure this instrumentation is use correctly in the presence of multiple SDKs. + +## Explanation + +We proposes these new fundamental concepts in OpenTelemetry: + +- `Resource` *remains immutable* + - Building on [OTEP 264`](0264-resource-and-entities.md), identifying attirbutes + are clearly outlined in Resource going forward, addressing unclear real world usage of Resource attributes ([e,g, identifying attributes in OpAMP](https://github.com/open-telemetry/opamp-spec/blob/main/specification.md#agentdescriptionidentifying_attributes)). + - SDK will be given an explicit initialization stage where `Resource` is not in a complete state, addressing OpenTelemetry JS concerns around async resource detection. +- `InstrumentationScope` is enhanced with `Entity` similar to `Resource`. + - When using `Get a Meter` or `Get a Tracer` operations in OpenTelemetry, we can already optionally specific a set of `Attributes`. Now + we can specify a full `Entity`. + - When an `Entity` X exists on `InstrumentationScope`, this now means "observing X in the context of the current `Resource`". For example, + on a Mobile device. An immutable `Resource` that idnetifies the device would be constructed. "Session" now becomes an Entity + attached to the `InstrumentationScope`, meaning events, spans and metrics generated are against the "sesion in the context of the device". + +## Internal details + +TODO + +## Trade-offs and mitigations + +TODO + +## Prior art and alternatives + +OpenCensus previously allowed contextual tags to be specified dynamically and used everywhere metric +measurements were reported. Users were then required to select which of these were useful to them via +the definition of "Views". OpenTelemetry has aimed for a simpler solution where every +metric has an implicit View definition, and we leverage metric advice to allow sending attributes +than is naturally used when reporting the metric. + +As called out in the description, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) +proposes making resource fully mutable, which comes with its won set of tradeoffs. + +## Open questions + +Allowing a multi-tenant agency in the OpenTelemetry SDK has a set of known issues we must resolve. + +### How do we Garbage collecting "stale" scopes + +As seen in [Issue #3062](https://github.com/open-telemetry/opentelemetry-specification/issues/3062), +systems observing multiple tenants need to ensure that tenant which are only observed briefly do not +continue to consume resources (particularly memory) for long periods of time. There needs to be +some level of control (either direct or implicit) in allowing new "Scope with Entity" to be created. + +## Prototypes + +**In Progress** + +## Future possibilities + +This proposal brings strong multi-tenant capabilities to the OpenTelemetry SDK. One possibility +is to improve the interaction between dynamic `Context` and `IntrumentationScope`, e.g. allowing +some level of interaction between `InstrumentationScope`'s entity set and `Context`. From b536e824c0033a94325122a93d604c77282e8ce5 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 09:54:26 -0400 Subject: [PATCH 02/26] Fix number to match PR. --- .../{0000-scope-and-entities.md => 4665-scope-and-entities.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename oteps/entities/{0000-scope-and-entities.md => 4665-scope-and-entities.md} (100%) diff --git a/oteps/entities/0000-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md similarity index 100% rename from oteps/entities/0000-scope-and-entities.md rename to oteps/entities/4665-scope-and-entities.md From 7cf59ad3f1650b47906a6cb84fe7b55f527978ea Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 11:30:05 -0400 Subject: [PATCH 03/26] Add more TODOs from semconv discussion. --- oteps/entities/4665-scope-and-entities.md | 95 ++++++++++++++++++++++- 1 file changed, 93 insertions(+), 2 deletions(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index f3c15179505..bdf9f18b192 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -49,11 +49,74 @@ We proposes these new fundamental concepts in OpenTelemetry: ## Internal details -TODO +TODO - introduction. + +### API Details + +Previously, every `Get A {signal reporter}` operation must accept the following +instrumentation scope parameters: + +* `name`: Specifies the name of the [instrumentation scope](../../specification/common/instrumentation-scope.md). + +* `version` (optional): Specifies the version of the instrumentation scope if + the scope has a version (e.g. a library version). Example value: 1.0.0. + +* `schema_url` (optional): Specifies the Schema URL that should be recorded in + the emitted telemetry. + +* `attributes` (optional): Specifies the instrumentation scope attributes to + associate with emitted telemetry. This API MUST be structured to accept a + variable number of attributes, including none. + + +Going forward, these operations (`Get a Logger`, `Get a Meter`, `Get a Tracer`) +will be updated to include an additional optional parameter: + +* `entities` (optional): Specifies the `Entity` set to associate with + emitted telemetry. + +Any `attributes` provided which conflict with those attributes in the provided +`entities` consitutes an error and MUST be reported by the SDK. The SDK +MAY resolve the discrepency without causing a fatal error. In this case, +the entity data model SHOULD take precedence over the `attributes` provided. + +### SDK Details + +TODO: + +- EntityDetector definition +- ResourceInitializer SDK component +- Implications on MeterProvider. + +### Protocol Details + +The OpenTelemetry Protocol (OTLP) will be modified such that +`InstrumentationScope` will support carrying `Entity` objects. There are two +ways this could be done: + +- A new `entity_refs` field on `InstrumentationScope` similar to what was + done for resource. This would avoid breaking implementations which make use + of additional attributes in `InstrumentationScope`, as this field would still + contain all new attributes for the scope. +- A new `entities` field on `InstrumentationScope` which would contain the + entire set of entities and all values. Optional attributes would be stored + separately to `entities`, and consumers of OTLP would need to know about + both fields. ## Trade-offs and mitigations -TODO +The primary trade-offs to make here are around "breaking changes" and subtle +differences in generated telemetry for code leveraging Entity vs. code which +does not. We need give room for consumers of OTLP (vendors, databases, collector) +time to support the new semantics of Entity prior to data showing up which would +not be correctly interpreted without understanding these new semantics. As such, +primarily: + +- Understanding that two `InstrumentationScope`s are different because of the + `Entity` data within them. +- Appropriately aggregating or finding data about an `Entity`. If that entity + is now reported in `InstrumentationScope`, then older systems may no longer + be able to interact with it. ## Prior art and alternatives @@ -77,6 +140,34 @@ systems observing multiple tenants need to ensure that tenant which are only obs continue to consume resources (particularly memory) for long periods of time. There needs to be some level of control (either direct or implicit) in allowing new "Scope with Entity" to be created. +### What happens when an entity exists within Resource and InstrumentationScope? + +TODO + +### How to represent Entity in InstrumentationScope? + +Should we do the same thing as `Resource` or something more direct? + +Due to mitigations and concerns around compatibility, we recommend sticking to +the same compatibility decision as `Resource` for `InstrumentationScope`. + +### Are descriptive attributes allowed to change for Resource? + +Its not clear how resource immutability is kept or what is meant by immutable. +Will the resource emitted on the first export be the same as the one emitted for +the entire lifetime of the process? Are descriptive attributes on entities +attached to resource still allowed to change? What about attaching new entities +to that resource? + +For now: + +- The set of entities reported on Resource becomes locked. + All identifying attributes are also locked. +- Whether we want to allow descriptive attributes to change - this can be + determined or evolve over time. Until the ecosystem around OTLP is leveraging + the "identity" attributes of Entity for Resource, we should not allow + mutation of descriptive attributes. + ## Prototypes **In Progress** From 46715626613930ebb7730a0f03bdf3a25c9d29ee Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 14:12:13 -0400 Subject: [PATCH 04/26] Add more TODO sections. --- oteps/entities/4665-scope-and-entities.md | 153 +++++++++++++++++++++- 1 file changed, 148 insertions(+), 5 deletions(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index bdf9f18b192..b8314c63a70 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -82,11 +82,134 @@ the entity data model SHOULD take precedence over the `attributes` provided. ### SDK Details -TODO: +The SDK is updated to, explicitly, include three new components: -- EntityDetector definition -- ResourceInitializer SDK component -- Implications on MeterProvider. +- `ResourceInitializer`: A formalized component that is responsible for + determining the `Resource` an SDK will use. +- `ResourceListener`: A formalized component for notifying the rest of the + SDK that resource initialization has completed. +- `Resource Detector`: A formalized component that is responsible for + detecting `entity`s on the initial `Resource`. + +Additionally the following key changes are made to the SDK: + +- All `{Tracer/Meter/Logger}Provider` components are updated to use the + `ResourceInitializer` instead of directly accepting a `Resource`. + - This MAY be done via the `ResourceListener` component defined. + - This MAY be done via delaying SDK initializtion until a `Resource` + is available on `ResourceInitializer` + - The default SDK constructors MUST use (or recommend) `ResourceInitializer`. +- `MeterProvider` MUST treat entity found on InstrumentationScope as identifying, + an aggregate reported events separately by scope. Note: this is the case in + the specification today, however many implementations do not yet respect + InstrumentationScope loose attributes. + +#### ResourceListener + +The `ResourceListener` MUST provide the following operations in the SDK: + +* `On Resource Initialize` + +##### On ResourceInitialize + +`On ResourceInitialize` MUST accept the following parameters: + +* `Resource`: represents the entire set of resources after initialize. +* `status`: represents the status of resource initialization. + +This operation MUST only be called once per ResourceInitializer. + +#### ResourceInitializer + +The `ResourceInitializer` MUST provide the following operations in the SDK: + +* `On Change` +* (optional) `GetResource` + + +ResourceInitializer has two states: + +- *DETECTING*: The provider is waiting for ResourceDetectors to complete. +- *INITALIZED*: A complete resource is available and ResourceListeners will + be notified of the final resource. + +##### ResourceInitializer creation + +Creation of a ResourceInitializer MUST accept the following parameters: + +* `detectors`: a list of ResourceDetectors. +* (optional) `initialization_timeout`: A timeout for when to abandon slow + ResourceDetectors and consider a resource initialized. + +ResourceInitializer MUST allow initial resource detection during its creation. This +initialization SHOULD not block other SDK providers from initializing (e.g. +MeterProvider, TracerProvider). + +A ResourceInitializer MAY allow customizable concurrency behavior, e.g. using a +separate thread for ResourceListener events. + +Internally, the entities discovered via resource detection MUST be merged in +the order provided to create the initial resource. + +During resource detection, ResourceInitializer MUST NOT fire any ResourceListener +events. + +Upon completion of the resource detection phase, ResourceInitializer MUST fire +an `On ResourceInitialize` event to all ResourceListeners. + +Any calls to `GetResource` operation, if provided, SHOULD block until +resource detection is completed. + +Upon failure for resource detection to complete within a timeout, a resource +SHOULD be constructed with available completed detection, `GetResource` +jsuereth marked this conversation as resolved. +operations MUST be unblocked and `On ResourceInitialize` event MUST be fired +to all ResourceListeners. This call MUST provide a failure status. + +##### On Change + +`On Change` registers an `ResourceListener` to be called when a Resource has +been detected or detection has failed. + +If the `EntityProvider` is already initialized, then it MUST call +`On ResourceInitialize` immediately with the current resource held by the +`ResourceInitializer`. + +##### Get Resource + +`Get Resource` MUST return a reference to the current resource held by the +ResourceInitializer. + +This operation MAY block when ResourceInitializer is in a DETECTING, +but MUST NOT block indefinitely. + + +#### ResourceDetector + +A `ResourceDetector` concept is updated. + +ResourceDetector MUST provide the following operations: + +- `Detect entities` + +##### Detect Entities operation + +The Detect Entities Operation SHOULD allow asynchronous or long-running +exeuction, e.g. issuing an HTTP request to a local service for identity, reading +from disk, etc. + +The Detect Entities Operation SHOULD return a status to indicate completion or +failure. + +The Detect Entities Operation MUST have a mechanism to report detected `Entity` +to the ResourceInitializer. This MUST include the following: + +- The `type` of the Entity +- (optional) The `schema_url` of the Entity +- The `id` of the Entity (as a set of attributes). +- (optional) The `description` of the Entity (as a set of attributes). + +The Detect Entities operation MAY be asynchronous. ### Protocol Details @@ -129,11 +252,19 @@ than is naturally used when reporting the metric. As called out in the description, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) proposes making resource fully mutable, which comes with its won set of tradeoffs. +Today, Semantic Conventions already defined `Entity` is uses it to group and +report `Resource` attributes cohesively. Additionally, Semantic convention only +models "entity associations", that is requiring a signal (e.g. a metric, event +or span) to be attached to an entity. For example, the `system.cpu.time` metric +is expected to be associated with a `host` entity. This association makes no +assumption about whether that is through `Resource` or some other mechanism, +and can therefore be extended to support `InstrumentationScope` based entities. + ## Open questions Allowing a multi-tenant agency in the OpenTelemetry SDK has a set of known issues we must resolve. -### How do we Garbage collecting "stale" scopes +### What are the SDK safeguards against high-cardinality Entities? As seen in [Issue #3062](https://github.com/open-telemetry/opentelemetry-specification/issues/3062), systems observing multiple tenants need to ensure that tenant which are only observed briefly do not @@ -168,6 +299,18 @@ For now: the "identity" attributes of Entity for Resource, we should not allow mutation of descriptive attributes. +### What is the expected impact on Collector components? + +How should standard Collector processors (like the attributes processor, +resource processor, or filter processor) be updated to interact with Entities +within the InstrumentationScope? + +### How do we guide developers on when to use InstrumenationScope vs. Resource? + +The distinction between an immutable Resource entity and a "mutable" Scope +entity is subtle. The Resource begins to take on the identity of the SDK itself +while Scope allows "refinement" of that identity, within some bound. + ## Prototypes **In Progress** From 18403e84358d103ce75f33c3b4393c3639c9c310 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 15:38:53 -0400 Subject: [PATCH 05/26] Fix lint. --- oteps/entities/4665-scope-and-entities.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index b8314c63a70..4015c0f82cd 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -30,7 +30,7 @@ relevant to the metric or not. Additionally, today when reporting data from one "process" about mulitple resources, the only recourse available is to instantiate multiple SDKs and define different resources in each SDK. This absolute separation can be highly problematic with the notion of "built-in" instrumentation, where libraries (e.g. gRPC) come with an out-of-the-box OpenTelemetry support and it's unclear how -to ensure this instrumentation is use correctly in the presence of multiple SDKs. +to ensure this instrumentation is use correctly in the presence of multiple SDKs. ## Explanation @@ -102,7 +102,7 @@ Additionally the following key changes are made to the SDK: - `MeterProvider` MUST treat entity found on InstrumentationScope as identifying, an aggregate reported events separately by scope. Note: this is the case in the specification today, however many implementations do not yet respect - InstrumentationScope loose attributes. + InstrumentationScope loose #### ResourceListener @@ -126,7 +126,6 @@ The `ResourceInitializer` MUST provide the following operations in the SDK: * `On Change` * (optional) `GetResource` - ResourceInitializer has two states: - *DETECTING*: The provider is waiting for ResourceDetectors to complete. @@ -183,7 +182,6 @@ ResourceInitializer. This operation MAY block when ResourceInitializer is in a DETECTING, but MUST NOT block indefinitely. - #### ResourceDetector A `ResourceDetector` concept is updated. @@ -204,8 +202,8 @@ failure. The Detect Entities Operation MUST have a mechanism to report detected `Entity` to the ResourceInitializer. This MUST include the following: -- The `type` of the Entity -- (optional) The `schema_url` of the Entity +- The `type` of the Entity. +- (optional) The `schema_url` of the Entity. - The `id` of the Entity (as a set of attributes). - (optional) The `description` of the Entity (as a set of attributes). @@ -292,7 +290,7 @@ to that resource? For now: -- The set of entities reported on Resource becomes locked. +- The set of entities reported on Resource becomes locked. All identifying attributes are also locked. - Whether we want to allow descriptive attributes to change - this can be determined or evolve over time. Until the ecosystem around OTLP is leveraging @@ -313,7 +311,7 @@ while Scope allows "refinement" of that identity, within some bound. ## Prototypes -**In Progress** +TODO: **In Progress** ## Future possibilities From 6330b97884b2007b5808a6de5e868a9d260444ff Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 15:49:54 -0400 Subject: [PATCH 06/26] Fixes from review. --- oteps/entities/4665-scope-and-entities.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 4015c0f82cd..bcb38cb0956 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -37,14 +37,14 @@ to ensure this instrumentation is use correctly in the presence of multiple SDKs We proposes these new fundamental concepts in OpenTelemetry: - `Resource` *remains immutable* - - Building on [OTEP 264`](0264-resource-and-entities.md), identifying attirbutes + - Building on [OTEP 264](0264-resource-and-entities.md), identifying attributes are clearly outlined in Resource going forward, addressing unclear real world usage of Resource attributes ([e,g, identifying attributes in OpAMP](https://github.com/open-telemetry/opamp-spec/blob/main/specification.md#agentdescriptionidentifying_attributes)). - SDK will be given an explicit initialization stage where `Resource` is not in a complete state, addressing OpenTelemetry JS concerns around async resource detection. - `InstrumentationScope` is enhanced with `Entity` similar to `Resource`. - - When using `Get a Meter` or `Get a Tracer` operations in OpenTelemetry, we can already optionally specific a set of `Attributes`. Now + - When using `Get a Meter` or `Get a Tracer` operations in OpenTelemetry, we could already optionally specify a set of `Attribute`s. Now we can specify a full `Entity`. - When an `Entity` X exists on `InstrumentationScope`, this now means "observing X in the context of the current `Resource`". For example, - on a Mobile device. An immutable `Resource` that idnetifies the device would be constructed. "Session" now becomes an Entity + on a Mobile device, an immutable `Resource` that identifies the device would be constructed. "Session" now becomes an Entity attached to the `InstrumentationScope`, meaning events, spans and metrics generated are against the "sesion in the context of the device". ## Internal details From 835001e399626b356209e44c60c467d1f006f614 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 15:50:46 -0400 Subject: [PATCH 07/26] spellcheck. --- oteps/entities/4665-scope-and-entities.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index bcb38cb0956..de8e62206e3 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -23,11 +23,11 @@ requiring a single identity that is both immutable and matches the SDK lifetime However, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) explores relaxing the immutability restriction vs. providing a new mechanism. During prototyping, initially this seemed to be easily accomplished, but ran into major complications both in interactions -with OpAmp (where a stable idenitity for the SDK is desired), and in desinging a Metrics SDK, where +with OpAmp (where a stable idenitity for the SDK is desired), and in designing a Metrics SDK, where changes in Resource mean a dynamic and divergent storage strategy, without a priori knowledge of whether these resource mutations are relevant to the metric or not. -Additionally, today when reporting data from one "process" about mulitple resources, the only recourse available is to instantiate +Additionally, today when reporting data from one "process" about multiple resources, the only recourse available is to instantiate multiple SDKs and define different resources in each SDK. This absolute separation can be highly problematic with the notion of "built-in" instrumentation, where libraries (e.g. gRPC) come with an out-of-the-box OpenTelemetry support and it's unclear how to ensure this instrumentation is use correctly in the presence of multiple SDKs. @@ -129,7 +129,7 @@ The `ResourceInitializer` MUST provide the following operations in the SDK: ResourceInitializer has two states: - *DETECTING*: The provider is waiting for ResourceDetectors to complete. -- *INITALIZED*: A complete resource is available and ResourceListeners will +- *INITIALIZED*: A complete resource is available and ResourceListeners will be notified of the final resource. ##### ResourceInitializer creation From 4173b73790fad1e8ac8d2c0ee5c040456c2f6a77 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 15:53:07 -0400 Subject: [PATCH 08/26] Fix opening sentence. --- oteps/entities/4665-scope-and-entities.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index de8e62206e3..6db48f8af0c 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -260,7 +260,8 @@ and can therefore be extended to support `InstrumentationScope` based entities. ## Open questions -Allowing a multi-tenant agency in the OpenTelemetry SDK has a set of known issues we must resolve. +Adding entity in InstrumentationScope has a lot of implications that must be +resolved. ### What are the SDK safeguards against high-cardinality Entities? From 1121b2f9de688a1737645d4d41e8c3122f112bd7 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 17:17:00 -0400 Subject: [PATCH 09/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Christophe Kamphaus --- oteps/entities/4665-scope-and-entities.md | 1 - 1 file changed, 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 6db48f8af0c..516e784cd3d 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -161,7 +161,6 @@ resource detection is completed. Upon failure for resource detection to complete within a timeout, a resource SHOULD be constructed with available completed detection, `GetResource` -jsuereth marked this conversation as resolved. operations MUST be unblocked and `On ResourceInitialize` event MUST be fired to all ResourceListeners. This call MUST provide a failure status. From 1a81e01b69354a5c9c3ceb2afee7bb95630335e8 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 17:17:10 -0400 Subject: [PATCH 10/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Christophe Kamphaus --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 516e784cd3d..9859fb09f65 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -23,7 +23,7 @@ requiring a single identity that is both immutable and matches the SDK lifetime However, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) explores relaxing the immutability restriction vs. providing a new mechanism. During prototyping, initially this seemed to be easily accomplished, but ran into major complications both in interactions -with OpAmp (where a stable idenitity for the SDK is desired), and in designing a Metrics SDK, where +with OpAmp (where a stable identity for the SDK is desired), and in designing a Metrics SDK, where changes in Resource mean a dynamic and divergent storage strategy, without a priori knowledge of whether these resource mutations are relevant to the metric or not. From f5d58a2cc46ace7c455dea077280906e49e0ba2e Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 17:17:18 -0400 Subject: [PATCH 11/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Christophe Kamphaus --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 9859fb09f65..67b556280ce 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -166,7 +166,7 @@ to all ResourceListeners. This call MUST provide a failure status. ##### On Change -`On Change` registers an `ResourceListener` to be called when a Resource has +`On Change` registers a `ResourceListener` to be called when a Resource has been detected or detection has failed. If the `EntityProvider` is already initialized, then it MUST call From 2f77f86b99e9bca280efb8ddaf467cf12d0b3f64 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 17:17:40 -0400 Subject: [PATCH 12/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Christophe Kamphaus --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 67b556280ce..86a0f32fdb3 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -247,7 +247,7 @@ metric has an implicit View definition, and we leverage metric advice to allow s than is naturally used when reporting the metric. As called out in the description, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) -proposes making resource fully mutable, which comes with its won set of tradeoffs. +proposes making resource fully mutable, which comes with its own set of tradeoffs. Today, Semantic Conventions already defined `Entity` is uses it to group and report `Resource` attributes cohesively. Additionally, Semantic convention only From 9630ce9c98749d2271c955a55d68e1b53f30318b Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 17:17:48 -0400 Subject: [PATCH 13/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Christophe Kamphaus --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 86a0f32fdb3..2339c145dba 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -265,7 +265,7 @@ resolved. ### What are the SDK safeguards against high-cardinality Entities? As seen in [Issue #3062](https://github.com/open-telemetry/opentelemetry-specification/issues/3062), -systems observing multiple tenants need to ensure that tenant which are only observed briefly do not +systems observing multiple tenants need to ensure that tenants which are only observed briefly do not continue to consume resources (particularly memory) for long periods of time. There needs to be some level of control (either direct or implicit) in allowing new "Scope with Entity" to be created. From 76860ba2b1f949258b8f8a97581a60a71a2be329 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 22 Sep 2025 17:18:02 -0400 Subject: [PATCH 14/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Christophe Kamphaus --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 2339c145dba..9ae56c72ba8 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -249,7 +249,7 @@ than is naturally used when reporting the metric. As called out in the description, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) proposes making resource fully mutable, which comes with its own set of tradeoffs. -Today, Semantic Conventions already defined `Entity` is uses it to group and +Today, Semantic Conventions already defines `Entity` and uses it to group and report `Resource` attributes cohesively. Additionally, Semantic convention only models "entity associations", that is requiring a signal (e.g. a metric, event or span) to be attached to an entity. For example, the `system.cpu.time` metric From bb38a2694e490e9d42517f746647406d18b3c6fc Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Tue, 23 Sep 2025 08:00:04 -0400 Subject: [PATCH 15/26] Finish remaining TODOs. --- oteps/entities/4665-scope-and-entities.md | 29 ++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 9ae56c72ba8..2a183b8adc9 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -80,6 +80,26 @@ Any `attributes` provided which conflict with those attributes in the provided MAY resolve the discrepency without causing a fatal error. In this case, the entity data model SHOULD take precedence over the `attributes` provided. +The set of `Entity` provided to these operations MUST only include one `Entity` +per `type`. + +#### Entity + +An `Entity` is a collection of the following values: + +- `type`: A string describing the class of the Entity. +- `id`: An attribute collection that identifies an instance of the Entity. +- (optional) `description`: An attribute collection that describes the instance + of the Entity. +- (optional) `schema_url`: Specifies the Schema URL that should be recorded for + this Entity. + +An `Entity` is uniquely identified by the combination of its `type` and `id`. + +`schema_url` defines version of the schema used to describe an `Entity`. If +two entities exist with the same identity and different `schema_url`s, they +MUST be considered in conflict with each other. + ### SDK Details The SDK is updated to, explicitly, include three new components: @@ -271,7 +291,14 @@ some level of control (either direct or implicit) in allowing new "Scope with En ### What happens when an entity exists within Resource and InstrumentationScope? -TODO +Should we consider this a failure or a feature? + +For simplicity in modeling, we plan to prototype where this is *disallowed* +until proven we need this, in which case we need crisp language about +what this models exactly. + +For now, only ONE entity of a given type will be allowed across both +`Resource` and `InstrumentationScope` within the SDK. ### How to represent Entity in InstrumentationScope? From 944d19e7bc8beada82e583184b9ad6b8ee5e5339 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Tue, 23 Sep 2025 09:25:09 -0400 Subject: [PATCH 16/26] Fix lint. --- oteps/entities/4665-scope-and-entities.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 2a183b8adc9..7492fe6d0ab 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -68,11 +68,10 @@ instrumentation scope parameters: associate with emitted telemetry. This API MUST be structured to accept a variable number of attributes, including none. - Going forward, these operations (`Get a Logger`, `Get a Meter`, `Get a Tracer`) will be updated to include an additional optional parameter: -* `entities` (optional): Specifies the `Entity` set to associate with +* `entities` (optional): Specifies the `Entity` set to associate with emitted telemetry. Any `attributes` provided which conflict with those attributes in the provided @@ -260,7 +259,7 @@ primarily: ## Prior art and alternatives -OpenCensus previously allowed contextual tags to be specified dynamically and used everywhere metric +OpenCensus previously allowed contextual tags to be specified dynamically and used everywhere metric measurements were reported. Users were then required to select which of these were useful to them via the definition of "Views". OpenTelemetry has aimed for a simpler solution where every metric has an implicit View definition, and we leverage metric advice to allow sending attributes From 28720b35f00c2af6dc764ed6204a8febec92e36f Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Wed, 24 Sep 2025 14:59:55 -0400 Subject: [PATCH 17/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Nathan L Smith --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 7492fe6d0ab..74fef30b0d2 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -269,7 +269,7 @@ As called out in the description, [OTEP 4316](https://github.com/open-telemetry/ proposes making resource fully mutable, which comes with its own set of tradeoffs. Today, Semantic Conventions already defines `Entity` and uses it to group and -report `Resource` attributes cohesively. Additionally, Semantic convention only +report `Resource` attributes cohesively. Additionally, Semantic Conventions only models "entity associations", that is requiring a signal (e.g. a metric, event or span) to be attached to an entity. For example, the `system.cpu.time` metric is expected to be associated with a `host` entity. This association makes no From 35f5cec4df371f8eafd2caf35b3f93df620aba1b Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Wed, 24 Sep 2025 15:00:09 -0400 Subject: [PATCH 18/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Nathan L Smith --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 74fef30b0d2..3d73cf08ed6 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -197,7 +197,7 @@ If the `EntityProvider` is already initialized, then it MUST call `Get Resource` MUST return a reference to the current resource held by the ResourceInitializer. -This operation MAY block when ResourceInitializer is in a DETECTING, +This operation MAY block when ResourceInitializer is in a DETECTING state, but MUST NOT block indefinitely. #### ResourceDetector From afb3d0344c4eb945d858aa5d8d553670fee468aa Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Wed, 24 Sep 2025 15:00:56 -0400 Subject: [PATCH 19/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Nathan L Smith --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 3d73cf08ed6..106401697b8 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -119,7 +119,7 @@ Additionally the following key changes are made to the SDK: is available on `ResourceInitializer` - The default SDK constructors MUST use (or recommend) `ResourceInitializer`. - `MeterProvider` MUST treat entity found on InstrumentationScope as identifying, - an aggregate reported events separately by scope. Note: this is the case in + and aggregate reported events separately by scope. Note: this is the case in the specification today, however many implementations do not yet respect InstrumentationScope loose From 379f49df06f7c2bc5da649900003ae165873b20d Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Wed, 24 Sep 2025 15:01:06 -0400 Subject: [PATCH 20/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Nathan L Smith --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index 106401697b8..f8ab6fe3a8f 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -118,7 +118,7 @@ Additionally the following key changes are made to the SDK: - This MAY be done via delaying SDK initializtion until a `Resource` is available on `ResourceInitializer` - The default SDK constructors MUST use (or recommend) `ResourceInitializer`. -- `MeterProvider` MUST treat entity found on InstrumentationScope as identifying, +- `MeterProvider` MUST treat entities found on InstrumentationScope as identifying, and aggregate reported events separately by scope. Note: this is the case in the specification today, however many implementations do not yet respect InstrumentationScope loose From de53cb00ec29135c7db42f05ad8c806280a21349 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Wed, 24 Sep 2025 15:01:18 -0400 Subject: [PATCH 21/26] Update oteps/entities/4665-scope-and-entities.md Co-authored-by: Nathan L Smith --- oteps/entities/4665-scope-and-entities.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md index f8ab6fe3a8f..774f6cb7ed9 100644 --- a/oteps/entities/4665-scope-and-entities.md +++ b/oteps/entities/4665-scope-and-entities.md @@ -107,7 +107,7 @@ The SDK is updated to, explicitly, include three new components: determining the `Resource` an SDK will use. - `ResourceListener`: A formalized component for notifying the rest of the SDK that resource initialization has completed. -- `Resource Detector`: A formalized component that is responsible for +- `ResourceDetector`: A formalized component that is responsible for detecting `entity`s on the initial `Resource`. Additionally the following key changes are made to the SDK: From 7f696f7cdcf7c5ec342c68f0c76d83fb99291e2a Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 20 Oct 2025 10:36:24 -0400 Subject: [PATCH 22/26] merge conflcits. --- CHANGELOG.md | 3 + .../entities/4665-multiple-resource-in-sdk.md | 206 +++++++++++ oteps/entities/4665-scope-and-entities.md | 346 ------------------ 3 files changed, 209 insertions(+), 346 deletions(-) create mode 100644 oteps/entities/4665-multiple-resource-in-sdk.md delete mode 100644 oteps/entities/4665-scope-and-entities.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a19a6fcce5d..487c86afbaf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,9 @@ release. ### OTEPs +- Support multiple Resources within an SDK. + ([#4665](https://github.com/open-telemetry/opentelemetry-specification/pull/4665)) + ## v1.52.0 (2025-12-12) ### Context diff --git a/oteps/entities/4665-multiple-resource-in-sdk.md b/oteps/entities/4665-multiple-resource-in-sdk.md new file mode 100644 index 00000000000..33bad89db7d --- /dev/null +++ b/oteps/entities/4665-multiple-resource-in-sdk.md @@ -0,0 +1,206 @@ +# Multiple Resource in SDK + +Allow multiple `Resource` instances per SDK. + +## Motivation + +OpenTelemetry needs to address two fundamental problems: + +- Reporting data against "mutable" or "changing" entities, where currently an + SDK is allowed a single `Resource`, whose lifetime must match the lifetime of + the SDK itself. +- Providing true multi-tenant capabilities, where, e.g. metrics about one tenant + will be implicitly separated from metrics about another tenant. + +The first problem is well outlined in (not accepted) [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316). +Fundamentally, while we need an immutable identity, the reality is that `Resource` +in today's OpenTelemetry usage is not strong enough to support key use cases. For example, +OpenTelemetry JS, in the node.js environment, cannot guarantee that all identifying +attributes for Resource are discovered prior to SDK startup, leading to an "eventual identity" situation +that must be addressed in the Specification. Additionally, our Client/Browser SIG has been +trying to model the notion of "User Session" which has a much shorter lifespan than the SDK itself, so +requiring a single identity that is both immutable and matches the SDK lifetime prevents any good mechanism of reporting user session. + +However, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) explores +relaxing the immutability restriction vs. providing a new mechanism. During prototyping, +initially this seemed to be easily accomplished, but ran into major complications both in interactions +with OpAmp (where a stable identity for the SDK is desired), and in designing a Metrics SDK, where +changes in Resource mean a dynamic and divergent storage strategy, without a priori knowledge of whether these resource mutations are +relevant to the metric or not. + +Additionally, today when reporting data from one "process" about multiple resources, the only recourse available is to instantiate +multiple SDKs and define different resources in each SDK. This absolute separation can be highly problematic with the notion of +"built-in" instrumentation, where libraries (e.g. gRPC) come with an out-of-the-box OpenTelemetry support and it's unclear how +to ensure this instrumentation is use correctly in the presence of multiple SDKs. + +## Explanation + +We proposes these new fundamental concepts in OpenTelemetry: + +- `Resource` *remains immutable* + - Building on [OTEP 264](0264-resource-and-entities.md), identifying attributes + are clearly outlined in Resource going forward, addressing unclear real world usage of Resource attributes ([e,g, identifying attributes in OpAMP](https://github.com/open-telemetry/opamp-spec/blob/main/specification.md#agentdescriptionidentifying_attributes)). + - SDK will be given an explicit initialization stage where `Resource` is not in a complete state, addressing OpenTelemetry JS concerns around async resource detection. +- The SDK will be identified by a single `Resource` provided during SDK startup. + - ResourceDetection will be expanded, as described in [OTEP 264](0264-resource-and-entities.md). + - An explicit section about SDK initialization will be created. +- Signal Providers in the SDK will allow "specialization" of the default SDK + resource. We create new `{Signal}Provider` instances by providing a new + `Entity` on the existing provider. + - This will construct a new `Resource` specific to that provider. + - The new provider will re-use all configuraiton (e.g. export pipeline) + defined from the base provider. + +## Internal details + +TODO - introduction. + +### API Details + +Previously, every `{Signal}Provider` API defined a single +`Get a {Signal}` operation. These will be expanded with a new +`For Entity` operation, which will construct a new `{SignalProvider}` +API component for reporting against a specific `Entity`. + +#### For Entity + +This API MUST accept the following parameters: + +* `entities`: Specifies the `Entity` set to associate with + emitted telemetry. + +Any `entities` provided which conflict with those entities already provided in +the SDK `Resource` represent an *override* of identity. The SDK MUST resolve the +conflict without causing a fatal error. + +The set of `Entity` provided to these operations MUST only include one `Entity` +per `type`. + +#### Entity + +An `Entity` is a collection of the following values: + +- `type`: A string describing the class of the Entity. +- `id`: An attribute collection that identifies an instance of the Entity. +- (optional) `description`: An attribute collection that describes the instance + of the Entity. +- (optional) `schema_url`: Specifies the Schema URL that should be recorded for + this Entity. + +An `Entity` is uniquely identified by the combination of its `type` and `id`. + +`schema_url` defines version of the schema used to describe an `Entity`. If +two entities exist with the same identity and different `schema_url`s, they +MUST be considered in conflict with each other. + +### SDK Details + +TODO + +## Trade-offs and mitigations + +The primary trade-offs to make here are around "breaking changes" and subtle +differences in generated telemetry for code leveraging Entity vs. code which +does not. We need give room for consumers of OTLP (vendors, databases, collector) +time to support the new semantics of Entity prior to data showing up which would +not be correctly interpreted without understanding these new semantics. As such, +primarily: + +- `Entity`, as defined in OTLP, is an opt-in construct. `Resource` should be + usable as an identity independent of `Entity`. +- Consumers should now expect SDKs reporting multiple resources in the same + batch. Theoretically, this SHOULD already be supported due to how OTLP is + deisgned to allow aggregation / batching of data at any point. + +## Prior art and alternatives + +OpenCensus previously allowed contextual tags to be specified dynamically and +used everywhere metric measurements were reported. Users were then required to +select which of these were useful to them via the definition of "Views". +OpenTelemetry has aimed for a simpler solution where every metric has an +implicit View definition, and we leverage metric advice to allow sending +attributes than is naturally used when reporting the metric. + +As called out in the description, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) +proposes making resource fully mutable, which comes with its own set of tradeoffs. + +Today, Semantic Conventions already defines `Entity` and uses it to group and +report `Resource` attributes cohesively. Additionally, Semantic convention only +models "entity associations", that is requiring a signal (e.g. a metric, event +or span) to be attached to an entity. For example, the `system.cpu.time` metric +is expected to be associated with a `host` entity. This association makes no +assumption about whether that is through `Resource` or some other mechanism, +and can therefore be extended to support `InstrumentationScope` based entities. + +## Open questions + +Adding entity in InstrumentationScope has a lot of implications that must be +resolved. + +### What are the SDK safeguards against high-cardinality Entities? + +As seen in [Issue #3062](https://github.com/open-telemetry/opentelemetry-specification/issues/3062), +systems observing multiple tenants need to ensure that tenants which are only observed briefly do not +continue to consume resources (particularly memory) for long periods of time. There needs to be +some level of control (either direct or implicit) in allowing new "Scope with Entity" to be created. + +### What happens when an entity already exists within Resource? + +Should we consider this a failure or a feature? + +We currently consider this a feature. Upon conflict, the *new* Entity would be +used in the resulting `Resource` reported for a new `{SignalProvider}`. + +The SDK needs some form of stable identity for itself, however when reporting +Telemetry, it may be recording data on behalf of some other system. + +### Are descriptive attributes allowed to change for Resource? + +Its not clear how resource immutability is kept or what is meant by immutable. +Will the resource emitted on the first export be the same as the one emitted for +the entire lifetime of the process? Are descriptive attributes on entities +attached to resource still allowed to change? What about attaching new entities +to that resource? + +For now: + +- The set of entities reported on Resource becomes locked. + All identifying attributes are also locked. +- Whether we want to allow descriptive attributes to change - this can be + determined or evolve over time. Until the ecosystem around OTLP is leveraging + the "identity" attributes of Entity for Resource, we should not allow + mutation of descriptive attributes. + +### What is the expected impact on Collector components? + +There should be *no* impact on collector components beyond those defined in +[OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316). + +### How do we guide developers on when to use `For Entity`? + +We will have clear guidance on the `For Entity` methods + +## Prototypes + +TODO: **In Progress** + +## Future possibilities + +This proposal brings strong multi-tenant capabilities to the OpenTelemetry SDK. One possibility +is to improve the interaction between dynamic `Context` and signals, e.g. allowing +some level of interaction `Context` and attributes / entities. + +For example, rather than a lexical scope: + +```js +const myMeterProvider = globalMeterProvider.forEntity(getCurrentSession()) +doSomething(myMeterProvider) +``` + +We could allow runtime scope: + +```js +const ctx = api.context.active(); +api.context.with(ctx.setValue(CURRENT_ENTITY_KEY, getCurrentSession())) +doSomething() +``` diff --git a/oteps/entities/4665-scope-and-entities.md b/oteps/entities/4665-scope-and-entities.md deleted file mode 100644 index 774f6cb7ed9..00000000000 --- a/oteps/entities/4665-scope-and-entities.md +++ /dev/null @@ -1,346 +0,0 @@ -# Scope and Entities - -Define the relationship between `InstrumentationScope` and `Entity`. - -## Motivation - -OpenTelemetry needs to address two fundamental problems: - -- Reporting data against "mutable" or "changing" entities, where `Resource` - is currently defined as an "immutable" identity in the data model. -- Providing true multi-tenant capabilities, where, e.g. metrics about one tenant - will be implicitly separated from metrics about another tenant. - -The first problem is well outlined in (not accepted) [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316). -Fundamentally, while we need an immutable identity, the reality is that `Resource` -in today's OpenTelemetry usage is not strong enough to support key use cases. For example, -OpenTelemetry JS, in the node.js environment, cannot guarantee that all identifying -attributes for Resource are discovered prior to SDK startup, leading to an "eventual identity" situation -that must be addressed in the Specification. Additionally, our Client/Browser SIG has been -trying to model the notion of "User Session" which has a much shorter lifespan than the SDK itself, so -requiring a single identity that is both immutable and matches the SDK lifetime prevents any good mechanism of reporting user session. - -However, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) explores -relaxing the immutability restriction vs. providing a new mechanism. During prototyping, -initially this seemed to be easily accomplished, but ran into major complications both in interactions -with OpAmp (where a stable identity for the SDK is desired), and in designing a Metrics SDK, where -changes in Resource mean a dynamic and divergent storage strategy, without a priori knowledge of whether these resource mutations are -relevant to the metric or not. - -Additionally, today when reporting data from one "process" about multiple resources, the only recourse available is to instantiate -multiple SDKs and define different resources in each SDK. This absolute separation can be highly problematic with the notion of -"built-in" instrumentation, where libraries (e.g. gRPC) come with an out-of-the-box OpenTelemetry support and it's unclear how -to ensure this instrumentation is use correctly in the presence of multiple SDKs. - -## Explanation - -We proposes these new fundamental concepts in OpenTelemetry: - -- `Resource` *remains immutable* - - Building on [OTEP 264](0264-resource-and-entities.md), identifying attributes - are clearly outlined in Resource going forward, addressing unclear real world usage of Resource attributes ([e,g, identifying attributes in OpAMP](https://github.com/open-telemetry/opamp-spec/blob/main/specification.md#agentdescriptionidentifying_attributes)). - - SDK will be given an explicit initialization stage where `Resource` is not in a complete state, addressing OpenTelemetry JS concerns around async resource detection. -- `InstrumentationScope` is enhanced with `Entity` similar to `Resource`. - - When using `Get a Meter` or `Get a Tracer` operations in OpenTelemetry, we could already optionally specify a set of `Attribute`s. Now - we can specify a full `Entity`. - - When an `Entity` X exists on `InstrumentationScope`, this now means "observing X in the context of the current `Resource`". For example, - on a Mobile device, an immutable `Resource` that identifies the device would be constructed. "Session" now becomes an Entity - attached to the `InstrumentationScope`, meaning events, spans and metrics generated are against the "sesion in the context of the device". - -## Internal details - -TODO - introduction. - -### API Details - -Previously, every `Get A {signal reporter}` operation must accept the following -instrumentation scope parameters: - -* `name`: Specifies the name of the [instrumentation scope](../../specification/common/instrumentation-scope.md). - -* `version` (optional): Specifies the version of the instrumentation scope if - the scope has a version (e.g. a library version). Example value: 1.0.0. - -* `schema_url` (optional): Specifies the Schema URL that should be recorded in - the emitted telemetry. - -* `attributes` (optional): Specifies the instrumentation scope attributes to - associate with emitted telemetry. This API MUST be structured to accept a - variable number of attributes, including none. - -Going forward, these operations (`Get a Logger`, `Get a Meter`, `Get a Tracer`) -will be updated to include an additional optional parameter: - -* `entities` (optional): Specifies the `Entity` set to associate with - emitted telemetry. - -Any `attributes` provided which conflict with those attributes in the provided -`entities` consitutes an error and MUST be reported by the SDK. The SDK -MAY resolve the discrepency without causing a fatal error. In this case, -the entity data model SHOULD take precedence over the `attributes` provided. - -The set of `Entity` provided to these operations MUST only include one `Entity` -per `type`. - -#### Entity - -An `Entity` is a collection of the following values: - -- `type`: A string describing the class of the Entity. -- `id`: An attribute collection that identifies an instance of the Entity. -- (optional) `description`: An attribute collection that describes the instance - of the Entity. -- (optional) `schema_url`: Specifies the Schema URL that should be recorded for - this Entity. - -An `Entity` is uniquely identified by the combination of its `type` and `id`. - -`schema_url` defines version of the schema used to describe an `Entity`. If -two entities exist with the same identity and different `schema_url`s, they -MUST be considered in conflict with each other. - -### SDK Details - -The SDK is updated to, explicitly, include three new components: - -- `ResourceInitializer`: A formalized component that is responsible for - determining the `Resource` an SDK will use. -- `ResourceListener`: A formalized component for notifying the rest of the - SDK that resource initialization has completed. -- `ResourceDetector`: A formalized component that is responsible for - detecting `entity`s on the initial `Resource`. - -Additionally the following key changes are made to the SDK: - -- All `{Tracer/Meter/Logger}Provider` components are updated to use the - `ResourceInitializer` instead of directly accepting a `Resource`. - - This MAY be done via the `ResourceListener` component defined. - - This MAY be done via delaying SDK initializtion until a `Resource` - is available on `ResourceInitializer` - - The default SDK constructors MUST use (or recommend) `ResourceInitializer`. -- `MeterProvider` MUST treat entities found on InstrumentationScope as identifying, - and aggregate reported events separately by scope. Note: this is the case in - the specification today, however many implementations do not yet respect - InstrumentationScope loose - -#### ResourceListener - -The `ResourceListener` MUST provide the following operations in the SDK: - -* `On Resource Initialize` - -##### On ResourceInitialize - -`On ResourceInitialize` MUST accept the following parameters: - -* `Resource`: represents the entire set of resources after initialize. -* `status`: represents the status of resource initialization. - -This operation MUST only be called once per ResourceInitializer. - -#### ResourceInitializer - -The `ResourceInitializer` MUST provide the following operations in the SDK: - -* `On Change` -* (optional) `GetResource` - -ResourceInitializer has two states: - -- *DETECTING*: The provider is waiting for ResourceDetectors to complete. -- *INITIALIZED*: A complete resource is available and ResourceListeners will - be notified of the final resource. - -##### ResourceInitializer creation - -Creation of a ResourceInitializer MUST accept the following parameters: - -* `detectors`: a list of ResourceDetectors. -* (optional) `initialization_timeout`: A timeout for when to abandon slow - ResourceDetectors and consider a resource initialized. - -ResourceInitializer MUST allow initial resource detection during its creation. This -initialization SHOULD not block other SDK providers from initializing (e.g. -MeterProvider, TracerProvider). - -A ResourceInitializer MAY allow customizable concurrency behavior, e.g. using a -separate thread for ResourceListener events. - -Internally, the entities discovered via resource detection MUST be merged in -the order provided to create the initial resource. - -During resource detection, ResourceInitializer MUST NOT fire any ResourceListener -events. - -Upon completion of the resource detection phase, ResourceInitializer MUST fire -an `On ResourceInitialize` event to all ResourceListeners. - -Any calls to `GetResource` operation, if provided, SHOULD block until -resource detection is completed. - -Upon failure for resource detection to complete within a timeout, a resource -SHOULD be constructed with available completed detection, `GetResource` -operations MUST be unblocked and `On ResourceInitialize` event MUST be fired -to all ResourceListeners. This call MUST provide a failure status. - -##### On Change - -`On Change` registers a `ResourceListener` to be called when a Resource has -been detected or detection has failed. - -If the `EntityProvider` is already initialized, then it MUST call -`On ResourceInitialize` immediately with the current resource held by the -`ResourceInitializer`. - -##### Get Resource - -`Get Resource` MUST return a reference to the current resource held by the -ResourceInitializer. - -This operation MAY block when ResourceInitializer is in a DETECTING state, -but MUST NOT block indefinitely. - -#### ResourceDetector - -A `ResourceDetector` concept is updated. - -ResourceDetector MUST provide the following operations: - -- `Detect entities` - -##### Detect Entities operation - -The Detect Entities Operation SHOULD allow asynchronous or long-running -exeuction, e.g. issuing an HTTP request to a local service for identity, reading -from disk, etc. - -The Detect Entities Operation SHOULD return a status to indicate completion or -failure. - -The Detect Entities Operation MUST have a mechanism to report detected `Entity` -to the ResourceInitializer. This MUST include the following: - -- The `type` of the Entity. -- (optional) The `schema_url` of the Entity. -- The `id` of the Entity (as a set of attributes). -- (optional) The `description` of the Entity (as a set of attributes). - -The Detect Entities operation MAY be asynchronous. - -### Protocol Details - -The OpenTelemetry Protocol (OTLP) will be modified such that -`InstrumentationScope` will support carrying `Entity` objects. There are two -ways this could be done: - -- A new `entity_refs` field on `InstrumentationScope` similar to what was - done for resource. This would avoid breaking implementations which make use - of additional attributes in `InstrumentationScope`, as this field would still - contain all new attributes for the scope. -- A new `entities` field on `InstrumentationScope` which would contain the - entire set of entities and all values. Optional attributes would be stored - separately to `entities`, and consumers of OTLP would need to know about - both fields. - -## Trade-offs and mitigations - -The primary trade-offs to make here are around "breaking changes" and subtle -differences in generated telemetry for code leveraging Entity vs. code which -does not. We need give room for consumers of OTLP (vendors, databases, collector) -time to support the new semantics of Entity prior to data showing up which would -not be correctly interpreted without understanding these new semantics. As such, -primarily: - -- Understanding that two `InstrumentationScope`s are different because of the - `Entity` data within them. -- Appropriately aggregating or finding data about an `Entity`. If that entity - is now reported in `InstrumentationScope`, then older systems may no longer - be able to interact with it. - -## Prior art and alternatives - -OpenCensus previously allowed contextual tags to be specified dynamically and used everywhere metric -measurements were reported. Users were then required to select which of these were useful to them via -the definition of "Views". OpenTelemetry has aimed for a simpler solution where every -metric has an implicit View definition, and we leverage metric advice to allow sending attributes -than is naturally used when reporting the metric. - -As called out in the description, [OTEP 4316](https://github.com/open-telemetry/opentelemetry-specification/pull/4316) -proposes making resource fully mutable, which comes with its own set of tradeoffs. - -Today, Semantic Conventions already defines `Entity` and uses it to group and -report `Resource` attributes cohesively. Additionally, Semantic Conventions only -models "entity associations", that is requiring a signal (e.g. a metric, event -or span) to be attached to an entity. For example, the `system.cpu.time` metric -is expected to be associated with a `host` entity. This association makes no -assumption about whether that is through `Resource` or some other mechanism, -and can therefore be extended to support `InstrumentationScope` based entities. - -## Open questions - -Adding entity in InstrumentationScope has a lot of implications that must be -resolved. - -### What are the SDK safeguards against high-cardinality Entities? - -As seen in [Issue #3062](https://github.com/open-telemetry/opentelemetry-specification/issues/3062), -systems observing multiple tenants need to ensure that tenants which are only observed briefly do not -continue to consume resources (particularly memory) for long periods of time. There needs to be -some level of control (either direct or implicit) in allowing new "Scope with Entity" to be created. - -### What happens when an entity exists within Resource and InstrumentationScope? - -Should we consider this a failure or a feature? - -For simplicity in modeling, we plan to prototype where this is *disallowed* -until proven we need this, in which case we need crisp language about -what this models exactly. - -For now, only ONE entity of a given type will be allowed across both -`Resource` and `InstrumentationScope` within the SDK. - -### How to represent Entity in InstrumentationScope? - -Should we do the same thing as `Resource` or something more direct? - -Due to mitigations and concerns around compatibility, we recommend sticking to -the same compatibility decision as `Resource` for `InstrumentationScope`. - -### Are descriptive attributes allowed to change for Resource? - -Its not clear how resource immutability is kept or what is meant by immutable. -Will the resource emitted on the first export be the same as the one emitted for -the entire lifetime of the process? Are descriptive attributes on entities -attached to resource still allowed to change? What about attaching new entities -to that resource? - -For now: - -- The set of entities reported on Resource becomes locked. - All identifying attributes are also locked. -- Whether we want to allow descriptive attributes to change - this can be - determined or evolve over time. Until the ecosystem around OTLP is leveraging - the "identity" attributes of Entity for Resource, we should not allow - mutation of descriptive attributes. - -### What is the expected impact on Collector components? - -How should standard Collector processors (like the attributes processor, -resource processor, or filter processor) be updated to interact with Entities -within the InstrumentationScope? - -### How do we guide developers on when to use InstrumenationScope vs. Resource? - -The distinction between an immutable Resource entity and a "mutable" Scope -entity is subtle. The Resource begins to take on the identity of the SDK itself -while Scope allows "refinement" of that identity, within some bound. - -## Prototypes - -TODO: **In Progress** - -## Future possibilities - -This proposal brings strong multi-tenant capabilities to the OpenTelemetry SDK. One possibility -is to improve the interaction between dynamic `Context` and `IntrumentationScope`, e.g. allowing -some level of interaction between `InstrumentationScope`'s entity set and `Context`. From 3b4c5d4d7e0a6906679d3b68d94140f79712ea8b Mon Sep 17 00:00:00 2001 From: Daniel Dyla Date: Tue, 25 Nov 2025 12:32:20 -0500 Subject: [PATCH 23/26] Insert SDK details from SIG discussions --- .../entities/4665-multiple-resource-in-sdk.md | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/oteps/entities/4665-multiple-resource-in-sdk.md b/oteps/entities/4665-multiple-resource-in-sdk.md index 33bad89db7d..764c5f536cf 100644 --- a/oteps/entities/4665-multiple-resource-in-sdk.md +++ b/oteps/entities/4665-multiple-resource-in-sdk.md @@ -95,7 +95,30 @@ MUST be considered in conflict with each other. ### SDK Details -TODO +When `For Entity` operation is received by a provider, A new child +`Entity Bound Provider` of the same type MUST be created and returned with the +following restrictions: + +- `Entity Bound Provider` MUST be associated with a newly created `Resource` + which is the result of the incoming `Entity` set merged into the original + `Provider`'s resource following the existing `Resource` merging algorithm. + Telemetry created by the parent MUST continue to be associated with the + original unmodified resource. +- The `Bound Provider` MUST share an export pipeline with its parent. The export + component (`SpanProcessor`, `MetricReader`, `LogsProcessor`, etc) MUST not be + `Shutdown` by the `Bound Provider`. This MAY be achieved by wrapping the export + component in a proxy component which ignores calls to `Shutdown` or translates + them into `Force Flush`. +- The `Bound Provider` MUST be configured exactly the same as its parent. + A configuration change on a parent `Provider` MUST be reflected in all of its + child `Entity Bound Providers`. This MAY be achieved by directly sharing + the configuration object between `Providers`. +- A `Bound Provider` MUST NOT be directly configurable. + All configuration comes from its parent. +- If `ForceFlush` or `Shutdown` is called on a `Provider` it MUST also flush all + of its child `Entity Bound Providers`. +- If `Shutdown` is called on a `Bound Provider` it MUST be treated as a + `Force Flush`. It MUST NOT shut down its export pipeline. ## Trade-offs and mitigations From 9e5f432d4c0ccbe97970650c1f4fe092702e7718 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 15 Dec 2025 11:37:46 -0500 Subject: [PATCH 24/26] Remove TODOs. --- oteps/entities/4665-multiple-resource-in-sdk.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/oteps/entities/4665-multiple-resource-in-sdk.md b/oteps/entities/4665-multiple-resource-in-sdk.md index 764c5f536cf..f30864a5fc0 100644 --- a/oteps/entities/4665-multiple-resource-in-sdk.md +++ b/oteps/entities/4665-multiple-resource-in-sdk.md @@ -205,7 +205,8 @@ We will have clear guidance on the `For Entity` methods ## Prototypes -TODO: **In Progress** +- Java: https://github.com/open-telemetry/opentelemetry-java/compare/main...jsuereth:opentelemetry-java:wip-entity-and-providers +- TypeScript: https://github.com/open-telemetry/opentelemetry-js/pull/5620 ## Future possibilities From 7bf8a73612e68d12cc0f556fbc916001fc06b0a7 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 15 Dec 2025 11:38:47 -0500 Subject: [PATCH 25/26] Remove another TODO. --- oteps/entities/4665-multiple-resource-in-sdk.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/oteps/entities/4665-multiple-resource-in-sdk.md b/oteps/entities/4665-multiple-resource-in-sdk.md index f30864a5fc0..b3ad6df98bf 100644 --- a/oteps/entities/4665-multiple-resource-in-sdk.md +++ b/oteps/entities/4665-multiple-resource-in-sdk.md @@ -53,7 +53,8 @@ We proposes these new fundamental concepts in OpenTelemetry: ## Internal details -TODO - introduction. +This proposal splits between an instrumentation facing API, and required +behavior of that API in the SDK. ### API Details From b9a76af1e0b03a5fde5b6621da3bb2b593e7d08c Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 23 Mar 2026 14:59:47 +0000 Subject: [PATCH 26/26] Fix typos to address failing misspell check --- oteps/entities/4665-multiple-resource-in-sdk.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/oteps/entities/4665-multiple-resource-in-sdk.md b/oteps/entities/4665-multiple-resource-in-sdk.md index b3ad6df98bf..8a314f0d522 100644 --- a/oteps/entities/4665-multiple-resource-in-sdk.md +++ b/oteps/entities/4665-multiple-resource-in-sdk.md @@ -48,7 +48,7 @@ We proposes these new fundamental concepts in OpenTelemetry: resource. We create new `{Signal}Provider` instances by providing a new `Entity` on the existing provider. - This will construct a new `Resource` specific to that provider. - - The new provider will re-use all configuraiton (e.g. export pipeline) + - The new provider will re-use all configuration (e.g. export pipeline) defined from the base provider. ## Internal details @@ -134,7 +134,7 @@ primarily: usable as an identity independent of `Entity`. - Consumers should now expect SDKs reporting multiple resources in the same batch. Theoretically, this SHOULD already be supported due to how OTLP is - deisgned to allow aggregation / batching of data at any point. + designed to allow aggregation / batching of data at any point. ## Prior art and alternatives