Is this actually true? I've not heard of baking in instrumentation making it difficult for users who also wish to augment autoinstrumentation with custom instrumentation, nor those who wish to just turn off autoinstrumentation and do it all themselves.

For baked-in, the customer will probably always need to wait for the whole release of the agent framework if they want to have some customized instrumentation; with otel, it will be relatively easier, only the instrumentation needs update and customer can package that, this is reasonable?

For baked-in, the customer will probably always need to wait for the whole release of the agent framework if they want to have some customized instrumentation; with otel, it will be relatively easier, only the instrumentation needs update and customer can package that, this is reasonable?

Is it just users that need to be able to easily enable/disable the telemetry?

Isn't it other instrumentation packages, also, that need to be able to suppress the instrumentation (in order to prevent duplicative instrumentation)?

Do we want to recommend a uniform approach to this setting?

For example, it looks like one approach used is to have a per-instrumentation/per-library key in the OTel context (e.g. "suppress_instrumentation_${library_name}") and to check for the presence of that in the context to avoid instrumentation. (This way, calling libraries that generate duplicate information can set the key).

@michaelsafyan this is the baked-in option, if you take a look at crewai at https://docs.crewai.com/telemetry, you will see it has a parameter named as OTEL_SDK_DISABLED to config.

I'm not sure that this is actually true given the zero-configuration mechanisms in OTel for setup.

Customer still need to install otel instrumentation packages for different llm providers, sometimes, they may need to config otel collector to integrate with some 3rd party observability platforms.

With baked-in option, the agent may have build-in UI for observability which does not request any configuration, hope this helps.

A related con/risk to call out:

• You may not get feedback/review from OTel contributors familiar with current Semantic Conventions
• Your instrumentation may lag with respect to best practices/conventions (not just the version of the OTel library dependencies).

Done

Is the number of dependencies in the long run truly that different?

Yes, I think so. Like for otel, we need to install instrumentation code, otel collector, but with baked-in, do not need to install those dependencies manually.

Isn't it still likely that manual setup may still be required? For example, even if the library includes instrumentation, it is not going to wire-up Open Telemetry to the appropriate backends.

Here the manual set up will be mainly for the agent framework itself, like CrewAI or others. After the agent framework installed, there is no need to setup observability manually as it should be build-in with crewai or agent framework itself. Hope this helps.

I think Option 2 actually should be split up into:

"Option 2: External instrumentation"
  - "Option 2a: External instrumentation in your own repository/package"
  - "Option 2b: External instrumentation in an Open Telemetry-owned repository/package"

(Or maybe you just make "2a" and "2b" into 2 and 3).

Done

Some additional benefits:

• More likely to leverage best practices around Semantic Conventions
• More likely to leverage best practices around zero-code instrumentation

Done

This is untrue. With zero-code setup, the instrumentation libraries can be auto-discovered.

See:

• https://opentelemetry.io/docs/zero-code/python/

It is actually easier to leverage this with this approach, because opentelemetry-boostrap -a will auto-install instrumentation libraries from this repo, but it won't auto-install other instrumentation packages.

removed

Can you explain? How would they leverage "incompatible" ones? Won't that result in an error when attempting to install the dependencies?

yes, but sometimes the install can be succeed, and there maybe some runtime inconsistencies, incompatibilities, and maintenance issues that can arise when different users or frameworks depend on different versions of contributed (contrib) packages

True, though:

1. The quality is likely to be higher given the higher bar held in that repo for telemetry quality.
2. There is still the ability to contribute to it in order to improve it.

I think the "less control" is more relevant in relation to velocity rather than in relation to quality.

good point, updated to below

Development velocity slows down when there are too many PRs in the OpenTelemetry review queue.

Done

Done

removed

yes, but sometimes the install can be succeed, and there maybe some runtime inconsistencies, incompatibilities, and maintenance issues that can arise when different users or frameworks depend on different versions of contributed (contrib) packages

good point, updated to below

Development velocity slows down when there are too many PRs in the OpenTelemetry review queue.