Post-Graduation Roadmap for OpenTelemetry #3452
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -144,6 +144,8 @@ words: | |
| - denisivan | ||
| - dennisme | ||
| - dineshg | ||
| - distro | ||
| - distros | ||
| - dmitrii | ||
| - dmitryax | ||
| - dpauls | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,214 @@ | ||||||
| # Post-Graduation Roadmap: Completing our initial scope of work | ||||||
|
|
||||||
| This document identifies the remaining workstreams needed for the OpenTelemetry project as a whole | ||||||
| to be considered a stable end-to-end platform where users can install, deploy, and operate tracing, | ||||||
| metrics, and logs at scale. While we have many new and exciting projects that expand the scope of | ||||||
| OpenTelemetry, we want to make sure that the original scope of the project is actually completed. | ||||||
|
|
||||||
| ## Background and description | ||||||
|
|
||||||
| As part of the [due diligence](https://github.com/cncf/toc/blob/main/projects/open-telemetry/otel-graduation-dd.md) | ||||||
| for OpenTelemetry's graduation, a number of features were identified by our end users as either | ||||||
| missing or still marked as experimental, even though they were necessary components for OpenTelemetry to be | ||||||
| used as a complete telemetry system. | ||||||
|
|
||||||
| The feedback on the initial proposal was that it was too open-ended, so this document attempts | ||||||
| to redefine the workstreams to be more specific as to which SIG is indented to work on them, and | ||||||
| the specific set of deliverables needed to meet our stability goals. | ||||||
|
|
||||||
| This document is a high-level roadmap, not an individual project. The purpose is to reach consensus | ||||||
| on the overall scope of work needed for the original OpenTelemetry components to be considered a complete | ||||||
| telemetry system consisting entirely of stable components, in a way that satisfies the feedback we received | ||||||
| during graduation. | ||||||
|
|
||||||
| ## Current challenges | ||||||
|
|
||||||
| ### De facto stable | ||||||
|
|
||||||
| Many individual components (language APIs, SDKs, and a growing set of instrumentation libraries) | ||||||
| are already at v1.0 today. However, there are also a number of important components that are | ||||||
| "de facto" stable, meaning that they are versioned as 0.X and are technically still in beta, | ||||||
| but are recommended to be used in production. This is confusing, as OpenTelemetry also has components | ||||||
| marked 0.X that are genuinely experimental and should not be used in production. Additionally, | ||||||
| some end user organizations have rules that prohibit them from deploying 0.X software to production. | ||||||
|
|
||||||
| As part of graduation, it was requested that we provide a mechanism for indicating to end | ||||||
|
Member
There was a problem hiding this comment. I think it would be great if we can come up with a consistent labeling mechanism for Otel docs. Different SDKs and parts of OTel specs use different labels currently. OTLP in particular labels individual features in a way that I like (for example here), similarly OpAMP (see example here). This approach is incidentally similar to how Kubernetes docs use stability labels (for example here). I would love to have this consistent |
||||||
| users which OpenTelemetry components are "production ready." Actually, we already have a | ||||||
| mechanism for indicating this – the version number of the component. Going forward, we need to | ||||||
|
Member
There was a problem hiding this comment. The version numbers typically apply to whole components (like the entire language SDK). However any actively evolving component often ends up having features at different stability levels. It would be useful to label individual features in addition to the version number of the component. I also like how Kubernetes includes both the stability level of the feature and the version number that stability level was attained by the feature. See for example this, saying |
||||||
| align "production readiness" with versioning by advancing all of these "de-facto stable" components | ||||||
| to v1.0 or greater. | ||||||
|
|
||||||
| ### Deploying at scale | ||||||
|
|
||||||
| While there is a way to install every individual component in OpenTelemetry, we do not | ||||||
| currently have tools that can install and manage all of OpenTelemetry at scale. We have the | ||||||
| beginnings of these tools for Kubernetes and Linux, but they are not complete, and can only | ||||||
| manage a portion of the OpenTelemetry components needed for a complete deployment. | ||||||
|
|
||||||
| ### Future-proofing the project | ||||||
|
|
||||||
| We've made it this far, but there are several aspects of the project that need to be improved | ||||||
| in order continue maintaining all of these stable OpenTelemetry components over the next several | ||||||
| years. The specific areas that were identified during graduation include security, project management, | ||||||
| performance, and long term support. | ||||||
|
|
||||||
| ## Workstreams | ||||||
|
|
||||||
| Based on the above challenges, the following workstreams need to be developed and managed as | ||||||
| a roadmap that can be presented to the community. | ||||||
|
|
||||||
| * Stability | ||||||
| * Collector v1.0 | ||||||
| * Tooling for Instrumentation | ||||||
| * Instrumentation Stability & Management | ||||||
| * Deployment | ||||||
| * OpAMP v1.0 | ||||||
| * Linux Package Management v1.0 | ||||||
| * Kubernetes Operator v1.0 | ||||||
| * Kubernetes Helm charts v1.0 | ||||||
| * Improvements | ||||||
| * Security | ||||||
| * Roadmaps & Project Management | ||||||
| * Performance & Benchmarking | ||||||
| * Self-Observability | ||||||
| * Redefine Long-Term Support | ||||||
|
|
||||||
| ## Stability | ||||||
|
|
||||||
| ### Collector v1.0 | ||||||
|
|
||||||
| Managed by Collector SIG, the OpenTelemetry Collector needs to complete [its roadmap for v1.0](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/ga-roadmap.md). | ||||||
| This includes marking core APIs as well as a minimal OTLP distro as 1.x. | ||||||
|
|
||||||
| The opentelemetry-collector-contrib repository contains over 200 hundred components. [Initial adopter interview findings](https://docs.google.com/document/d/1SQMdfYpCiBfpxtWDwASXVIl-PIzD9X4vdDPXYUphAF0/edit?tab=t.0) | ||||||
| revealed that, although many of these are considered not 'core' and are instead community-supported, | ||||||
| end-users rely on them. v1.0 therefore will include | ||||||
| [the stability of additional priority components identified in 'phase 1'](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/44130). | ||||||
|
|
||||||
| Marking further components or distros as 1.x is explicitly out of scope. | ||||||
|
|
||||||
| ### Tooling for Instrumentation | ||||||
|
|
||||||
| In order to lower the cost of manage instrumentation at scale, and to better support native | ||||||
| instrumentation efforts, we need tooling that improves correctness while reducing the cost of | ||||||
| maintaining instrumentation. The SemConv Tooling SIG is in charge of this project. | ||||||
|
Member
There was a problem hiding this comment. nit 😉
Suggested change
|
||||||
|
|
||||||
| * Weaver | ||||||
| * AI coding | ||||||
|
Member
There was a problem hiding this comment. AI coding - bit vague - could we add a one-liner to clarify what we mean?
Contributor
Author
There was a problem hiding this comment. Definitely need more details here, but I want the SemConv Tooling SIG to take a look at it before I put too many words in their mouth 🙂 |
||||||
| * Test harnesses | ||||||
|
|
||||||
| ### Instrumentation Stability & Management | ||||||
|
|
||||||
| Currently, the biggest barrier to stability is instrumentation. The Specification SIG is | ||||||
| in charge of this project, but it requires significant input from maintainers and the community. | ||||||
|
|
||||||
| * Move away from the “community contrib” model for critical instrumentation packages. | ||||||
|
Member
There was a problem hiding this comment. From my experience, I would be against moving away from the community/contrib model for "critical" instrumentation packages. Instrumentation is not one-size-fits-all. Even with semantic conventions, there are still many valid choices an instrumentation package has to make: which spans, metrics, attributes, and events to emit; which features should be enabled by default; how much configuration should be exposed; how to handle high-cardinality data; and how much runtime overhead is acceptable. Different users often make different trade-offs here. Because of that, I think it is healthy to allow multiple instrumentation packages for the same library or framework, as long as they follow semantic conventions and are discoverable. A single “official” package can easily become a bottleneck or end up optimizing for one set of users while being unsuitable for another. For example, in OTel Go,
Instead of replacing the community/contrib model, I would prefer improving the quality bar around it: clearer stability signals, better discoverability, documented compatibility, semantic convention compliance, and guidance on which packages are recommended for which use cases. That would preserve the flexibility and innovation of the ecosystem while still helping users make safe choices for critical instrumentation. This is the main reason I am not approving this PR as I am against this bullet 😉 There was a problem hiding this comment. I read this as "move critical instrumentation from contrib into core". I don't think this is proposing moving away from the community/contrib model generally. @tedsuo maybe you can clarify what you had in mind here? |
||||||
| * Deploy the new SemConv tooling across all language ecosystems. | ||||||
| * Badges and other forms of recognition. | ||||||
|
Member
There was a problem hiding this comment. What about https://opentelemetry.io/ecosystem/registry/? |
||||||
| * Native instrumentation push to move instrumentation out of OpenTelemetry. | ||||||
|
Member
There was a problem hiding this comment. Native instrumentation - do we mean libraries/frameworks natively picking a dependency on opentelemetry api, and isntrumenting themselves, without the need for us to make a instrumentation library package?
Contributor
Author
There was a problem hiding this comment. Yes. There are two directions a community/contrib instrumentation package could gain more support. One is that a trusted group within OpenTelemetry maintains the package. The second is that the library itself includes the instrumentation natively, so there is no need for OpenTelemetry to maintain a separate package. In both cases, we've identified a lack of tooling as a barrier. It's difficult to write instrumentation that matches the semantic conventions without making mistakes. So, once we have better tooling, we have an opportunity to try to upstream instrumentation. This would be preferable to maintaining a separate instrumentation package.
Member
There was a problem hiding this comment. I am not sure on framing tooling (or lack of it) as the main barrier. If I'm a library owner deciding whether to take a direct dependency on the OTel API, my decision tree looks roughly like this:
The current wording treats (4) as the primary blocker, but (1)–(3) come first IMHO for any library owner. I also don't recall library authors citing tooling as their blocker in prior discussions — the concerns I've consistently heard are stability and performance. If the SemConv Tooling SIG has data showing otherwise, it would be great to link it. (I only have anecdotal evidence only, so happy to correct my position once I learn more) For this workstream to actually deliver native instrumentation at scale, I think we need explicit commitments on (1)–(3) — API LTS guarantees, a no-op/hot-path performance commitment, and semconv stability — alongside the tooling work.
Contributor
There was a problem hiding this comment.
What do you actually expect from this at this time? Creating documentation and (stable) tooling that gets us ready to start upstreaming instrumentation? |
||||||
| * Update the spec with the new policy for defining stable telemetry. | ||||||
|
|
||||||
| ## Deployment | ||||||
|
|
||||||
| ### OpAMP v1.0 | ||||||
|
|
||||||
| In order to manage OpenTelemetry at scale, we need a control plane. Therefore, OpAMP needs | ||||||
| to be stable and feature complete for its core set of management tasks. | ||||||
|
|
||||||
| The OpAMP SIG is in charge of this project. [Detailed roadmap here](https://docs.google.com/document/d/1DXXAZmm_pG2ls1zZTzXzUI9bBscunBmcpVbbbN267og/edit). | ||||||
|
|
||||||
| * Collector supervision | ||||||
| * Remote configuration | ||||||
| * Health reporting | ||||||
| * Release management | ||||||
|
|
||||||
| ### Linux Package Management v1.0 | ||||||
|
|
||||||
| The Packaging SIG is in charge of this project. | ||||||
|
|
||||||
| * Official packages for Debian and RHEL | ||||||
| * Cross language definitions for distributions and versioning | ||||||
| * Language distributions for SDKs, plugins, and instrumentation | ||||||
| * Declarative configuration for managing instrumentation and stability | ||||||
|
Contributor
There was a problem hiding this comment. Maybe @jack-berg has an opinion (or maybe not, but pinging him just in case). |
||||||
|
|
||||||
| ### Kubernetes Operator v1.0 | ||||||
|
|
||||||
| The Kubernetes Operator is a critical component for managing OpenTelemetry at scale. | ||||||
| The Kubernetes SIG is in charge of this project. | ||||||
|
|
||||||
| * The need for pod attribution and other manual configuration requirements that interfere | ||||||
| with deploying OpenTelemetry at scale. | ||||||
| * All languages that have both auto-instrumentation and a stable SDK should be supported. | ||||||
| * Works with the same distributions and configuration options as developed by the package | ||||||
| management SIG, so that end users only need to learn a single set of configuration patterns. | ||||||
|
|
||||||
| ### Kubernetes Helm Charts v1.0 | ||||||
|
|
||||||
| In addition to the Operator, OpenTelemetry provides a set of helm charts for managing our suite | ||||||
| of components. All of these charts are still versioned 0.X. At minimum, the following helm charts | ||||||
| must become stable: | ||||||
|
|
||||||
| * opentelemetry-operator when the Operator becomes v1.0 | ||||||
| * opentelemetry-collector when the Collector becomes v1.0 | ||||||
|
|
||||||
| The remaining helm charts either need a roadmap for v1.0, or a warning that they are still experimental | ||||||
| and not meant to be run in production. The Kubernetes SIG is in charge of this project. | ||||||
|
|
||||||
| ## Improvements | ||||||
|
|
||||||
| ### Security | ||||||
|
|
||||||
| While project-wide security protocols can always be improved. The following tasks have been | ||||||
| identified as high priority. The Security SIG is in charge of this project. | ||||||
|
Comment on lines
+161
to
+162
Member
There was a problem hiding this comment. What about mitigating supply chain attacks? E.g. applying immutable releases, build attestations, artifact signing for all projects. Take https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.29.0 as an example that could be improved. I think this is an area which is overlooked. |
||||||
|
|
||||||
| * Better staffing of the Security SIG. | ||||||
|
Contributor
There was a problem hiding this comment. Any initial idea on how to accomplish this? Some SIGs are (or feel?) understaffed and the only way to improve things is to get companies onboard.
Contributor
Author
There was a problem hiding this comment. Well, I think that security pulls from a different part of the organization than staffing other parts of OpenTelemetry. Right now, there is a heightened awareness around security, and many companies (end users, cloud providers, and vendors alike) are aware to the fact that OpenTelemetry is an important part of their technology stack. If OpenTelemetry is vulnerable, then they are vulnerable. So I think that companies could be convinced to contribute time from their security professional in two different ways: one-off projects that improve security for OpenTelemetry. These projects would need to have limited scope and clear benefits, but I think that it's reasonable to assume we could get them staffed if the value is high enough. For longer-term support, if we can set clear expectations for what the monthly time commitment is, the value is that these security professionals would have early access to CVE reports. I think that some companies with a serious commitments to OpenTelemetry would rest easier with that knowledge, enough so that the security investment would seem worth the cost of having to triage that backlog and work with other maintainers to ensure that any problems are dealt with quickly and effectively. |
||||||
| * Triage that can keep up with AI-powered CVE reporting noise. | ||||||
| * Oversight to ensure that security response protocols are being followed. | ||||||
| * OTel scanning tools to counter noisy and inaccurate scanning tools used by end users. | ||||||
|
Member
There was a problem hiding this comment. Shouldn't we also do our own scanning? I think we should also add References:
Member
There was a problem hiding this comment. Should we also recommend fuzzing? Currently we have some fuzz tests in OTel Collector and OTel Go |
||||||
|
|
||||||
| ### Roadmaps & Project Management | ||||||
|
|
||||||
| While OpenTelemetry has been making many incremental improvements to its project management | ||||||
| tools and workflows, we have identified a need for larger changes to our project structure. | ||||||
|
|
||||||
| OpenTelemetry always had a de-facto roadmap – first traces, then metrics, then logs. That | ||||||
| roadmap is now complete. Once we have finished delivering this original scope of work, the roadmap | ||||||
| will be unwritten. How will we write it as a community? | ||||||
|
|
||||||
| The Governance Committee is in charge of this project, but everyone's input is needed. | ||||||
|
|
||||||
| * More agency and responsibility for the roadmap in the hands of the maintainers. | ||||||
| * Better ways to socialize the projects that need input. | ||||||
| * Better ways to visualize the long term roadmap. | ||||||
|
|
||||||
| ### Performance & Benchmarking | ||||||
|
|
||||||
| To facilitate performance improvements, SDK maintainers have created a number of benchmarks for | ||||||
|
Member
There was a problem hiding this comment. thanks for adding this. it was also part of our original OTel GA plan :) |
||||||
| use within their individual SIGs. Based on a review of this work, we would like to create a | ||||||
| set of standard benchmarks to be applied across all SDKs. This is both to help with performance | ||||||
| improvements, and to help give end users a reasonable set of expectations when considering the use | ||||||
| of our SDKs. | ||||||
|
|
||||||
| The Specification SIG is in charge of this project. | ||||||
|
|
||||||
|
|
||||||
| ### Self-Observability | ||||||
|
|
||||||
| OpenTelemetry is itself a high-throughput distributed sub-system. This means it needs to emit | ||||||
|
Member
There was a problem hiding this comment. this may require sub-sections covering collector, operator, language sdks. |
||||||
| sufficient telemetry about itself in order to be observable, along with a set of playbooks | ||||||
| to explain how to interpret this telemetry, and advise operators on how to implement basic dashboards | ||||||
| and alerting needed to respond to common problems. In other words, OpenTelemetry itself should | ||||||
| be a model citizen when it comes to being an observable system. As part of encouraging native | ||||||
| instrumentation, we want to implement our own best practices in order to encourage others to | ||||||
| improve the observability of their systems. | ||||||
|
|
||||||
| The Specification SIG is in charge of this project. | ||||||
|
Member
There was a problem hiding this comment. Nit: More like a join effort with SemanticConventions group, as the self-observability semantics are going to be defined in the semantic conventions. |
||||||
|
|
||||||
| ### Redefine Long term support | ||||||
|
|
||||||
| The primary goal of this initiative is to get all of the necessary components to v1.0. | ||||||
|
Member
There was a problem hiding this comment. This probably need to be rewritten to match the new title. |
||||||
| OpenTelemetry does have compatibility and support requirements for various types of | ||||||
| components, such as APIs and plugin architectures. As part of reaching this milestone, | ||||||
| we need to revisit our compatibility requirements and long-term support guarantees, to make | ||||||
| sure that they accurately reflect our intentions. | ||||||
|
|
||||||
| The Specification SIG is in charge of this project. | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 this is very good! It also makes sense to do a blog after this PR is approved to let the world know what is OTel doing after graduation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 to the new name.