-
Notifications
You must be signed in to change notification settings - Fork 311
Add Central OpenTelemetry Benchmarks Repository project proposal #3495
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
svrnm
merged 7 commits into
open-telemetry:main
from
cijothomas:cijothomas/central-benchmarks-project
Jun 25, 2026
+128
−0
Merged
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
03b1b43
Add Central OpenTelemetry Benchmarks Repository project proposal
cijothomas 2861c6c
Fix CI: add cijothomas to cspell, drop link to not-yet-created repo
cijothomas 0f20694
Link OTEP 5109 in Deliverables
cijothomas 869608e
Add @dashpole as TC sponsor
cijothomas 5472353
Add @trask as GC sponsor
cijothomas 03fe038
Clarify relationship to existing per-language benchmarks
cijothomas 3f8259d
Clarify long-term maintenance of the benchmarks repository
cijothomas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,7 @@ words: | |
| - calendar-localization-ptbr | ||
| - Causely | ||
| - Cheler | ||
| - cijothomas | ||
| - Ciukaj | ||
| - Collibra | ||
| - Coralogix | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| # Central OpenTelemetry Benchmarks Repository | ||
|
|
||
| ## Description | ||
|
|
||
| This project establishes | ||
| `open-telemetry/benchmarks` as | ||
| the project-owned home for cross-language performance scenarios, with a public | ||
| dashboard that tracks release-over-release results. | ||
|
|
||
| The initial scope is intentionally narrow: stand up the repository, land one | ||
| shared scenario (S1: counter API call), and publish a dashboard with data | ||
| points from 2-3 language implementations. Working prototypes for .NET, Rust, | ||
| and Java already exist at | ||
| [`cijothomas/otel-benchmarks`](https://github.com/cijothomas/otel-benchmarks) | ||
| with a live dashboard at | ||
| <https://cijothomas.github.io/otel-benchmarks/>, so v1 is essentially a move | ||
| and a permissions handshake. | ||
|
|
||
| ## Background | ||
|
|
||
| The technical proposal is captured in | ||
| [OTEP 5109](https://github.com/open-telemetry/opentelemetry-specification/pull/5118). | ||
| Today many language SIGs run their own benchmarks, in their own repos, on | ||
| their own schedules, with no cross-language view. A central repository | ||
| gives the project a single, citable answer to "how does OTel perform across | ||
| languages, and how is that changing over time?" without each SIG having to | ||
| give up its own internal CI benchmarking. | ||
|
|
||
| ## Why It Matters | ||
|
|
||
| OpenTelemetry's | ||
| [mission](https://github.com/open-telemetry/community/blob/main/mission-vision-values.md) | ||
| is to make high-quality telemetry a built-in feature of cloud-native software. | ||
| Performance directly determines whether that adoption actually happens — | ||
| library authors will not instrument with OTel natively if doing so adds | ||
| meaningful overhead, and operators will not enable signals that cost too | ||
| much. A central, public benchmarks repository is how the project shows the | ||
| world that we walk the talk on performance: results are visible, comparable | ||
| across languages, and tracked release-over-release. | ||
|
|
||
| Concretely: | ||
|
|
||
| - **Cross-language visibility**: today there is no single place to see, for | ||
| the same scenario, how each language implementation performs. | ||
| - **Release-over-release trend**: per-release data points show whether the | ||
| project is getting faster or slower over time, per language. | ||
| - **Library author confidence**: scenarios like S1 (cost of a counter API | ||
| call when no SDK is wired up) let library authors evaluate the cost of | ||
| instrumenting their libraries natively with OTel. | ||
|
cijothomas marked this conversation as resolved.
|
||
|
|
||
| ## Deliverables | ||
|
|
||
| This is a short project with a definitive ending. v1 is done when: | ||
|
|
||
| 1. `open-telemetry/benchmarks` exists, with the layout from | ||
| [OTEP 5109](https://github.com/open-telemetry/opentelemetry-specification/pull/5118) | ||
| and `CODEOWNERS` set up. | ||
| 2. Scenario S1 is implemented and published for at least 2 languages. | ||
|
cijothomas marked this conversation as resolved.
|
||
| 3. The dashboard is public and reachable from the repo README. | ||
|
cijothomas marked this conversation as resolved.
|
||
|
|
||
| Everything beyond this (more scenarios, more languages, demo-based e2e | ||
| scenarios) is deliberately out of scope for v1 and tracked as future work in | ||
| the OTEP. | ||
|
|
||
| ## Staffing | ||
|
|
||
| This is a small project; staffing is intentionally minimal. | ||
|
|
||
| ### TC Sponsor | ||
|
|
||
| - David Ashpole ([@dashpole](https://github.com/dashpole)) | ||
|
|
||
| ### GC Sponsor | ||
|
|
||
| - Trask Stalnaker ([@trask](https://github.com/trask)) | ||
|
|
||
| ### Repository maintainers | ||
|
|
||
| - Cijo Thomas ([@cijothomas](https://github.com/cijothomas)) | ||
| - Martin Costello ([@martincostello](https://github.com/martincostello)) | ||
|
|
||
|
cijothomas marked this conversation as resolved.
|
||
| The maintainers will provide guidance to language SIGs on contributing a | ||
| harness for their implementation, and to anyone proposing a new scenario | ||
| (scope, methodology, expected outputs). Ownership follows the same model | ||
| as [`opentelemetry.io`](https://github.com/open-telemetry/opentelemetry.io): | ||
| the repository has its own maintainers (above) for shared content | ||
| (scenarios, workflows, top-level docs), while each per-language harness | ||
|
cijothomas marked this conversation as resolved.
|
||
| folder is owned via `CODEOWNERS` by the approvers of the corresponding | ||
| language SIG. This means onboarding a new language does not require any | ||
| additional project-level staffing. | ||
|
|
||
| ## Timeline | ||
|
|
||
| Once OTEP 5109 is accepted, the repository can be created and v1 wrapped | ||
| up in 1-2 weeks of elapsed effort. The prototypes already work end-to-end; | ||
| this project is mostly the move and the `CODEOWNERS` setup. | ||
|
|
||
| ## Future Possibilities | ||
|
|
||
| Out of scope for v1, but plausible follow-ups (also captured in the OTEP): | ||
|
|
||
| - Additional scenarios (span/log emission, sampled-out fast path, exporter | ||
| cost, enrichment cost, etc.). | ||
| - Onboarding additional languages. | ||
| - End-to-end scenarios run against the OpenTelemetry Demo (e.g. measuring | ||
| the cost of moving span filtering between SDK and Collector). | ||
|
|
||
| If any of those grow large enough to need their own staffing or governance, | ||
| they can be proposed as separate projects at that time. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.