OTEP: A Central OpenTelemetry Benchmarks Repository

[cijothomas]

Proposes open-telemetry/benchmarks — a project-owned repository for cross-language scenario definitions, release-over-release benchmark tracking, and a public GitHub Pages dashboard.

Scope is intentionally minimal: one scenario (API-only counter increment), at least two language harnesses, and a manual per-release update cadence.

A working prototype with .NET, Rust, and Java harnesses and a live dashboard is linked in the Prototypes section.

Project proposal (staffing, maintainers, TC/GC sponsors) is handled in the companion community PR: open-telemetry/community#3495

[pellared]

CC @XSAM

[XSAM]

It looks good overall. My only concern is that some repos' benchmarks are still unstable on bare machines. Like OTel Go's benchmarks with codspeed's runner. related issue: CodSpeedHQ/codspeed-go#62

So, we might need to check the historical benchmark results before adopting a repo's benchmark here.

[bboreham]

I applaud this initiative. The 5000x spread between results exhibited at https://cijothomas.github.io/otel-benchmarks/ is a little extreme (0.02ns? is it doing anything?) but I guess we can iron out the details.

[reyang]

Proposes open-telemetry/benchmarks — a project-owned repository for cross-language scenario definitions, release-over-release benchmark tracking, and a public GitHub Pages dashboard.

Could we work with the opentelemetry-demo project maintainers to use the demo for benchmarks, instead of creating a new thing?

[cijothomas]

I applaud this initiative. The 5000x spread between results exhibited at https://cijothomas.github.io/otel-benchmarks/ is a little extreme (0.02ns? is it doing anything?) but I guess we can iron out the details.

Thanks for taking a look! That spread is the reality today. Scenario S1b used Enabled() guard and since Metrics SDK is not enabled, this is effectively no-op (or a single bool check internally).

Without the Enabled() check, the code has to do work to assemble the attributes and pass it to OTel API. The benchmark is correctly pointing out that (OTel Rust/Java) has non-trivial cost just to assemble the attributes, even when there is no SDK is enabled!.

This is exactly the kind of finding the OTEP is designed to surface. We picked this scenario first deliberately: to make it visible whether the OTel API in each language has near-zero cost when no SDK is enabled. That property is what allows library authors to confidently add native OTel instrumentation without paying a tax on users who aren't observing them - a real blocker for many library owners. The S1/S1b deltas above are concrete evidence of where each language's API surface helps or hinders that goal (e.g. IsEnabled on counters being available in some languages and not others).

(note: prototype uses Github runners, so not very stable and each run get a diff machine. Which is why this proposes to use the dedicated hardware).

[cijothomas]

Proposes open-telemetry/benchmarks — a project-owned repository for cross-language scenario definitions, release-over-release benchmark tracking, and a public GitHub Pages dashboard.

Could we work with the opentelemetry-demo project maintainers to use the demo for benchmarks, instead of creating a new thing?

Thanks @reyang!

@martincostello and I did consider this too, and we'll keep re-evaluating as more scenarios land.

The way I see it: demo is great for the end-to-end "what does deploying OTel cost a real polyglot app" question - OTel clients + collector + network all bundled into one number. This proposal is for narrower, targeted scenarios that pin down a specific cost (S1 is just API/no-op; future ones include span/log emission, enrichment costs in sdks, sampled-out fast paths, etc.). Both feel valuable and don't really overlap.

Totally agree we could host a "run the demo and measure e2e throughput including the sdk/network/collector/backend" scenario here too down the line — that should fit nicely alongside the more focused ones. I can imagine things like "what if a Span filtering is done at the sdk vs in collector? What if we switch from gRCP to HTTP? What if OTel Arrow's proposed OTAP wire-format is used instead of OTLP" etc., with Demo offering A/B testing and the benchmarks showing the total difference etc.

There is potential tie-up needed with OTel blueprint efforts too - it is defining reference architectures, and this repo could provide the perf numbers backing them.

Intentionally starting with an extremely lean scope and get it running E2E, and then expand!

[cijothomas]

Would it make sense to have a more generic repository for this (e.g. something like open-telemetry/test-scenarios)?

Even if we only focus on benchmarks at first, I could see basically the same approach being useful for a language-agnostic test suite to ensure spec correctness/compliance with the same structure

Totally! I've had the same thought a while ago while checking the W3C Trace Context test suite (https://github.com/w3c/trace-context/tree/main/test), which is an important compliance check. I can imagine the spec-compliance matrix in this repo being auto-generated from actual runs instead of maintained by hand.

On the name: since this repo will also hold historical perf results (not just scenario definitions), test-scenarios doesn't quite capture the second half. I don't have a better name yet either. My preference is to start narrow with benchmarks, and once we actually expand scope, pick a name that fits - either renaming the repo or creating a sibling and cross-linking. The results are anyway ultimately in the opentelemetry.io website, so the repo itself is not that end-user exposed.

I'm intentionally keeping the scope as lean as possible to get an end-to-end version landed first. Historically, language-agnostic initiatives have stalled, partly due to scope creep, so hopefully this plan of deliberately staying slim to get things rolling makes sense.

[mx-psi]

I'm happy if we rename the repo later too, I can't come up with a better name right now 😅 My suggestion was only about the name really, I fully agree we should keep the scope as lean as possible :)

[jack-berg]

Can't volunteer for repo ownership but can help contribute the bits for opentelemetry-java.

[braydonk]

I had conversations with multiple distinct groups at KubeCon EU 2026 where this idea had come up. I think it's a great idea and I support it.

[pellared]

I already approved open-telemetry/community#3495

Details can always be tuned later 😉

[cijothomas]

@dashpole I think we have enough approvals for this. Could you merge? (and then I'll ask maintainers of community repo to merge the companion community project PR)