Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .cspell.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ words:
- calendar-localization-ptbr
- Causely
- Cheler
- cijothomas
- Ciukaj
- Collibra
- Coralogix
Expand Down
109 changes: 109 additions & 0 deletions projects/central-benchmarks-repository.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Central OpenTelemetry Benchmarks Repository

## Description

This project establishes
`open-telemetry/benchmarks` as
the project-owned home for cross-language performance scenarios, with a public
dashboard that tracks release-over-release results.
Comment thread
cijothomas marked this conversation as resolved.

The initial scope is intentionally narrow: stand up the repository, land one
shared scenario (S1: counter API call), and publish a dashboard with data
points from 2-3 language implementations. Working prototypes for .NET, Rust,
and Java already exist at
[`cijothomas/otel-benchmarks`](https://github.com/cijothomas/otel-benchmarks)
with a live dashboard at
<https://cijothomas.github.io/otel-benchmarks/>, so v1 is essentially a move
and a permissions handshake.

## Background

The technical proposal is captured in
[OTEP 5109](https://github.com/open-telemetry/opentelemetry-specification/pull/5118).
Today many language SIGs run their own benchmarks, in their own repos, on
their own schedules, with no cross-language view. A central repository
gives the project a single, citable answer to "how does OTel perform across
languages, and how is that changing over time?" without each SIG having to
give up its own internal CI benchmarking.

## Why It Matters

OpenTelemetry's
[mission](https://github.com/open-telemetry/community/blob/main/mission-vision-values.md)
is to make high-quality telemetry a built-in feature of cloud-native software.
Performance directly determines whether that adoption actually happens —
library authors will not instrument with OTel natively if doing so adds
meaningful overhead, and operators will not enable signals that cost too
much. A central, public benchmarks repository is how the project shows the
world that we walk the talk on performance: results are visible, comparable
across languages, and tracked release-over-release.

Concretely:

- **Cross-language visibility**: today there is no single place to see, for
the same scenario, how each language implementation performs.
- **Release-over-release trend**: per-release data points show whether the
project is getting faster or slower over time, per language.
- **Library author confidence**: scenarios like S1 (cost of a counter API
call when no SDK is wired up) let library authors evaluate the cost of
instrumenting their libraries natively with OTel.
Comment thread
cijothomas marked this conversation as resolved.

## Deliverables

This is a short project with a definitive ending. v1 is done when:

1. `open-telemetry/benchmarks` exists, with the layout from
[OTEP 5109](https://github.com/open-telemetry/opentelemetry-specification/pull/5118)
and `CODEOWNERS` set up.
2. Scenario S1 is implemented and published for at least 2 languages.
Comment thread
cijothomas marked this conversation as resolved.
3. The dashboard is public and reachable from the repo README.
Comment thread
cijothomas marked this conversation as resolved.

Everything beyond this (more scenarios, more languages, demo-based e2e
scenarios) is deliberately out of scope for v1 and tracked as future work in
the OTEP.

## Staffing

This is a small project; staffing is intentionally minimal.

### TC Sponsor

- David Ashpole ([@dashpole](https://github.com/dashpole))

### GC Sponsor

- Trask Stalnaker ([@trask](https://github.com/trask))

### Repository maintainers

- Cijo Thomas ([@cijothomas](https://github.com/cijothomas))
- Martin Costello ([@martincostello](https://github.com/martincostello))

Comment thread
cijothomas marked this conversation as resolved.
The maintainers will provide guidance to language SIGs on contributing a
harness for their implementation, and to anyone proposing a new scenario
(scope, methodology, expected outputs). Ownership follows the same model
as [`opentelemetry.io`](https://github.com/open-telemetry/opentelemetry.io):
the repository has its own maintainers (above) for shared content
(scenarios, workflows, top-level docs), while each per-language harness
Comment thread
cijothomas marked this conversation as resolved.
folder is owned via `CODEOWNERS` by the approvers of the corresponding
language SIG. This means onboarding a new language does not require any
additional project-level staffing.

## Timeline

Once OTEP 5109 is accepted, the repository can be created and v1 wrapped
up in 1-2 weeks of elapsed effort. The prototypes already work end-to-end;
this project is mostly the move and the `CODEOWNERS` setup.

## Future Possibilities

Out of scope for v1, but plausible follow-ups (also captured in the OTEP):

- Additional scenarios (span/log emission, sampled-out fast path, exporter
cost, enrichment cost, etc.).
- Onboarding additional languages.
- End-to-end scenarios run against the OpenTelemetry Demo (e.g. measuring
the cost of moving span filtering between SDK and Collector).

If any of those grow large enough to need their own staffing or governance,
they can be proposed as separate projects at that time.
Loading