Use sync.Map and atomics for fixed bucket histograms by dashpole · Pull Request #7474 · open-telemetry/opentelemetry-go

dashpole · 2025-10-08T14:31:21Z

Implement a lockless histogram using atomics, and use a sync.Map for attribute access. This improves performance by ~2x.

The design is very similar to #7427, but with one additional change to make the histogram data point itself atomic:

For cumulative histograms, which do not use a hot/cold limitedSyncMap, we use a hot/cold data point. This way, we maintain the keys in the sync map, but still ensure that collection gets a consistent view of measure() calls.

Parallel benchmarks:

                                                                       │  main.txt   │              hist.txt              │
                                                                       │   sec/op    │   sec/op     vs base               │
SyncMeasure/NoView/ExemplarsDisabled/Int64Histogram/Attributes/10-24     274.5n ± 2%   125.2n ± 5%  -54.42% (p=0.002 n=6)
SyncMeasure/NoView/ExemplarsDisabled/Float64Histogram/Attributes/10-24   274.1n ± 2%   132.5n ± 2%  -51.65% (p=0.002 n=6)
geomean                                                                  274.3n        128.8n       -53.05%

zero memory allocations before and after this change for Measure(). Omitted for brevity

Benchmarks for collect:

                                                    │   main.txt    │               hist.txt               │
                                                    │    sec/op     │    sec/op     vs base                │
Collect/NoView/Int64Histogram/1/Attributes/0-24       1.799µ ±  60%    1.702µ ± 6%         ~ (p=1.000 n=6)
Collect/NoView/Int64Histogram/1/Attributes/1-24       973.7n ±  28%   1720.0n ± 5%   +76.65% (p=0.002 n=6)
Collect/NoView/Int64Histogram/1/Attributes/10-24      881.0n ±  17%   1710.0n ± 5%   +94.09% (p=0.002 n=6)
Collect/NoView/Int64Histogram/10/Attributes/0-24      996.1n ±  14%   1781.5n ± 4%   +78.85% (p=0.002 n=6)
Collect/NoView/Int64Histogram/10/Attributes/1-24      1.029µ ±  67%    1.733µ ± 3%   +68.42% (p=0.009 n=6)
Collect/NoView/Int64Histogram/10/Attributes/10-24     1.533µ ±  18%    1.708µ ± 4%         ~ (p=0.240 n=6)
Collect/NoView/Float64Histogram/1/Attributes/0-24     1.222µ ± 120%    1.733µ ± 4%         ~ (p=0.065 n=6)
Collect/NoView/Float64Histogram/1/Attributes/1-24     893.3n ±   8%   1733.0n ± 4%   +94.00% (p=0.002 n=6)
Collect/NoView/Float64Histogram/1/Attributes/10-24    860.7n ±   2%   1732.0n ± 5%  +101.23% (p=0.002 n=6)
Collect/NoView/Float64Histogram/10/Attributes/0-24    852.5n ±   4%   1758.0n ± 3%  +106.22% (p=0.002 n=6)
Collect/NoView/Float64Histogram/10/Attributes/1-24    853.8n ±   3%   1725.0n ± 3%  +102.04% (p=0.002 n=6)
Collect/NoView/Float64Histogram/10/Attributes/10-24   843.4n ±   2%   1755.0n ± 4%  +108.10% (p=0.002 n=6)
geomean                                               1.028µ           1.732µ        +68.46%

                                                    │  main.txt  │               hist.txt               │
                                                    │    B/op    │     B/op      vs base                │
Collect/NoView/Int64Histogram/1/Attributes/0-24       336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Int64Histogram/1/Attributes/1-24       336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Int64Histogram/1/Attributes/10-24      336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Int64Histogram/10/Attributes/0-24      336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Int64Histogram/10/Attributes/1-24      336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Int64Histogram/10/Attributes/10-24     336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Float64Histogram/1/Attributes/0-24     336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Float64Histogram/1/Attributes/1-24     336.0 ± 0%    2130.5 ± 0%  +534.08% (p=0.002 n=6)
Collect/NoView/Float64Histogram/1/Attributes/10-24    336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Float64Histogram/10/Attributes/0-24    336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Float64Histogram/10/Attributes/1-24    336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
Collect/NoView/Float64Histogram/10/Attributes/10-24   336.0 ± 0%    2131.0 ± 0%  +534.23% (p=0.002 n=6)
geomean                                               336.0        2.081Ki       +534.21%

                                                    │  main.txt  │             hist.txt              │
                                                    │ allocs/op  │ allocs/op   vs base               │
Collect/NoView/Int64Histogram/1/Attributes/0-24       5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Int64Histogram/1/Attributes/1-24       5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Int64Histogram/1/Attributes/10-24      5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Int64Histogram/10/Attributes/0-24      5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Int64Histogram/10/Attributes/1-24      5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Int64Histogram/10/Attributes/10-24     5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Float64Histogram/1/Attributes/0-24     5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Float64Histogram/1/Attributes/1-24     5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Float64Histogram/1/Attributes/10-24    5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Float64Histogram/10/Attributes/0-24    5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Float64Histogram/10/Attributes/1-24    5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
Collect/NoView/Float64Histogram/10/Attributes/10-24   5.000 ± 0%   6.000 ± 0%  +20.00% (p=0.002 n=6)
geomean                                               5.000        6.000       +20.00%

Collect does get substantially worse, but Measure is expected to be called significantly more often than collect.

codecov · 2025-10-08T14:34:56Z

Codecov Report

❌ Patch coverage is 94.85714% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.2%. Comparing base (b4578c8) to head (d6c4ded).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sdk/metric/internal/aggregate/atomic.go	83.6%	9 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##            main   #7474    +/-   ##
======================================
  Coverage   86.1%   86.2%            
======================================
  Files        298     298            
  Lines      21726   21829   +103     
======================================
+ Hits       18722   18818    +96     
- Misses      2627    2635     +8     
+ Partials     377     376     -1

Files with missing lines	Coverage Δ
sdk/metric/internal/aggregate/aggregate.go	`100.0% <100.0%> (ø)`
sdk/metric/internal/aggregate/histogram.go	`100.0% <100.0%> (ø)`
sdk/metric/internal/aggregate/sum.go	`100.0% <100.0%> (ø)`
sdk/metric/internal/aggregate/atomic.go	`90.0% <83.6%> (-2.8%)`	⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bwplotka

Trying to help a bit with review of this.

Generally looks good, just small things, but also I'm not a maintainer and a bit new to the codebase (I maintain Prometheus client_golang SDK though).

Great work, great to see amazing results! Do you mid benchmarking with allocs too?

dashpole · 2025-11-07T15:19:44Z

Great work, great to see amazing results! Do you mid benchmarking with allocs too?

Yeah, zero allocs on the Measure() path before and after, so I didn't post it.

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>

MrAlias · 2025-12-04T18:09:42Z

@dmathieu @flc1125: would either of you have time to review this? @dashpole is able to answer questions if that helps.

dashpole · 2025-12-09T16:50:12Z

Not sure why coverage is failing

MrAlias · 2025-12-09T16:57:34Z

Not sure why coverage is failing

I think the changes are just exposing an area where we have had coverage overlap. I think this has more to do with how we merge coverage instead of the changes here.

MrAlias · 2025-12-09T17:10:55Z

Likely worth looking into using Go 1.20 coverage directory feature to replace our current approach.

dashpole · 2025-12-11T16:49:43Z

Seems to have resolved itself. Now just links are failing (503s from github due to outage).

Depends on #7474 This applies similar optimizations as #7427 to the last value aggregation. Changes for last value are contained in 27e1482. Parallel benchmarks: ``` │ main.txt │ lv.txt │ │ sec/op │ sec/op vs base │ SyncMeasure/NoView/ExemplarsDisabled/Int64Gauge/Attributes/10-24 264.60n ± 3% 66.46n ± 1% -74.88% (p=0.002 n=6) SyncMeasure/NoView/ExemplarsDisabled/Float64Gauge/Attributes/10-24 270.25n ± 4% 69.69n ± 1% -74.21% (p=0.002 n=6) geomean 267.4n 68.05n -74.55% ``` Co-authored-by: Damien Mathieu <42@dmathieu.com>

### Added - Add `Enabled` method to all synchronous instrument interfaces (`Float64Counter`, `Float64UpDownCounter`, `Float64Histogram`, `Float64Gauge`, `Int64Counter`, `Int64UpDownCounter`, `Int64Histogram`, `Int64Gauge`,) in `go.opentelemetry.io/otel/metric`. This stabilizes the synchronous instrument enabled feature, allowing users to check if an instrument will process measurements before performing computationally expensive operations. (#7763) - Add `AlwaysRecord` sampler in `go.opentelemetry.io/otel/sdk/trace`. (#7724) - Add `go.opentelemetry.io/otel/semconv/v1.39.0` package. The package contains semantic conventions from the `v1.39.0` version of the OpenTelemetry Semantic Conventions. See the [migration documentation](https://github.com/open-telemetry/opentelemetry-go/blob/298cbedf256b7a9ab3c21e41fc5e3e6d6e4e94aa/semconv/v1.39.0/MIGRATION.md) for information on how to upgrade from `go.opentelemetry.io/otel/semconv/v1.38.0.` (#7783, #7789) ### Changed - `Exporter` in `go.opentelemetry.io/otel/exporter/prometheus` ignores metrics with the scope `go.opentelemetry.io/contrib/bridges/prometheus`. This prevents scrape failures when the Prometheus exporter is misconfigured to get data from the Prometheus bridge. (#7688) - Improve performance of concurrent histogram measurements in `go.opentelemetry.io/otel/sdk/metric`. (#7474) - Add experimental observability metrics in `go.opentelemetry.io/otel/exporters/stdout/stdoutmetric`. (#7492) - Improve the concurrent performance of `HistogramReservoir` in `go.opentelemetry.io/otel/sdk/metric/exemplar` by 4x. (#7443) - Improve performance of concurrent synchronous gauge measurements in `go.opentelemetry.io/otel/sdk/metric`. (#7478) - Improve performance of concurrent exponential histogram measurements in `go.opentelemetry.io/otel/sdk/metric`. (#7702) - Improve the concurrent performance of `FixedSizeReservoir` in `go.opentelemetry.io/otel/sdk/metric/exemplar`. (#7447) - The `rpc.grpc.status_code` attribute in the experimental metrics emitted from `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` is replaced with the `rpc.response.status_code` attribute to align with the semantic conventions. (#7854) - The `rpc.grpc.status_code` attribute in the experimental metrics emitted from `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc` is replaced with the `rpc.response.status_code` attribute to align with the semantic conventions. (#7854) ### Fixed - Fix bad log message when key-value pairs are dropped because of key duplication in `go.opentelemetry.io/otel/sdk/log`. (#7662) - Fix `DroppedAttributes` on `Record` in `go.opentelemetry.io/otel/sdk/log` to not count the non-attribute key-value pairs dropped because of key duplication. (#7662) - Fix `SetAttributes` on `Record` in `go.opentelemetry.io/otel/sdk/log` to not log that attributes are dropped when they are actually not dropped. (#7662) - `WithHostID` detector in `go.opentelemetry.io/otel/sdk/resource` to use full path for `ioreg` command on Darwin (macOS). (#7818) - Fix missing `request.GetBody` in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` to correctly handle HTTP2 GOAWAY frame. (#7794) ### Deprecated - Deprecate `go.opentelemetry.io/otel/exporters/zipkin`. For more information, see the [OTel blog post deprecating the Zipkin exporter](https://opentelemetry.io/blog/2025/deprecating-zipkin-exporters/). (#7670) --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

dashpole force-pushed the optimize_syncmap_histogram branch 4 times, most recently from ac02811 to 389d0bd Compare October 8, 2025 18:22

dashpole marked this pull request as ready for review October 8, 2025 18:23

dashpole requested review from MrAlias, XSAM, dmathieu, flc1125 and pellared as code owners October 8, 2025 18:23

dashpole force-pushed the optimize_syncmap_histogram branch 3 times, most recently from 34d7502 to f0b28ca Compare October 8, 2025 18:55

dashpole mentioned this pull request Oct 8, 2025

Use sync.Map and atomics for lastvalue aggregations #7478

Merged

MrAlias reviewed Oct 8, 2025

View reviewed changes

Comment thread sdk/metric/internal/aggregate/atomic.go Outdated

pellared mentioned this pull request Oct 10, 2025

SIG meeting notes #6648

Open

MrAlias reviewed Oct 10, 2025

View reviewed changes

dashpole force-pushed the optimize_syncmap_histogram branch 4 times, most recently from 45effea to b18ff3d Compare October 15, 2025 20:14

MrAlias added this to the v1.39.0 milestone Oct 16, 2025

dashpole mentioned this pull request Oct 24, 2025

Improve concurrent performance of exponential histogram measurements #7535

Closed

bwplotka reviewed Nov 7, 2025

View reviewed changes

dashpole mentioned this pull request Nov 7, 2025

Reconsider usage of generics in sdk/metric/internal/aggregate #7589

Open

dashpole added 3 commits November 10, 2025 16:21

rename valueMap in sum to sumValueMap

3a833d1

use sync Map for explicit bucket histogram aggregation

6e47936

use atomics for explicit bucket histogram internals

104cb3b

dashpole and others added 3 commits November 10, 2025 16:21

lint

44172bb

Apply suggestions from code review

61230f9

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>

find index before starting write operation

219e89e

dashpole force-pushed the optimize_syncmap_histogram branch from e3c2257 to 219e89e Compare November 10, 2025 16:24

dashpole and others added 2 commits November 10, 2025 11:24

Update sdk/metric/internal/aggregate/histogram.go

bf7f307

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>

Merge branch 'main' into optimize_syncmap_histogram

a75ed56

MrAlias approved these changes Dec 4, 2025

View reviewed changes

Merge branch 'main' into optimize_syncmap_histogram

d69bdba

MrAlias modified the milestones: v1.39.0, v1.40.0 Dec 4, 2025

dmathieu approved these changes Dec 9, 2025

View reviewed changes

Merge branch 'main' into optimize_syncmap_histogram

28ba797

dashpole commented Dec 9, 2025

View reviewed changes

Comment thread CHANGELOG.md Outdated

Update CHANGELOG.md

7ce277b

dashpole added 2 commits December 9, 2025 16:11

Merge branch 'main' into optimize_syncmap_histogram

ec3ed6c

Merge branch 'main' into optimize_syncmap_histogram

d6c4ded

dashpole merged commit f57bf14 into open-telemetry:main Dec 11, 2025
33 of 35 checks passed

dashpole deleted the optimize_syncmap_histogram branch December 11, 2025 16:56

MrAlias mentioned this pull request Jan 16, 2026

Optimize the metric SDK #7796

Open

39 tasks

MrAlias mentioned this pull request Feb 2, 2026

Release v1.40.0 #7859

Merged

This was referenced Mar 3, 2026

Bump go.opentelemetry.io/otel/sdk from 1.37.0 to 1.40.0 in /flytectl flyteorg/flyte#6959

Open

Bump go.opentelemetry.io/otel/sdk from 1.35.0 to 1.40.0 in /flytecopilot flyteorg/flyte#6960

Open

dashpole mentioned this pull request Mar 23, 2026

Use sync.Map for exponential histogram aggregations #8077

Draft

Conversation

dashpole commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dashpole commented Nov 7, 2025

Uh oh!

MrAlias commented Dec 4, 2025

Uh oh!

Uh oh!

dashpole commented Dec 9, 2025

Uh oh!

MrAlias commented Dec 9, 2025

Uh oh!

MrAlias commented Dec 9, 2025

Uh oh!

dashpole commented Dec 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dashpole commented Oct 8, 2025 •

edited

Loading

codecov bot commented Oct 8, 2025 •

edited

Loading