Optimize histogram reservoir

[dashpole]

This improves the concurrent performance of the histogram reservoir's Offer function by 4x (i.e. 75% reduction).

Accomplish this by locking each measurement, rather than locking around the entire storage. Also, defer extracting the trace context from context.Context until collection time. This improves the performance of Offer, which is on the measure hot path. Exemplars are often overwritten, so deferring the operation until Collect reduces the overall work.

                           │   main.txt   │              hist.txt              │
                           │    sec/op    │   sec/op     vs base               │
FixedSizeReservoirOffer-24    211.4n ± 3%   177.5n ± 3%  -16.04% (p=0.002 n=6)
HistogramReservoirOffer-24   200.85n ± 2%   47.41n ± 2%  -76.40% (p=0.002 n=6)
geomean                       206.1n        91.73n       -55.48%

Benchmarks for Measure:

                                                                                 │  main.txt   │             histres.txt             │
                                                                                 │   sec/op    │    sec/op     vs base               │
SyncMeasure/NoView/ExemplarsEnabled/Int64Histogram/Attributes/0-24                 436.7n ± 4%   114.8n ±  5%  -73.72% (p=0.002 n=6)
SyncMeasure/NoView/ExemplarsEnabled/Int64Histogram/Attributes/10-24                472.2n ± 2%   169.7n ±  8%  -64.08% (p=0.002 n=6)
SyncMeasure/NoView/ExemplarsEnabled/Float64Histogram/Attributes/0-24               431.0n ± 2%   116.3n ±  2%  -73.01% (p=0.002 n=6)
SyncMeasure/NoView/ExemplarsEnabled/Float64Histogram/Attributes/10-24              470.9n ± 1%   171.0n ±  5%  -63.70% (p=0.002 n=6)

I explored using a []atomic.Pointer[measurement], but this had similar performance while being much more complex (needing a sync.Pool to eliminate allocations). The single-threaded performance was also much worse for that solution. See main...dashpole:optimize_histogram_reservoir_old.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.2%. Comparing base (c15644d) to head (95f44c1).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #7443     +/-   ##
=======================================
- Coverage   86.2%   86.2%   -0.1%     
=======================================
  Files        302     302             
  Lines      21973   21971      -2     
=======================================
- Hits       18949   18947      -2     
  Misses      2643    2643             
  Partials     381     381             

Files with missing lines	Coverage Δ
sdk/metric/exemplar/fixed_size_reservoir.go	94.7% <100.0%> (ø)
sdk/metric/exemplar/histogram_reservoir.go	90.9% <100.0%> (-1.7%)	⬇️
sdk/metric/exemplar/storage.go	100.0% <100.0%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bboreham]

On further reflection, I fixed the copying issue before running the benchmark, so it is perhaps reasonable that less racy code runs slower.

Would be good if the tests and/or linter detected the issue. I note that NoCopy was removed from atomic.Value here: golang/go#21504.

[dashpole]

I also see slightly worse results, but agree it is definitely better to be correct. I'll work on a test.

[dashpole]

I added a ConcurrentSafe test, and verified that it fails (quite spectacularly) with the previous atomic.Value implementation.

[bboreham]

lgtm

[dashpole]

The concurrent safe test found another race condition around my usage of sync.Pool, which i'm looking into

[bboreham]

Much simpler now.

[MrAlias]

Overall, looks good to me. Just testing cleanup.

[dashpole]

I put the Measure benchmarks in the description. Not as good of an improvement as I had expected, so i'll have to dig into that more...

[dashpole]

@dmathieu if you have the chance to review, this is related to other optimization PRs.

[dashpole]

I figured out why the benchmark results were so poor: The benchmark was recording all observations in a single bucket, and each bucket has its own lock, so there were effectively no parallelism gains. I switched the benchmark to record observations in different buckets, and that shows that this is a ~70% performance improvement when exemplars are being recorded.