[sdk-metrics] Promote overflow attribute from experimental to stable

[xiang17]

Fixes #5641.
Design discussion issue #

Changes

Remove the OTEL_DOTNET_EXPERIMENTAL_METRICS_EMIT_OVERFLOW_ATTRIBUTE variable and make the feature enabled by default.

Merge requirement checklist

• CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
• Unit tests added/updated
• Appropriate CHANGELOG.md files updated for non-trivial changes
• Changes in public API reviewed (if applicable)

[codecov]

Codecov Report

Attention: Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 86.27%. Comparing base (6250307) to head (e4e21ca).
Report is 360 commits behind head on main.

Files with missing lines	Patch %	Lines
src/OpenTelemetry/Metrics/AggregatorStore.cs	77.77%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5909      +/-   ##
==========================================
+ Coverage   83.38%   86.27%   +2.88%     
==========================================
  Files         297      260      -37     
  Lines       12531    11437    -1094     
==========================================
- Hits        10449     9867     -582     
+ Misses       2082     1570     -512     

Flag	Coverage Δ
unittests	?
unittests-Project-Experimental	86.23% <80.00%> (?)
unittests-Project-Stable	86.21% <80.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/OpenTelemetry/Metrics/MeterProviderSdk.cs	93.38% <100.00%> (+5.54%)	⬆️
src/OpenTelemetry/Metrics/Metric.cs	97.27% <ø> (+0.75%)	⬆️
...rc/OpenTelemetry/Metrics/Reader/MetricReaderExt.cs	92.03% <ø> (ø)
src/OpenTelemetry/Metrics/AggregatorStore.cs	86.15% <77.77%> (+5.76%)	⬆️

... and 233 files with indirect coverage changes

[cijothomas]

Changes look good overall.
I am requesting couple of changes

1. Remove internal log for cardinality hit.
2. Rephrase changelog.

(I left detailed comment about each of above in PR).

Also, we could use some refactoring on the docs, but this can be followed up separately. If you need help, I can help with the doc part.

[reyang]

LGTM with a minor suggestion.

[cijothomas]

i think its best to mention the pre 1.10.0 behavior too here.
Versions prior to 1.10.0 had a different behavior when cardinality limit was hit. The measurements was either dropped (default) and an internal log was emitted once or aggregated using overflow attribute(opt-in basis)

(we have some places in this repo where were use such style in document)

[rajkumar-rangaraj]

I feel it's better if the README reflects the current scenario. Adding prior behavior may increase the length of the document.

[CodeBlanch]

I think the whole thing is a bit confusing 🤣

Here is my stab at improving it:

Given a metric, once the cardinality limit is reached, any new measurement which cannot be independently aggregated because of the limit will be dropped. Starting with version 1.10.0, when a measurement is dropped, the the overflow attribute is updated automatically.

I think what is important is the first statement "Given a metric, once the cardinality limit is reached, any new measurement which cannot be independently aggregated because of the limit will be dropped." That is a hard stop thing. The way it is currently written, it seems this is new to 1.10.0 😄

[xiang17]

I feel this is tricky. One is the Previous behavior where there is default and experimental opt-in overflow attribute. The other one is the New behavior with the always on overflow attribute. It will also be too long if we explain everything in README.

I've changed the README to include minimal info to mention things has changed a few times, and put more details in the CHANGELOG in case anyone is really interested and want to better understand.

[xiang17]

I think what is important is the first statement "Given a metric, once the cardinality limit is reached, any new measurement which cannot be independently aggregated because of the limit will be dropped." That is a hard stop thing. The way it is currently written, it seems this is new to 1.10.0 😄

That's a good point. I've updated it to make that first statement more prominent and followed with how our SDK handles and how approaches changed.

One small thing, the spec seems to not see it as a "drop" when it's done with the overflow attribute, but rather a "synthetic aggregation". So it's still "aggregated", but not "independently aggregated".

An overflow attribute set is defined, containing a single attribute otel.metric.overflow having (boolean) value true, which is used to report a synthetic aggregation of the Measurements that could not be independently aggregated because of the limit.

[cijothomas]

I have some suggestions on changelog and readme/doc part, rest looks good.
They can be a follow up PR too.