[XRay Sampler] - XRay Sampler Statistics reports much higher RequestCount/SampleCount than expected for Sampling Rules

[jj22ee]

There is a bug of XRay Sampler Implementation in all OTel SDK Languages. This impacts Sampling Statistics reported by XRay sampler, particularly the SamplingStatisticsDocuments’s metrics of [BorrowCount, RequestCount, SampledCount], which is sent to XRay Service via the /SamplingTargets endpoint.

Example: If a single request to an Application using XRay Sampler creates 1 Server Span (makes decision Sampled=true) with 8 child Internal Spans (copies parent's Sampled value) :

• Actual: XRay Sampler will produce Sampler Metrics of RequestCount=9, SampledCount=9
  • XRay Sampler produces Sampler metrics based on number of Spans of any kind
• Expected: XRay Sampler should produce Sampler Metrics of RequestCount=1, SampledCount=1
  • XRay Sampler should produce Sampler metrics based on number of Traces (aka the root Span that makes the Sampling Decision)

In summary, the bug is that while Spans with a parent Span will copy their parent’s XRay Sampling Decision, they will still make additions to the XRay Sampling metric (they shouldn’t because they do not make the decision, only copy it)

How does this negatively affect OTel users of XRay Sampler?

• If user observes Sampling Rules Console, a Sampling Rule’s Trend Graph will report metric numbers that are much greater than the actual number of Traces sampled.

Given the above example, the graph will incorrectly show 9 req matching the rule, with 9 sampled instead of the actual 1 req matching the rule, with 1 sampled.

Root cause should be because only subcomponents of XRay Sampler uses ParentBased logic to short-circuit the Parent's Sampling Decision, while the XRay Sampling Statistics recording logic is not skipped when a Parent's Sampling Decision is found.

• Thus the fix is to wrap the entire XRay Sampler Internal Logic under a single ParentBased Sampler, which will skip recording the "extra" sampling statistics when a Parent's Sampling Decision is available.

[jj22ee]

In Python, this should also significantly reduce statistics lock contention in the XRay Sampler when there are a lot of child spans.