fix: span compression bug where a buffered span would not be sent when an incompressible sibling ended

[trentm]

https://github.com/elastic/apm/blob/main/specs/agents/handling-huge-traces/tracing-spans-compress.md#span-buffering says:

A buffered span gets reported when

1. its parent ends
2. a non-compressible sibling ends

Before this change, case 2 was not being handled. Further, a possibly
buffered span on the ending sibling would incorrectly get sent twice.
The scenario is as follows:

   trans t0
     span s1: _bufferdSpan=s2
       span s2: ended, compressible -> buffered on s1
       span s3: incompressible, _bufferdSpan=s4
         span s4: ended, compressible -> buffered on s3

What happens when s3 ends? We expect:

• s4 is encoded and sent
• s2 is encoded and sent
• s3 is encoded and sent

Before this change the agent would send: s4, s4, s3

Refs: https://discuss.elastic.co/t/apm-agent-send-duplicate-record-in-mysql/321440

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-12-20T20:18:55.528+0000

• Duration: 28 min 38 sec

Test stats 🧪

Test	Results
Failed	0
Passed	320238
Skipped	0
Total	320238

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run module tests for <modules> : Run TAV tests for one or more modules, where <modules> can be either a comma separated list of modules (e.g. memcached,redis) or the string literal ALL to test all modules

• run benchmark tests : Run the benchmark test only.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[trentm]

REVIEW NOTE: This clearing avoids the bug where we would encode and send a span twice -- the first time here, the second time in the next if-block.

[trentm]

REVIEW NOTE: This corrects the logic to make the spec pseudo code: this block is meant to be able looking at the possibly buffered span on the parent. This fixes the case where a buffered span would not get sent if it was followed by an incompressible sibling span.

[trentm]

Here is a snippet of the output of a test run showing the newly added test failing before the fix:

# ensure a span buffered on parent is sent when incompressible span ends
ok 18 s2 is a child of s1
ok 19 s2 is buffered on s1
ok 20 s3 is a child of s1
ok 21 s4 is a child of s3
ok 22 s4 is buffered on s3
ok 23 3 events sent
ok 24 3 spans sent
not ok 25 sent spans are s4,s2,s3; in that order
  ---
    operator: equal
    expected: 's4,s2,s3'
    actual:   's4,s4,s3'
    at: <anonymous> (/Users/trentm/el/apm-agent-nodejs6/test/instrumentation/span-compression.test.js:185:9)
    stack: |-
      Error: sent spans are s4,s2,s3; in that order
          at Test.assert [as _assert] (/Users/trentm/el/apm-agent-nodejs6/node_modules/tape/lib/test.js:312:48)
          at Test.bound [as _assert] (/Users/trentm/el/apm-agent-nodejs6/node_modules/tape/lib/test.js:95:17)
          at Test.strictEqual (/Users/trentm/el/apm-agent-nodejs6/node_modules/tape/lib/test.js:476:7)
          at Test.bound [as equals] (/Users/trentm/el/apm-agent-nodejs6/node_modules/tape/lib/test.js:95:17)
          at /Users/trentm/el/apm-agent-nodejs6/test/instrumentation/span-compression.test.js:185:9
          at Timeout._onTimeout (/Users/trentm/el/apm-agent-nodejs6/test/_mock_http_client.js:80:7)
          at listOnTimeout (node:internal/timers:559:17)
          at processTimers (node:internal/timers:502:7)
  ...

1..25
# tests 25
# pass  24
# fail  1

[trentm]

[email protected] is published with this fix now.