fix for ignoring recording status of spans

[jackshirazi]

1. APMTracer receives the parent task's trace context and sets that as the parent of a new span.

            final Context parentContext = getParentContext(traceContext);
                spanBuilder.setParent(parentContext);

2. APMTracer requests a new span, setting the start time to the same as the transaction start time

            Instant startTime = traceContext.getTransient(Task.TRACE_START_TIME);
                spanBuilder.setStartTimestamp(startTime);
            final Span span = spanBuilder.startSpan();

3. The span is created but is too many for the transaction because transaction_max_spans is exceeded for this span, so it is set as a non-recording span
4. APMTracer stores the Context containing the span in its spans map, ie this new span is now the next parent span so will be stored as the task's trace context

• this is incorrect because the span is a non-recording span, so should be discarded
• the result of the incorrect approach is that you can have a chain of descendent spans in a transaction which can be unlimited in size, and all with the same start time

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mosche]

Unfortunately that would spam our logs massively thinking about the 500k discarded spams we've just seen.
I'll have a closer look, I think we need to track when we transition to not recording in the context somehow.

[mosche]

Thanks a lot and great find @jackshirazi 💚
I've opened #137015 including your fix to address the necessary changes in tests needed for this. (Since this PR was opened against your main branch of your fork, I wasn't able to push to this PR)