Add specs for host.id and profiler registration message

[JonasKunz]

The UI for profiling data relies on the host.id field for certain calculations, including the cost and CO2-consumption estimates. In order to correctly provide those values for profiling data stored on transaction documents, those need to have the host.id too.

Therefore this spec adds the host.id as an optional, new metadata field for IntakeV2. The field is only required to be populated when an agent actually is correlating transactions with profiling data.

Because the otel-compliant host.id can sometimes only be derived correctly with root permissions, this spec change also adds a way of receiving the host.id directly from the profiling host agent.

• Create PR as draft
• Approval by at least one other agent
• Mark as Ready for Review (automatically requests reviews from all agents and PM via CODEOWNERS)
  • Remove PM from reviewers if impact on product is negligible
  • Remove agents from reviewers if the change is not relevant for them
• Merge after 2 business days passed without objections
  To auto-merge the PR, add /schedule YYYY-MM-DD to the PR description.

[danielmitterdorfer]

As we have two agents involved here (the APM agent and the profiling host agent) we should always qualify which agent we actually mean. Also there are inconsistencies in terminology throughout the document. We call the APM agent:

• APM agent
• APM-agent
• agent

and the profiling host agent:

• profiling host agent
• profiler host agent
• host agent

[JonasKunz]

I've clarified the "agent" usages in this section.
For the rest of the document, the inconsistencies come from the fact that this document was originally intended as only an APM-agent spec. This metadata spec is also becoming less important as we move to Otel SemConv, which effectively will replace this spec.

I'm just adding the host.id here, because we intend to also add the universal profiling integration to the old elastic-apm-agent java to allow easier adoption for existing users.

[danielmitterdorfer]

Thanks! LGTM

[gregkalapos]

As discussed in the agents weekly: host.id as a new metadata field on intake lgtm.

[SylvainJuge]

[nit] Do we have use-cases where the agent should populate this TLS without waiting for the registration message ? If so, then a "SHOULD" sounds more appropriate as updating this TLS seems useless if the profiler is not available.

[JonasKunz]

The use case would be if you collect traces for the application startup and what to have profiling data for those at the very beginning:
There will be a short delay between application startup and receiving this initial registration message. As a result, spans started and activated before that receival would not get correlated.
This can be overcome for those edge cases by eagerly populating the TLS, even if it is not known whether a profiler is already there.

I'm planning to implement this by having a tri-state enabled config option:

• false: No correlation, the native lib won't even be loaded
• true: Correlation active, TLS will be eagerly populated
• auto (default for OTel-extension): Correlation will be active, TLS will only be populated after receiving the profiler registration message (So basically close to zero overhead if no profiler is active)

[SylvainJuge]

Does the W3C spec contains anything about the "transactions" ? From what I recall it's only about tracestate and traceparent (with a trace-id and parent-id as fields) (ref) but I might definitely have missed something.

[JonasKunz]

Ooops, this should be The APM W3C span id of the transaction

[athre0z]

👍

[trentm]

As mentioned at elastic/beats#38689 (comment) using host.id for host correlation is probably the best path for future host correlation. Should we get a meta issue so agents other than the Java agent start adding host.id?

[JonasKunz]

The agent also won't always collect the host.id, it only does so if the profiler is active and the profiler provides the host.id. The problem with host.id is that sometimes it might not be possible to discover the host.id without root permissions, at least accoding to the Otel SemConv specs.

[xrmx]

The agent also won't always collect the host.id, it only does so if the profiler is active and the profiler provides the host.id.

That's not what I get from the specs. If we shouldn't provide it ourselves without the profiler it would be nice to write it explicitly.

[JonasKunz]

The agent also won't always collect the host.id, it only does so if the profiler is active and the profiler provides the host.id.

Sorry, I was referring to the planned implementation details of the java-agent.

The current wording is

APM agents MAY collect the host.id as an unique identifier for the host.

This means agents are not required to do so, but might if they want to. It should be changed to MUST if we desire to force all agents to provide the host.id.