APM telemetry collection Otel agent distro diferentiation - issue 489

[marcogavaz]

Summary

This PRs follows the closed PR and Closes https://github.com/elastic/observability-bi/issues/489

As requested in this comment and tracked by the issue #186281 this PR handles main changes introduced to address the request for capturing open-ended OTel distro agent names with patter opentelemetry/<LANGUAGE>/<DISTRO_NAME>. These changes ensure that new agent names won’t be dropped in telemetry.

Changes introduced in services task

Previously telemetry code enumerated OTel agents using a static array (OPEN_TELEMETRY_AGENT_NAMES) listing known agent names (like opentelemetry/java, opentelemetry/go, etc.), while now it starts with an empty object and dynamically add whatever agent names es query returns, so after retrieving the results, we let the data from the terms aggregator define which agent names exist, rather than forcing them to match any fixed array.

This is taken care mainly in:

    const servicesPerOtelAgents = await OPEN_TELEMETRY_BASE_AGENT_NAMES.reduce(
        (prevJob, baseAgentName) => {
          return prevJob.then(async (accData) => {
            const response = await telemetryClient.search({
              index: [indices.error, indices.span, indices.metric, indices.transaction],
              body: {
                size: 0,
                track_total_hits: false,
                timeout,
                query: {
                  bool: {
                    filter: [{ prefix: { [AGENT_NAME]: baseAgentName } }, range1d],
                  },
                },
                aggs: {
                  agent_name: {
                    terms: {
                      field: AGENT_NAME,
                      size: 1000,
                    },
                    aggs: {
                      services: {
                        cardinality: {
                          field: SERVICE_NAME,
                        },
                      },
                    },
                  },
                },
              },
            });
            const aggregatedServices: Record<string, number> = {};
            for (const bucket of response.aggregations?.agent_name.buckets ?? []) {
              const fullAgentName = bucket.key as string;
              aggregatedServices[fullAgentName] = bucket.services.value || 0;
            }
            return {
              ...accData,
              ...aggregatedServices,
            };
          });
        },
        Promise.resolve({} as Record<string, number>)
      );

so for example if a new agent.name like opentelemetry/java/elastic appears in the data, we automatically add it to aggregatedServices

Changes introduced in agents task

Similarly telemetry code started with a large dictionary built from OPEN_TELEMETRY_AGENT_NAMES and then for each prefix opentelemetr or otlp it aggregated subfields (framework, language, runtime versions) and tried to map them to the dictionary.
While now after the prefix-based aggregator runs it look at the returned buckets. For each agentBucket (with a key opentelemetry/<LANGUAGE>/<DISTRO_NAME> ) an entry is dynamically created.

      const agentDataWithOtel = await OPEN_TELEMETRY_BASE_AGENT_NAMES.reduce(
        async (prevJob, baseAgentName) => {
          const data = await prevJob;

          const response = await telemetryClient.search({
            index: [indices.error, indices.metric, indices.transaction],
            body: {
              track_total_hits: false,
              size: 0,
              timeout,
              query: {
                bool: {
                  filter: [{ prefix: { [AGENT_NAME]: baseAgentName } }, range1d],
                },
              },
              sort: {
                '@timestamp': 'desc',
              },
              aggs: {
                agent_name: {
                  terms: {
                    field: AGENT_NAME,
                    size: 1000,
                  },
                  aggs: agentNameAggs,
                },
              },
            },
          });

This PR should be enough in order to be able to differentiate our distro from any other OTel distro as requested.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[marcogavaz]

@elasticmachine merge upstream

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 982d280

Metrics [docs]

✅ unchanged

History

• 💚 Build #275614 succeeded f95e2b6

cc @marcogavaz

[kpatticha]

LGTM. Nice work.

it would be great if you could update the PR description to include a short summary (for future reference_

[trentm]

FYI: I discussed with @basepi about backporting this to 8.19. He agrees that having this in 8.19 (along with 9.1) would be "just fine with us [BI team]".

My motivation for backporting this is so that x-pack/solutions/observability/plugins/apm/server/lib/apm_telemetry/collect_data_telemetry/__snapshots__/tasks.test.ts.snap is the same in 8.19 and 9.1 (main) branches so that backporting changes that touch that file don't run into merge conflicts. My recent #222883 has run into a merge conflict. There will be more similar PRs coming (e.g. this one in draft: #223565).

I'm not sure if adding the "8.19" label to the PR after it has been merged will trigger automatic backporting PR generation. I will try that. Failing that, I'll use ./scripts/backport manually.

[trentm]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation