Otel agent distro diferentiation in APM telemetry collection

[marcogavaz]

Summary

As requested in this comment and tracked by the issue #186281 this PR handles main changes introduced to address the request for capturing open-ended OTel distro agent names with patter opentelemetry/<LANGUAGE>/<DISTRO_NAME>. These changes ensure that new agent names won’t be dropped in telemetry.

The tasks that were changed are services and agents task.

Changes introduced in services task

Previously telemetry code enumerated OTel agents using a static array (OPEN_TELEMETRY_AGENT_NAMES) listing known agent names (like opentelemetry/java, opentelemetry/go, etc.), while now it starts with an empty object and dynamically add whatever agent names es query returns, so after retrieving the results, we let the data from the terms aggregator define which agent names exist, rather than forcing them to match any fixed array.

This is taken care mainly in

const servicesPerOtelAgents = await OPEN_TELEMETRY_BASE_AGENT_NAMES.reduce(
  (prevJob, baseAgentName) => {
    return prevJob.then(async (accData) => {
      const response = await telemetryClient.search({
        // same prefix query...
        aggs: {
          agent_name: {
            terms: {
              field: AGENT_NAME,
              size: 1000,
            },
            aggs: {
              services: {
                cardinality: {
                  field: SERVICE_NAME,
                },
              },
            },
          },
        },
      });

      const aggregatedServices: Record<string, number> = {};

      // For each agent_name bucket we actually find...
      for (const bucket of response.aggregations?.agent_name.buckets ?? []) {
        // could be "opentelemetry/java/elastic"
        aggregatedServices[bucket.key as string] = bucket.services.value || 0;
      }

      // merge into accumulated data
      return { ...accData, ...aggregatedServices };
    });
  },
  Promise.resolve({} as Record<string, number>)
);

so for example if a new agent.name like opentelemetry/java/elastic appears in the data, we automatically add it to aggregatedServices

Changes introduced in agents task

Similarly telemetry code started with a large dictionary built from OPEN_TELEMETRY_AGENT_NAMES and then for each prefix opentelemetr or otlp it aggregated subfields (framework, language, runtime versions) and tried to map them to the dictionary.
While now after the prefix-based aggregator runs it look at the returned buckets. For each agentBucket (with a key opentelemetry/<LANGUAGE>/<DISTRO_NAME> ) an entry is dynamically created.

      const agentDataWithOtel = await OPEN_TELEMETRY_BASE_AGENT_NAMES.reduce(
        async (prevJob, baseAgentName) => {
          const data = await prevJob;
  
          const response = await telemetryClient.search({
            index: [indices.error, indices.metric, indices.transaction],
            body: {
              track_total_hits: false,
              size: 0,
              timeout,
              query: {
                bool: {
                  filter: [{ prefix: { [AGENT_NAME]: baseAgentName } }, range1d],
                },
              },
              sort: {
                '@timestamp': 'desc',
              },
              aggs: {
                agent_name: {
                  terms: {
                    field: AGENT_NAME,
                    size: 1000,
                  },
                  aggs: agentNameAggs,
                },
              },
            },
          });
  
          if (!response.aggregations) {
            return data;
          }
  
          const dynamicAgentData: NonNullable<APMTelemetry['agents']> = {};
  
          for (const agentBucket of response.aggregations.agent_name.buckets) {
            const agentKey = agentBucket.key as string;
  
            dynamicAgentData[agentKey] = {

This PR should be enough in order to be able to differentiate our distro from any other OTel distro as requested.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[marcogavaz]

Closes https://github.com/elastic/observability-bi/issues/489

[kpatticha]

The code looks good to me, I tested it locally and it works as expected

I left few comments related to the types.

[kpatticha]

Not related to the work of this PR but I've noticed the following code.

 const servicesPerAgents: Record<AgentName, number> = {
        ...servicesPerAgentExcludingOtel,
        ...servicesPerOtelAgents,
      };

      return {
        has_any_services_per_official_agent: sum(Object.values(servicesPerAgents)) > 0,
        has_any_services: services?.hits?.total?.value > 0,
        services_per_agent: servicesPerAgents,
      };

and when I run my local environment with only otel data (using synthrace) I got the following response

"has_any_services_per_official_agent": true,
"has_any_services": true,
"services_per_agent": {
"dotnet": 0,
"go": 0,
"iOS/swift": 0,
"java": 0,
"js-base": 0,
"nodejs": 0,
"php": 0,
"python": 0,
"ruby": 0,
"rum-js": 0,
"android/java": 0,
"ios/swift": 0,
"otlp/elastic": 1,
"opentelemetry/java/elastic": 1
},

I would expect the has_any_services_per_official_agent to be false.

cc @marcogavaz @basepi

[marcogavaz]

The code looks good to me, I tested it locally and it works as expected

I left few comments related to the types.

Great, i'll integrate your hints asap! Could we have maybe a meeting together where you show me how you test in local? (like on which data, how you set up the deployment, ...)? It would be great so i can bring also this new knowledge to the rest of observability bi team. I'll send you an invitation!

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: fc46d50

Failed CI Steps

• Jest Tests #3
• Jest Tests #3

Test Failures

• [job] [logs] Jest Tests #3 / data telemetry collection tasks agents should return agent data per agent name
• [job] [logs] Jest Tests #3 / data telemetry collection tasks agents should return agent data per agent name
• [job] [logs] Jest Tests #3 / data telemetry collection tasks services should return services per agent name
• [job] [logs] Jest Tests #3 / data telemetry collection tasks services should return services per agent name

Metrics [docs]

✅ unchanged

History

• 💔 Build #274784 failed 340325a
• 💔 Build #273580 failed b704464cd009654208447e2a9554abbf05e68f58
• 💔 Build #273559 failed d829eb6

cc @marcogavaz