host.name assigned with pod.name

[crespocarlos]

Describe the bug

When viewing the Hosts UI with Go APM service active in a k8s pod, Go agent monitored services report the pod name in the host.name, which makes the Hosts View display hosts that do not correlate with hosts monitored by the system integration.

Steps to reproduce

• Run a Go APM instrumented service in a k8s pod
• Ingest that into ES
• Open the Infra UI

edge-oblt can be used for the above.

Expected behavior

Only hosts (VMs) are expected to be present in the Hosts View.

Additional context

Sample doc

{
  "_index": ".ds-metrics-apm.internal-default-2024.07.10-000004",
  "_id": "n6S5wJABi-i5Y9bCcNyn",
  "_score": 0,
  "_source": {
    "agent": {
      "name": "go",
      "version": "2.6.0"
    },
    "cloud": {
      "availability_zone": "us-central1-c",
      "instance": {
        "id": "5320670725633961051",
        "name": "gke-edge-oblt-edge-oblt-pool-8fc2868f-9w9b"
      },
      "machine": {
        "type": "e2-highcpu-4"
      },
      "project": {
        "id": "elastic-observability"
      },
      "provider": "gcp",
      "region": "us-central1"
    },
    "data_stream": {
      "dataset": "apm.internal",
      "namespace": "default",
      "type": "metrics"
    },
    "host": {
      "architecture": "amd64",
      "hostname": "opbeans-go-66d4d66948-288rd",
      "ip": "35.225.250.101",
      "name": "opbeans-go-66d4d66948-288rd",
      "os": {
        "platform": "linux"
      }
    },
    "metricset": {
      "name": "span_breakdown"
    }
  }
}
{
  "_index": ".ds-metrics-apm.internal-default-2024.06.19-000001",
  "_id": "fhKOVJAB-A68Bg7Z5cK5",
  "_score": 0,
  "_source": {
    "agent": {
      "name": "go",
      "version": "2.6.0"
    },
    "cloud": {
      "availability_zone": "us-central1-c",
      "instance": {
        "id": "5587317589758934282",
        "name": "gke-edge-oblt-edge-oblt-pool-8fc2868f-95sb"
      },
      "machine": {
        "type": "e2-highcpu-4"
      },
      "project": {
        "id": "elastic-observability"
      },
      "provider": "gcp",
      "region": "us-central1"
    },
    "data_stream": {
      "dataset": "apm.internal",
      "namespace": "default",
      "type": "metrics"
    },
    "host": {
      "architecture": "amd64",
      "hostname": "auditbeat-7zqt2",
      "ip": "35.239.183.88",
      "name": "auditbeat-7zqt2",
      "os": {
        "platform": "linux"
      }
    },
    "metricset": {
      "name": "span_breakdown"
    }
  }
}

[dmathieu]

I'm not seeing anything that sets host.name within this agent's codebase. Could that be set elsewhere?

[crespocarlos]

I found one service that's sending the right host name:

{
  "_index": ".ds-metrics-apm.internal-default-2024.07.10-000004",
  "_id": "r6G4wJABi-i5Y9bCcAks",
  "_score": 0,
  "_source": {
    "@timestamp": "2024-07-17T12:43:49.227Z",
    "agent": {
      "name": "go",
      "version": "2.6.0"
    },
    "cloud": {
      "availability_zone": "us-central1-c",
      "instance": {
        "id": "5320670725633961051",
        "name": "gke-edge-oblt-edge-oblt-pool-8fc2868f-9w9b"
      },
      "machine": {
        "type": "e2-highcpu-4"
      },
      "project": {
        "id": "elastic-observability"
      },
      "provider": "gcp",
      "region": "us-central1"
    },
    "data_stream": {
      "dataset": "apm.internal",
      "namespace": "default",
      "type": "metrics"
    },
    "host": {
      "architecture": "amd64",
      "hostname": "gke-edge-oblt-edge-oblt-pool-8fc2868f-9w9b",
      "ip": "35.225.250.101",
      "name": "gke-edge-oblt-edge-oblt-pool-8fc2868f-9w9b",
      "os": {
        "platform": "linux"
      }
    },
    "metricset": {
      "name": "span_breakdown"
    },
    "observer": {
      "hostname": "3dfdd56aefd7",
      "type": "apm-server",
      "version": "8.16.0"
    },
    "process": {
      "args": [
        "--strict.perms=false",
        "-E",
        "-c",
        "-e",
        "/etc/packetbeat.yml",
        "packetbeat",
        "packetbeat.interfaces.device=eth0"
      ],
      "parent": {
        "pid": 7
      },
      "pid": 13,
      "title": "packetbeat"
    },
    "processor": {
      "event": "metric"
    },
    "service": {
      "environment": "production",
      "language": {
        "name": "go",
        "version": "go1.21.11"
      },
      "name": "packetbeat",
      "node": {
        "name": "gke-edge-oblt-edge-oblt-pool-8fc2868f-9w9b"
      },
      "runtime": {
        "name": "gc",
        "version": "go1.21.11"
      },
      "version": "8.15.0"
    },
    "span": {
      "self_time": {
        "count": 352,
        "sum": {
          "us": 24374080
        }
      },
      "subtype": "elasticsearch",
      "type": "db"
    },
    "transaction": {
      "name": "publish",
      "type": "output"
    }
  }
}

[crespocarlos]

The same node is running auditbeat and packetbeat. packetbeat reports the correct host name, auditbeat doesn't. I can't find what is different between the two that causes this. Both are running the same go agent version

[simitt]

The go agent doesn't set the host.name, it sends a configured or detected name and then the apm-server has an algorithm to derive host.name depending on various factors, as described in elastic/apm-server#8118 (comment).

https://www.elastic.co/guide/en/apm/agent/go/current/configuration.html#config-hostname describes how a host name can be configured via the go agent. You'll need to check the setup of the instrumented services as to which configurations they have set.

[crespocarlos]

I had a quick call with @Mpdreamz , and he explained that the non-matching host names shipped by APM agents are due to missing configuration as exemplified in this comment https://github.com/elastic/observability-dev/issues/3321#issuecomment-2236909682.