Enabling Exemplars in Prometheus exporter

[irizzant]

Is your feature request related to a problem? Please describe.
I have deployed and enabled the Java agent for auto instrumentation and enabled Prometheus exporter with system property -Dotel.metrics.exporter=prometheus.
I cannot see any Exemplars in the metrics exported.

Describe the solution you'd like
I'd like to have Exemplars in the metrics exported by Prometheus exporter.

[trask]

@anuraaga @jkwatson can you transfer this to https://github.com/open-telemetry/opentelemetry-java?

@irizzant once this is implemented in the prometheus exporter, it should get automatically picked up in the following java agent release.

[irizzant]

Thanks for the repy @trask , so if I understand correctly is prometheus-exporter that currently does not implement this feature?

If I use the otel exporter with otel.metrics.exporter=otlp am I going to have exemplars exported instead?

[trask]

oh, it looks like the prometheus exporter has exemplar support already. @jsuereth have you happened to test the java agent + prometheus + exemplars?

[irizzant]

@trask @jsuereth I'd really appreciate it if you could share some guidance about enabling exemplars with Prometheus exporter and maybe update the docs

[irizzant]

@trask @jsuereth any update?

[trask]

hi @irizzant, can you test out using OpenTelemetry SDK + Prometheus exporter directly to help narrow down where the issue is?

[zmapleshine]

Hello @irizzant Have you accessed the prometheus port from the browser address bar to view?
Exemplar data acquisition need to request header contains:

Accept:application/openmetrics-text

You can refer to io.prometheus.client.exporter.common.TextFormat

public static String chooseContentType(String acceptHeader) {
    if (acceptHeader == null) {
      return CONTENT_TYPE_004;
    }

    for (String accepts : acceptHeader.split(",")) {
      if ("application/openmetrics-text".equals(accepts.split(";")[0].trim())) {
        return CONTENT_TYPE_OPENMETRICS_100;
      }
    }

    return CONTENT_TYPE_004;
  }

In addition, the exemplar data has a sampling policy, the default policy is "with_sampled_trace".

[irizzant]

Thanks @trask @zmapleshine for your replies.

@trask

hi @irizzant, can you test out using OpenTelemetry SDK + Prometheus exporter directly to help narrow down where the issue is?

As I originally wrote my Java application is not using SDK but the auto-instrumentation agent, I configured the exporter with the following properties:

-javaagent:opentelemetry-javaagent.jar
-Dotel.service.name=webportal
-Dotel.exporter.otlp.endpoint=http://distributor.grafana-test.svc.cluster.local:55680
-Dotel.traces.exporter=otlp,logging
-Dotel.exporter.otlp.compression=gzip
-Dotel.metrics.exporter=prometheus

As you can see I'm using the Prometheus exporter directly offered by the agent.

@zmapleshine

Hello @irizzant Have you accessed the prometheus port from the browser address bar to view?

Nope, I run a simple curl inside the container where the app is running including the custom header you mentioned:

kubectl -n eap exec -it wp-687dbdbbd5-k44hc -- curl -H "Accept: application/openmetrics-text" http://localhost:9464/metrics

I get the metrics but no trace info.

I also tried with otel.metrics.exemplar.filter=NONE and otel.metrics.exemplar.filter=ALL but I get no traces.

[trask]

hey @irizzant, we figured out the problem today. the agent is not passing the context when recording metrics (and there's no active context at that time, at least for server metrics), which is why exemplars aren't captured. will plan to fix this before 1.9.0 release next week.

[irizzant]

Hi @trask , perfect thank you

[irizzant]

Hi @trask @laurit , is there a release planned with this fix?

[laurit]

@irizzant we hope to release this week

[irizzant]

@laurit ok thank you

[trask]

@irizzant 1.9.0 is released, thanks again for reporting this!

[irizzant]

@trask thank you!

[nutslove]

@irizzant @trask
Is this issue solved?
I still can not see any exemplars in the metrics exported using v1.9.1.
※I checked from the browser(:9464/metrics).
Here are my properties.

-javaagent:~/spring-petclinic/opentelemetry-javaagent.jar
-Xshare:off
-Dotel.exporter.otlp.endpoint=http://<AWS ELB>:4317
-Dotel.metrics.exporter=prometheus
-Dotel.resource.attributes=service.name=test_20211205_02
-jar target/spring-petclinic-2.5.0-SNAPSHOT.jar

[irizzant]

@nutslove I tested it and it works fine.
Please check this comment and make sure you also send the required http header to export metrics in OpenTelemetry format.

curl -H "Accept: application/openmetrics-text"