OpenTelemetry SDK autoconfiguration ignores OTEL service name in favor of Quarkus app name

[michalvavrik]

Describe the bug

With #30033 I expected autoconfiguration work for properties listed here https://github.com/quarkusio/quarkus/blob/main/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/config/OTelFallbackConfigSourceInterceptor.java#L85 and OTEL documentation says that otel.service.name should have preference over otel.resource.attributes. However, Quarkus OpenTelemetry extension creates Tracer resource directly and is using Quarkus application name (or artifact if not available)

quarkus/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/tracing/TracerRecorder.java

Line 52 in 43a5358

ResourceAttributes.SERVICE_NAME, serviceName,

quarkus/extensions/opentelemetry/deployment/src/main/java/io/quarkus/opentelemetry/deployment/tracing/TracerProcessor.java

Line 158 in 43a5358

return new TracerResourceBuildItem(recorder.createResource(Version.getVersion(), serviceName, serviceVersion));

Expected behavior

Quarkus documentation says otel.service.name should have priority

quarkus/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/config/runtime/OTelRuntimeConfig.java

Line 52 in 43a5358

* Specify logical service name. Takes precedence over service.name defined with otel.resource.attributes

Actual behavior

Quarkus application name is used.

How to Reproduce?

Define otel.service.name and check traces (I'm using Jaeger). IMHO it's straight forward, but ping me if you really need one.

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

999-SNAPSHOT

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @brunobat (opentelemetry), @radcortez (opentelemetry)

[brunobat]

otel.service.name is not even being mapped. We use the standard quarkus.application.name for quarkus.

Somewhere around here:

quarkus/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/OpenTelemetryProducer.java

Line 86 in 43a5358

Resource consolidatedResource = existingResource.merge(

We should map otel.service.name into the oTelConfigs and agree on a priority for it, in runtime. Something like:

quarkus.application.name > otel.service.name > otel.resource.attributes

Test for the resource:

quarkus/extensions/opentelemetry/deployment/src/test/java/io/quarkus/opentelemetry/deployment/OpenTelemetryResourceTest.java

Line 54 in 3e6b2a5

assertEquals("authservice", server.getResource().getAttribute(AttributeKey.stringKey("service.name")));

[michalvavrik]

otel.service.name is not even being mapped. We use the standard quarkus.application.name for quarkus. We should map otel.service.name into the oTelConfigs and agree on a priority for it, in runtime. Something like:

I appreciate feedback @brunobat , but I am not sure I understand above-mentioned. If it is not even being mapped, what is this property?

quarkus/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/config/runtime/OTelRuntimeConfig.java

Line 52 in 43a5358

* Specify logical service name. Takes precedence over service.name defined with otel.resource.attributes

It exactly matches pattern otel.service.name plus quarkus. prefix. And it's description mentions opposite priorities, loose interpretation:

otel.service.name > quarkus.application.name > otel.resource.attributes

I think what you meant to say is that we simply need to use it somewhere around the line you linked to, correct?

[brunobat]

yeah, it is mapped. I just forgot about it. Sorry.
The current property is mapped like that: otel.service.name > quarkus.application.name > otel.resource.attributes... I have no strong feelings in relation to the first 2 priorities.
Saying this...
This might be a mapping failure at the OTelFallbackConfigSourceInterceptor, otherwise I'm not sure what we are doing wrong because the resource prop and the config props are being sent to the OTel SDK that interprets them...
A new IT test should clear things out.

[michalvavrik]

Thanks, makes sense, I'll look at this next week unless someones quicker.

[geoand]

@michalvavrik are you still planning on tackling this one?

[michalvavrik]

Thanks for the reminder, I'll take care of it this week.

[jorgebsa]

Are you sure this has been fixed? I'm trying to use the service name on 3.2.2 and 3.2.1, and on neither it takes precedence over the application's name. I've tried setting it via config file, via -D, via env vars, running via qaurkus:dev or built jib image. I always end up with the app name on the traces and jaeger.

I've put breakpoints on the changed lines of #34581. On OpenTelemetryProducer I can see that the service name is loaded from the property, but ends up discarded on the merged resource. On TraceUtil, the debug never stopped on the serviceName nullability check.

[brunobat]

There is a test for this here: https://github.com/michalvavrik/quarkus/blob/main/extensions/opentelemetry/deployment/src/test/java/io/quarkus/opentelemetry/deployment/OpenTelemetryServiceNameTest.java

Please note that the service name from OTel:
quarkus.otel.service.name is different from the one loaded in resources: quarkus.otel.resource.attributes=service.name=cats

[jorgebsa]

I've seen the test before, but not sure I understand what you mean at the end, as I'm relatively new to OTel.

Let me rephrase what I said and then maybe you can point out if I'm making completely wrong assumptions.

On the official OTel guide, there's a mention to the quarkus.otel.service.name config property that is meant to take precedence over other possible values, including the quarkus.application.name.

By reading its description, I've assumed that if I were to set this value to something else different than the app's name, then the traces generated by quarkus would use such value as the service name on the traces, such that I could find this service name on Jaeger.

This is important to me, as I'm working on (multiple) Quarkus service(s) running in multiple different environments with the same image built by Jib, therefore they all have the artifact id as the(ir) quarkus app name.

The tracing data of all environments is collected by jaeger collectors and stored on the platform's tenant of an OpenSearch cluster.

So, if I were to set the property with different names for each deployment, let's say app-dev, app-staging, app-prod, it would be extremely easy to differentiate the services on jaeger level.

From my experience with 3.2.1 and 3.2.2, such property has no impact at all on the name used on the traces exported to jaeger, everything is exported with the artifact id as the service name. So, if i were to differentiate them, I would need to apply extra filters on each trace, such as the host that exported it or any other.

Is my assumption completely wrong in this case? Should this property allow me to differentiate the services by name? If it is the case, would you be able to guess why I cannot reproduce the expected behavior?

As I mentioned, I've tried setting it via the file, via -D, via Env Var, and by running the app via quarkus:dev and via docker/jib, and nothing "fixed" it.

[michalvavrik]

By reading its description, I've assumed that if I were to set this value to something else different than the app's name, then the traces generated by quarkus would use such value as the service name on the traces, such that I could find this service name on Jaeger.

From my experience with 3.2.1 and 3.2.2, such property has no impact at all on the name used on the traces exported to jaeger, everything is exported with the artifact id as the service name. So, if i were to differentiate them, I would need to apply extra filters on each trace, such as the host that exported it or any other.

quarkus.otel.service.name should have precedence over quarkus.application.name. Please open an issue with reproducer. Comments in closed issue are not ideal platform since they don't provide way to to reproduce it.

The point is, we have test that shows it works under certain circumstances, if there are circumstances when it doesn't work, it needs to be fixed. Please help by providing reproducer or as much code / info as you manage.

Thanks for reporting it.

[michalvavrik]

From my experience with 3.2.1 and 3.2.2, such property has no impact at all on the name used on the traces exported to jaeger, everything is exported with the artifact id as the service name.

So if I get it right then you don't set quarkus.application.name and then it doesn't work? And when you set it to quarkus.application.name=ignored does it work? Reproducer would made the difference.

[jorgebsa]

@michalvavrik so, I was going to create a reproducer, but I guess I don't even need to do it. I've just created a new project on Idea using the built-in quarkus integration.

Told it to create the project with maven, java 17, name otel-service-name-reproducer, and extensions quarkus-opentelemetry and quarkus-resteasy-reactive and it ended up creating the project with 3.2.3 (which is good, as i had already reproduced it on 3.2.1 and 3.2.2 locally)

Once it was setup, I've started the all-in-one Jaeger container and then I've ran the application with the quarkus.application.name undefined and later defined as app-name, while the quarkus.otel.service.name was always set to service-name

Once it was up, I called the example GET endpoint

When I checked jaeger, I could find as services only the app-name and the artifact's id otel-service-name-reproducer, but not the service-name which was configured on the OTel service name property.

Would you like me to push this reproducer to github? Should I open a new Issue as well?

[jorgebsa]

If I try to configure quarkus.otel.resource.attributes=service.name=attribute-name it doesn't have any effect either:

And after executing the requests:

[michalvavrik]

@jorgebsa I created a reproducer and you are right indeed! I tested it with the configuration that we have tested in Quarkus project and the only difference I can see is a span exporter, however I don't have time to look properly today. I'll check over the weekend. Sorry for your trouble.

[jorgebsa]

@michalvavrik haha i was about to comment that I pushed the reproducer to github. Anyway, glad that you are able to reproduce it as well and it is not happening on my project only hehe.

Should a new ticket be opened for this? Looking forward to the fix :) let me take this opportunity here to say how much I appreciate Quarkus as a whole. Keep up the excellent work, your team makes my programmer life much more enjoyable and productive!

[michalvavrik]

I'm sure Quarkus team appreciates it. Here is the fix #35193.

[michalvavrik]

@jorgebsa as workaround, you can set some resource attributes which should do the trick as that was the difference between your case and our test: quarkus.otel.resource.attributes. IIRC 3.2.4.Final is due soon, I added backport label so maybe the fix will get in. Let see.

[jorgebsa]

@michalvavrik you are amazing, not only you implemented a fix this quickly after reported, you also provided a workaround that can be used on the current version :) thank you so much!

[brunobat]

@jorgebsa you mentioned that you have a reproducer... Do you have the link?

[jorgebsa]

@brunobat yes, here it is: https://github.com/jorgebsa/otel-service-name-reproducer

[brunobat]

Thanks @jorgebsa