OpenTelemetry extension re-write using the SDK AutoConfigure #30033

brunobat · 2022-12-22T11:38:41Z

Last part of #26444

OpenTelemetry (OTel) Extension re-write to support OTel Autoconfiguration.

What do we get from it:

Support configuration according to OTel Autoconfiguration
People using OTel standard configs will benefit from Quarkus configuration features.
Access to features already implemented in the autoconfig of the SDK can be easily commanded.
Make it easier to adopt future OTel features.
Allow using the stock OTel exporters in the future.

What was done:

Documentation to comply with the changes bellow and to improve some outdated aspects like the docker collector on Jaeger.
Cleanup and refactoring. In special, move away from the main processor instrumentation related build steps.
Simplify sampler and support its SPI hook
Jakarta rebase with main
Create substitutions for the Otel Auto-configuration code using reflection but not needed for runtime
Remove obsolete config scanning steps
Adapt IT tests, improve their resilience and speedup their execution
Fix additional issues from dev mode reload
Load missing properties, fix resources and sampler
Create Otel SDK on a bean producer
Autoconfigure wiring
New build and runtime configurations based on OTel guidance, ensuring backwards compatibility.

What was left for later:

Optimize Configuration relocations (might not even be needed because we will drop them soon)
TLS configs for the OTLP exporter... Was not available before anyway.
Support additional SPI hooks for propagation, span exporter if requested by the community.
Fixing OTel propagation tests relying on opentelemetry-aws-xray-propagator
Re-activate MP Telemetry Tracing using the new configs. See: Re-activate MP Telemetry Tracing using the new OTel extension configs #31721
We need to consider fixing JDBC tracing later as this depends on using a later created datasource. See: Reactivate opentelemetry-jdbc-instrumentation at the Misc4 build step #31720
Try the @ConfigMapping approach

quarkus-bot · 2022-12-22T11:39:48Z

/cc @radcortez(opentelemetry)

github-actions · 2023-02-02T11:56:12Z

🙈 The PR is closed and the preview is expired.

brunobat · 2023-02-10T17:57:08Z

@luneo7 Please be aware of this PR. A review from you would be much appreciated.

brunobat · 2023-03-08T19:35:02Z

@radcortez we can try the @ConfigMapping approach once we have the new version of the extension.

luneo7 · 2023-03-08T20:42:39Z

...try/deployment/src/main/java/io/quarkus/opentelemetry/deployment/OpenTelemetryProcessor.java

@@ -69,6 +70,14 @@ AdditionalBeanBuildItem ensureProducerIsRetained() {
                .build();
    }

+    @BuildStep
+    void registerNativeImageResources(BuildProducer<ServiceProviderBuildItem> services) throws IOException {
+        services.produce(ServiceProviderBuildItem.allProvidersFromClassPath(


Shouldn't we also add:
io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider
io.opentelemetry.sdk.autoconfigure.spi.ResourceProvider
io.opentelemetry.sdk.autoconfigure.spi.ConfigurablePropagatorProvider
?

I was planning to add official support for them only later. I can add the SPIs, however I'll leave testing and documentation for later because this PR is already too big.

Agree. At this point, we should move forward and add these missing pieces later. They are not a blocker at this point.

gsmet

I added a few questions and very minor suggestions.

What's the status of this work exactly? Is it expected to land before our 3.0.0.CR1?

gsmet · 2023-03-09T12:22:35Z

docs/src/main/asciidoc/opentelemetry.adoc

+
+[NOTE]
+====
+All configurations have been updated from `quarkus.opentelemetry.\*` -> `quarkus.otel.*`


Sorry to ask that but is it really an improvement? What's the rationale of this change?

This is to align with the OTel side property semantics. See: https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#disabling-opentelemetrysdk

We will transparently support them from now on.

When OTel introduced its configuration namespace, we discussed and decided to align with it as close as possible:

Since the goal of OTel is to be supported across many cloud providers, tools, servers, etc, we want to stay within it. Ideally, users should use the same configuration as what they use in other external OTel aware components

We cannot prevent users from plugin in the OTel Agent, which uses the OTel configuration as specified. This should unify that configuration, so when you plug in the agent, it will use the defined Quarkus configuration (if any)

OK. Thanks for the clarification.
I wonder if we should rewrite the properties as part of the upgrade process. Unfortunately, we cannot use regexps so we will have to map the old names to the new names manually.

Maybe it's not worth the hassle, I don't know.

docs/src/main/asciidoc/opentelemetry.adoc

extensions/opentelemetry/deployment/pom.xml

gsmet · 2023-03-09T12:28:05Z

...try/runtime/src/main/java/io/quarkus/opentelemetry/runtime/config/build/OtelBuildConfig.java

+    /**
+     * No Metrics exporter for now
+     */
+    @ConfigItem(name = "metrics.exporter", defaultValue = "none")


So you will have a singleton list with none in it, right?

Yes. This will effectively disable OTel metrics.

brunobat · 2023-03-09T13:03:40Z

Depends on the feedback @gsmet but yes, I'd like to add it to CR1 and issue follow up PRs later, according to the description in the 1st comment.
This contains the main changes and is already too big to keep adding functionality to it, in my opinion.

quarkus-bot · 2023-03-15T17:05:26Z

Failing Jobs - Building `db3bb13`

Status	Name	Step	Failures	Logs	Raw logs
✔️	Gradle Tests - JDK 11
⌛	Gradle Tests - JDK 11 Windows	`Build`	Failures	Logs	Raw logs
✔️	JVM Tests - JDK 11
✔️	JVM Tests - JDK 17
✖	JVM Tests - JDK 17 Windows	`Build`	Failures	Logs	Raw logs
⌛	JVM Tests - JDK 19	`Build`	⚠️ Check →	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ Gradle Tests - JDK 11 Windows #

📦 integration-tests/gradle

✖ io.quarkus.gradle.devmode.MultiSourceProjectDevModeTest.main line 22 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference, java.util.concurrent.atomic.AtomicReferencejava.lang.String, java.lang.Stringboolean was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

⚙️ JVM Tests - JDK 17 Windows #

- Failing: extensions/resteasy-classic/rest-client/runtime 
! Skipped: extensions/amazon-lambda-http/deployment extensions/amazon-lambda-http/http-event-server extensions/amazon-lambda-rest/deployment and 105 more

📦 extensions/resteasy-classic/rest-client/runtime

✖ io.quarkus.restclient.runtime.RestClientBaseTest. - More details - Source on GitHub

java.io.IOException: Failed to delete temp directory D:\a\quarkus\quarkus\extensions\resteasy-classic\rest-client\runtime\target\junit4822274520985205514. The following paths could not be deleted (see suppressed exceptions for details): 
	at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.createIOExceptionWithAttachedFailures(TempDirectory.java:350)
	at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:251)

gsmet · 2023-03-15T18:03:41Z

Let's merge. Thanks for your perseverance on this one!

brunobat · 2023-03-16T09:42:19Z

Thanks @gsmet. Super happy to see this merged.

hnourhani · 2023-03-28T12:54:26Z

...pentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/graal/Substitutions.java

+                ClassLoader serviceClassLoader,
+                BiFunction<? super MetricExporter, ConfigProperties, ? extends MetricExporter> metricExporterCustomizer,
+                List<Closeable> closeables) {
+            // OTel metrics not supported and there is no need to call


@brunobat does it mean that we cannot use these configuration to enable Prometheus exporter:
https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#prometheus-exporter

If not could start a PrometheusMetricReader manually using OpenTelemetry SDK and collect metrics?

Correct. No OTel Metrics on Quarkus for now. It requires a lot of rework in the instrumentation.

quarkus-bot bot added the area/tracing label Dec 22, 2022

brunobat mentioned this pull request Dec 22, 2022

OpenTelemetry SDK Autoconfigure #26444

Closed

5 tasks

brunobat force-pushed the otel-autoconfigure-4 branch from 42dd7ff to 06068d5 Compare January 4, 2023 16:53

brunobat force-pushed the otel-autoconfigure-4 branch from 06068d5 to 69329c7 Compare February 2, 2023 11:33

quarkus-bot bot added the area/documentation label Feb 2, 2023

gsmet added the triage/needs-rebase This PR needs to be rebased first because it has merge conflicts label Feb 4, 2023

brunobat force-pushed the otel-autoconfigure-4 branch 4 times, most recently from 51dea72 to 562e6e9 Compare February 9, 2023 19:29

brunobat marked this pull request as ready for review February 9, 2023 19:37

brunobat removed the triage/needs-rebase This PR needs to be rebased first because it has merge conflicts label Feb 9, 2023

This comment has been minimized.

Sign in to view

brunobat closed this Feb 10, 2023

quarkus-bot bot added the triage/invalid This doesn't seem right label Feb 10, 2023

brunobat reopened this Feb 10, 2023

quarkus-bot bot removed the triage/invalid This doesn't seem right label Feb 10, 2023

brunobat force-pushed the otel-autoconfigure-4 branch from 562e6e9 to 3aeb9f2 Compare February 10, 2023 17:47

brunobat self-assigned this Feb 10, 2023

brunobat requested review from radcortez, alesj, ebullient and famod February 10, 2023 17:54

brunobat force-pushed the otel-autoconfigure-4 branch from 6294b71 to 41fab87 Compare March 8, 2023 19:38

luneo7 reviewed Mar 8, 2023

View reviewed changes

This comment has been minimized.

Sign in to view

brunobat mentioned this pull request Mar 9, 2023

Reactivate opentelemetry-jdbc-instrumentation at the Misc4 build step #31720

Closed

brunobat force-pushed the otel-autoconfigure-4 branch from 41fab87 to b47e0d6 Compare March 9, 2023 09:49

quarkus-bot bot added the area/infra-automation anything related to CI, bots, etc. that are used to automated our infrastructure label Mar 9, 2023

brunobat mentioned this pull request Mar 9, 2023

Re-activate MP Telemetry Tracing using the new OTel extension configs #31721

Closed

gsmet reviewed Mar 9, 2023

View reviewed changes

This comment has been minimized.

Sign in to view

brunobat force-pushed the otel-autoconfigure-4 branch from b47e0d6 to 24518ec Compare March 9, 2023 16:30

This comment has been minimized.

Sign in to view

radcortez approved these changes Mar 10, 2023

View reviewed changes

luneo7 approved these changes Mar 10, 2023

View reviewed changes

brunobat requested a review from gsmet March 14, 2023 09:48

OpenTelemetry extension rewrite supporting Autoconfiguration

db3bb13

brunobat force-pushed the otel-autoconfigure-4 branch from 24518ec to db3bb13 Compare March 15, 2023 11:35

gsmet merged commit 56f2dab into quarkusio:main Mar 15, 2023

quarkus-bot bot added this to the 3.0 - main milestone Mar 15, 2023

hnourhani reviewed Mar 28, 2023

View reviewed changes

radcortez mentioned this pull request Mar 30, 2023

Use of the quarkus-opentelemetry extension yields config warnings #32210

Closed

This was referenced May 11, 2023

OpenTelemetry SDK autoconfiguration ignores OTEL service name in favor of Quarkus app name #33317

Closed

Quarkus 3 OpenTelemetry hardening quarkus-qe/quarkus-test-suite#1222

Merged

brunobat deleted the otel-autoconfigure-4 branch August 21, 2024 10:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenTelemetry extension re-write using the SDK AutoConfigure #30033

OpenTelemetry extension re-write using the SDK AutoConfigure #30033

brunobat commented Dec 22, 2022 •

edited

Loading

quarkus-bot bot commented Dec 22, 2022

github-actions bot commented Feb 2, 2023 •

edited

Loading

This comment has been minimized.

brunobat commented Feb 10, 2023

brunobat commented Mar 8, 2023

luneo7 Mar 8, 2023

brunobat Mar 9, 2023

radcortez Mar 10, 2023

This comment has been minimized.

gsmet left a comment

gsmet Mar 9, 2023

brunobat Mar 9, 2023

radcortez Mar 10, 2023

gsmet Mar 15, 2023

gsmet Mar 9, 2023

brunobat Mar 9, 2023

brunobat commented Mar 9, 2023

This comment has been minimized.

This comment has been minimized.

quarkus-bot bot commented Mar 15, 2023

gsmet commented Mar 15, 2023

brunobat commented Mar 16, 2023

hnourhani Mar 28, 2023 •

edited

Loading

brunobat Mar 28, 2023

OpenTelemetry extension re-write using the SDK AutoConfigure #30033

OpenTelemetry extension re-write using the SDK AutoConfigure #30033

Conversation

brunobat commented Dec 22, 2022 • edited Loading

quarkus-bot bot commented Dec 22, 2022

github-actions bot commented Feb 2, 2023 • edited Loading

This comment has been minimized.

brunobat commented Feb 10, 2023

brunobat commented Mar 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

gsmet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunobat commented Mar 9, 2023

This comment has been minimized.

This comment has been minimized.

quarkus-bot bot commented Mar 15, 2023

Failing Jobs - Building db3bb13

Failures

⚙️ Gradle Tests - JDK 11 Windows #

📦 integration-tests/gradle

⚙️ JVM Tests - JDK 17 Windows #

📦 extensions/resteasy-classic/rest-client/runtime

gsmet commented Mar 15, 2023

brunobat commented Mar 16, 2023

hnourhani Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunobat commented Dec 22, 2022 •

edited

Loading

github-actions bot commented Feb 2, 2023 •

edited

Loading

Failing Jobs - Building `db3bb13`

hnourhani Mar 28, 2023 •

edited

Loading