Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClassNotFoundException is thrown when loading protocol resolvers from ForkJoinPool task #42468

Closed
raestio opened this issue Sep 27, 2024 · 8 comments
Assignees
Labels
type: regression A regression from a previous release
Milestone

Comments

@raestio
Copy link

raestio commented Sep 27, 2024

Hi, we have upgraded Spring Boot app version from 3.3.3 to 3.3.4 and now the SSL bundles cannot be loaded. See the full stack trace of the exception below.

I've found there have been some changes related to SSL bundles in #42119. I can see the key stores are being lazily loaded but not sure how (and if) it can change the class loader that doesn't see Base64ProtocolResolver.

We have integration @SpringBootTest tests that use the SSL bundles and they are successfully passing. So most probably the app must be compiled and packaged as we see those exceptions only when we run the app (built as a JAR) in Docker container. But I could still be missing something.

When I downgrade to 3.3.3, it works OK as expected.

I don't have a sample application that reproduces the issue yet.

The app runs on reactive stack.

java.lang.IllegalStateException: Unable to create key store: Could not load store from 'classpath:{CERT}.p12'
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.createKeyStore(JksSslStoreBundle.java:96)
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.lambda$new$0(JksSslStoreBundle.java:59)
	at org.springframework.util.function.SingletonSupplier.get(SingletonSupplier.java:106)
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.getKeyStore(JksSslStoreBundle.java:65)
	at org.springframework.boot.ssl.DefaultSslManagerBundle.getKeyManagerFactory(DefaultSslManagerBundle.java:45)
	at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory$SslConfigurer.customizeSsl(ReactorClientHttpConnectorFactory.java:91)
	at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83)
	at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60)
	at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88)
	at reactor.netty.http.client.HttpClient.secure(HttpClient.java:1430)
	at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory$SslConfigurer.configure(ReactorClientHttpConnectorFactory.java:84)
	at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorNettyHttpClientMapper.lambda$of$1(ReactorNettyHttpClientMapper.java:65)
	at java.base/java.util.function.Function.lambda$andThen$1(Unknown Source)
	at java.base/java.util.function.Function.lambda$andThen$1(Unknown Source)
	at org.springframework.http.client.reactive.ReactorClientHttpConnector.createHttpClient(ReactorClientHttpConnector.java:125)
	at org.springframework.http.client.reactive.ReactorClientHttpConnector.connect(ReactorClientHttpConnector.java:142)
	at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.exchange(ExchangeFunctions.java:102)
	at org.springframework.web.reactive.function.client.DefaultWebClient$ObservationFilterFunction.filter(DefaultWebClient.java:737)
	at org.springframework.web.reactive.function.client.ExchangeFilterFunction.lambda$apply$2(ExchangeFilterFunction.java:73)
	at org.springframework.web.reactive.function.client.DefaultWebClient$DefaultRequestBodyUriSpec.lambda$exchange$11(DefaultWebClient.java:457)
	at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:47)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:210)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:144)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:144)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onSubscribe(MonoFlatMap.java:291)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:178)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:178)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.request(FluxMapFuseable.java:360)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139)
	at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.request(FluxOnErrorReturn.java:153)
	at reactor.core.publisher.FluxFlatMap$FlatMapInner.onSubscribe(FluxFlatMap.java:968)
	at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onSubscribe(FluxOnErrorReturn.java:95)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:152)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onSubscribe(FluxMapFuseable.java:265)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:202)
	at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:430)
	at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
	at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:348)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259)
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865)
	at reactor.core.publisher.FluxConcatMap$WeakScalarSubscription.request(FluxConcatMap.java:480)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:202)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:348)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
	at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
	at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:348)
	at reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
	at oracle.r2dbc.impl.OracleReactiveJdbcAdapter$1.onNext(OracleReactiveJdbcAdapter.java:783)
	at oracle.r2dbc.impl.AsyncLock$UsingConnectionSubscriber.onNext(AsyncLock.java:481)
	at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	at reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
	at org.reactivestreams.FlowAdapters$FlowToReactiveSubscriber.onNext(FlowAdapters.java:211)
	at oracle.jdbc.driver.PhasedPublisher$PhasedSubscription.lambda$emitNextItem$0(PhasedPublisher.java:403)
	at oracle.jdbc.driver.PhasedPublisher.handleOnNext(PhasedPublisher.java:267)
	at oracle.jdbc.driver.PhasedPublisher$PhasedSubscription.lambda$emitNextItem$1(PhasedPublisher.java:401)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Could not load store from 'classpath:{CERT}.p12'
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.loadKeyStore(JksSslStoreBundle.java:125)
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.createKeyStore(JksSslStoreBundle.java:91)
	... 88 common frames omitted
Caused by: java.lang.IllegalArgumentException: Unable to instantiate factory class [org.springframework.boot.io.Base64ProtocolResolver] for factory type [org.springframework.core.io.ProtocolResolver]
	at org.springframework.core.io.support.SpringFactoriesLoader$FailureHandler.lambda$throwing$0(SpringFactoriesLoader.java:647)
	at org.springframework.core.io.support.SpringFactoriesLoader$FailureHandler.lambda$handleMessage$3(SpringFactoriesLoader.java:671)
	at org.springframework.core.io.support.SpringFactoriesLoader.instantiateFactory(SpringFactoriesLoader.java:231)
	at org.springframework.core.io.support.SpringFactoriesLoader.load(SpringFactoriesLoader.java:206)
	at org.springframework.core.io.support.SpringFactoriesLoader.load(SpringFactoriesLoader.java:142)
	at org.springframework.boot.io.ApplicationResourceLoader.<init>(ApplicationResourceLoader.java:53)
	at org.springframework.boot.io.ApplicationResourceLoader.<init>(ApplicationResourceLoader.java:41)
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.loadKeyStore(JksSslStoreBundle.java:119)
	... 89 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.springframework.boot.io.Base64ProtocolResolver
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Unknown Source)
	at java.base/java.lang.Class.forName(Unknown Source)
	at org.springframework.util.ClassUtils.forName(ClassUtils.java:304)
	at org.springframework.core.io.support.SpringFactoriesLoader.instantiateFactory(SpringFactoriesLoader.java:224)
	... 94 common frames omitted
@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Sep 27, 2024
@wilkinsona
Copy link
Member

Thanks for the report. The ApplicationResourceLoader is being created in such a way that the thread context class loader will be used to load its protocol resolvers. It looks like this is ClassLoaders$AppClassLoader which, if you're running your application with java -jar or java -cp … org.springframework.boot.loader.launch.JarLauncher won't have org.springframework.boot.io.Base64ProtocolResolver on its classpath. It sounds like you're using java -jar in your Docker container. Can you confirm that's the case?

In the absence of a sample that reproduces the problem, you could test the above theory by patching line 119 of JksSslStoreBundle:

Resource resource = new ApplicationResourceLoader(getClass().getClassLoader()).getResource(location);

@wilkinsona wilkinsona added the status: waiting-for-feedback We need additional information before we can continue label Sep 27, 2024
@raestio
Copy link
Author

raestio commented Sep 28, 2024

Thank you for the hint. Our Dockerfile looked like this:

docs Spring Boot 3.2.0 - Dockerfiles

...
COPY --from=builder application/dependencies/ ./
COPY --from=builder application/spring-boot-loader/ ./
COPY --from=builder application/snapshot-dependencies/ ./
COPY --from=builder application/application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.launch.JarLauncher"]

So java org.springframework.boot.loader.launch.JarLauncher.

But I have updated the Dockerfile based on the latest docs Spring Boot 3.3.4 - Dockerfiles / Source and the issue is gone, works OK. Thank you for the help!

Now I'm thinking, I can see the docs for the tools mode were updated in version 3.3.0 - #40094 - which I missed. I believe this feature https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.3-Release-Notes#cds-support was the related one. Does it mean the changes made for the CDS feature caused some incompatibility with the launching the extracted JAR in an older way before 3.3.0 as I had it before? Because now I'm still running the extracted JAR but with java -jar application.jar (as stated in the latest docs) and it works.
I'm just trying to piece together the points from your comment and find out why it works. I guess a different class loader is used.

@spring-projects-issues spring-projects-issues added status: feedback-provided Feedback has been provided and removed status: waiting-for-feedback We need additional information before we can continue labels Sep 28, 2024
@wilkinsona
Copy link
Member

wilkinsona commented Sep 30, 2024

I guess a different class loader is used.

Yes, that's right. Since 3.3, the extracted jar uses a Class-Path manifest header to reference all of the application's dependencies so everything's loaded by the app class loader. As you suspected, this was primarily for CDS where taking Spring Boot's custom class loader out of the picture significantly improves the hit rate against the CDS archive and increases the startup time benefits.

@wilkinsona

This comment was marked as outdated.

@wilkinsona wilkinsona added the for: team-meeting An issue we'd like to discuss as a team to make progress label Sep 30, 2024
@philwebb philwebb self-assigned this Oct 21, 2024
@philwebb
Copy link
Member

Looking into this and I wonder if we should have a ApplicationResourceLoader constructor that allows us to specify a different classloader for the SpringFactoriesLoader? I think that might help bring StringToFileConverter at least closer to the older code.

For this failure specifically, I feel like we somehow need to set the classloader that gets used on the actual SslBundle.

@philwebb philwebb added type: bug A general bug type: regression A regression from a previous release and removed status: waiting-for-triage An issue we've not yet triaged status: feedback-provided Feedback has been provided for: team-meeting An issue we'd like to discuss as a team to make progress type: bug A general bug labels Oct 22, 2024
@philwebb philwebb added this to the 3.3.x milestone Oct 22, 2024
@philwebb
Copy link
Member

philwebb commented Oct 22, 2024

https://github.com/philwebb/gh-42468 reproduces the issue (in a slightly different form)

@philwebb philwebb changed the title SSL bundles: Unable to create key store - ClassNotFoundException ClassNotFoundException is thrown when loading protocol resolvers from ForkJoinPool task Oct 23, 2024
@philwebb
Copy link
Member

I've created #42838 to consider the broader classloader issue so we can target a focused fix for this specific problem in 3.3.x.

@philwebb
Copy link
Member

Reopening because I missed the BundleContentProperty class

ndwlocatieservices added a commit to ndwnu/nls-geometry that referenced this issue Oct 31, 2024
…ot-starter-parent to v3.3.5

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [org.springframework.boot:spring-boot-starter-parent](https://spring.io/projects/spring-boot) ([source](https://github.com/spring-projects/spring-boot)) | parent | patch | `3.3.4` -> `3.3.5` |

---

### Release Notes

<details>
<summary>spring-projects/spring-boot (org.springframework.boot:spring-boot-starter-parent)</summary>

### [`v3.3.5`](https://github.com/spring-projects/spring-boot/releases/tag/v3.3.5)

[Compare Source](spring-projects/spring-boot@v3.3.4...v3.3.5)

#### 🐞 Bug Fixes

-   Running mvn spring-boot:run with classpaths that exceeds Windows' length limits leaves temporary files [#&#8203;42841](spring-projects/spring-boot#42841)
-   Report produced by ConditionReportApplicationContextFailureProcessor is always empty in a failed test [#&#8203;42785](spring-projects/spring-boot#42785)
-   Case-insensitive comparisons may be adversely affected by the user's locale [#&#8203;42735](spring-projects/spring-boot#42735)
-   DataSourceProperties#driverClassIsLoadable should not print a stacktrace to the error stream when it fails [#&#8203;42683](spring-projects/spring-boot#42683)
-   Some `@ControllerEndpoint` and `@RestControllerEndpoint` infrastructure remains undeprecated [#&#8203;42498](spring-projects/spring-boot#42498)
-   Auto-configuration for Rabbit Streams doesn't consider RabbitConnectionDetails [#&#8203;42490](spring-projects/spring-boot#42490)
-   ClassNotFoundException is thrown when loading protocol resolvers from ForkJoinPool task [#&#8203;42468](spring-projects/spring-boot#42468)
-   ActiveMQ Artemis Connection Factory creation fails in native image [#&#8203;42421](spring-projects/spring-boot#42421)
-   Duplicate meter binding when context contains multiple registries, none are primary, and one or more is a composite [#&#8203;42397](spring-projects/spring-boot#42397)

#### 📔 Documentation

-   Document that embedded Tomcat must be at least 10.1.25 [#&#8203;42849](spring-projects/spring-boot#42849)
-   Fix systemd example configuration [#&#8203;42805](spring-projects/spring-boot#42805)
-   Document that the exact behavior of the maximum HTTP request header size property is server-specific [#&#8203;42789](spring-projects/spring-boot#42789)
-   Clarify why `@Primary` is recommended when defining your own ObjectMapper that replaces JacksonAutoConfiguration's [#&#8203;42787](spring-projects/spring-boot#42787)
-   Polish javadoc for Binder#bindOrCreate(String, Class) [#&#8203;42778](https://github.com/spring-projects/spring-boot/issu...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: regression A regression from a previous release
Projects
None yet
Development

No branches or pull requests

4 participants