Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pubsub: ClassNotFoundException org/eclipse/jetty/alpn/ALPN with versions 0.32, 0.35, ..., 0.38 #3044

Closed
j256 opened this issue Mar 14, 2018 · 13 comments
Assignees
Labels
dependencies priority: p2 Moderately-important priority. Fix may not be included in next release. type: question Request for information or clarification. Not an issue.

Comments

@j256
Copy link

j256 commented Mar 14, 2018

Although I've seen similar issues, they all seem to be closed, I'm still seeing the following error with recent versions of this API. Related to: #2740 #3036

Caused by: java.lang.ClassNotFoundException: org/eclipse/jetty/alpn/ALPN

My test program is doing the following:

ProjectSubscriptionName subName = ProjectSubscriptionName.of("project", "subscriber");
Builder builder = Subscriber.newBuilder(subName, new OurReceiver());
// also tried adding the following
// GoogleCredentials cred=GoogleCredentials.fromStream(new FileInputStream("creds.json"));
// builder.setCredentialsProvider(FixedCredentialsProvider.create(cred));
Subscriber subscriber = builder.build();
subscriber.startAsync().awaitRunning();

Here's my maven pom.xml dependency. Dependency tree attached:

<dependency>
	<groupId>com.google.cloud</groupId>
	<artifactId>google-cloud-pubsub</artifactId>
	<version>0.38.0-beta</version> <!-- also tried 32, 35, 36, 37 -->
</dependency>

Any idea what I'm doing wrong? Can I even run this outside of the Google network? Please excuse the obvious ignorance of this stuff.

dependencies.txt

@prayagupa
Copy link

prayagupa commented Mar 14, 2018

1) for me I could solve similar issue using jetty-alpn-agent jar - see my reproducable codebase here - devasthali-machine/nlu-pipeline#1 (I am using against google-dialogflow though)

wget http://central.maven.org/maven2/org/mortbay/jetty/alpn/jetty-alpn-agent/2.0.7/jetty-alpn-agent-2.0.7.jar

java -jar -javaagent:jetty-alpn-agent-2.0.7.jar nlu-intent-toolkit.jar agent1 "where is porcupine tree playing?" credentials.json 

But that is not a prod ready code so i m not sure if that's the best way i will fix it. This might be helpful - https://github.com/grpc/grpc-java/blob/master/SECURITY.md#tls-with-jdk-jetty-alpnnpn

2) ALSO, their google-cloud-compat-checker might be useful. Its bit painful to use it because they use parent versions. I don't understand why dont make it a jar that can be used anywhere.

the compat-checker tells if you have ALPN supported in your environment, example,

$ mvn exec:java
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building Google Cloud Java Compatibility Checker 0.39.0-alpha
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker ---
OS details:
  os.detected.name: osx
  os.detected.arch: x86_64
  os.detected.classifier: osx-x86_64
  os.detected.release: null
  os.detected.release.version: null
JVM details:
  Java version: 1.8.0_151
  Java specification version: 1.8
  JVM bit mode: 64
OpenSSL details:
  open ssl is available: false
  ALPN is supported: false
Checking compatibility...
  [PASS] This OS + architecture is supported.
  [PASS] 64-bit JVM is supported.
  [FAIL] Open SSL is NOT available
         Open SSL Unavailability cause:
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-osx-x86_64, netty-tcnative]
	at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178)
	at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403)
	at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
	at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58)
	at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
	at java.lang.Thread.run(Thread.java:748)
  [FAIL] Open SSL ALPN is NOT supported
Result: FAIL
  Your environment is not supported by Forked Tomcat Native.
  See http://netty.io/wiki/forked-tomcat-native.html for details.
  This means that you won't be able to use grpc-based APIs, but
  http1-based APIs should still work.

@j256
Copy link
Author

j256 commented Mar 14, 2018

That didn't seem to work for me. Neither jetty-alpn-agent nor jetty-alpn-client. The problem is in io.grpc.netty.JettyTlsUtil which is doing a Class.forName("org.eclipse.jetty.alpn.ALPN", true, null); in the isJettyAlpnConfigured() method.

@pongad
Copy link
Contributor

pongad commented Mar 14, 2018

@j256 I'm glad you figured this out, but this is rather surprising. We should get to the bottom of this.

@pongad pongad added type: question Request for information or clarification. Not an issue. priority: p2 Moderately-important priority. Fix may not be included in next release. dependencies labels Mar 14, 2018
@garrettjonesgoogle
Copy link
Member

ALPN is a red herring of sorts. The ssl layer that's supposed to be used is netty-tcnative-boringssl-static, which should be built into grpc-netty-shaded. If you are facing issues with ssl not being available, it's more likely a platform issue. Could you run google-cloud-compat-checker like @prayagupd suggested?

@prayagupa
Copy link

Hi @j256 I took your Test Subscriber code and tested in my MacOS with java 7/java 8. Its working fine - Here's my testcode for both emitting and consuming events from google pubsub -
https://github.com/duwamish-os/eventstream-skipper-ii.java (run mvn exec:java. It will show you google-cloud logs as well). Please let me know if you have different settings than how I tested.

Dependency jars same as yours;

mvn dependency:tree
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building eventstream-skipper-java 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ eventstream-skipper-java ---
[INFO] com.eventstream.googlepubsub:eventstream-skipper-java:jar:1.0-SNAPSHOT
[INFO] +- com.google.cloud:google-cloud-pubsub:jar:0.39.0-beta:compile
[INFO] |  +- com.google.cloud:google-cloud-core:jar:1.21.0:compile
[INFO] |  |  +- com.google.guava:guava:jar:20.0:compile
[INFO] |  |  +- joda-time:joda-time:jar:2.9.2:compile
[INFO] |  |  +- com.google.http-client:google-http-client:jar:1.23.0:compile
[INFO] |  |  |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO] |  |  |     +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
[INFO] |  |  |     +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] |  |  |     \- commons-codec:commons-codec:jar:1.3:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
[INFO] |  |  +- com.google.api:api-common:jar:1.4.0:compile
[INFO] |  |  +- com.google.api:gax:jar:1.19.0:compile
[INFO] |  |  |  +- org.threeten:threetenbp:jar:1.3.3:compile
[INFO] |  |  |  \- com.google.auth:google-auth-library-oauth2-http:jar:0.9.0:compile
[INFO] |  |  |     \- com.google.http-client:google-http-client-jackson2:jar:1.19.0:compile
[INFO] |  |  |        \- com.fasterxml.jackson.core:jackson-core:jar:2.1.3:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java-util:jar:3.5.1:compile
[INFO] |  |  |  \- com.google.code.gson:gson:jar:2.7:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-common-protos:jar:1.3.0:compile
[INFO] |  |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.4.0:compile
[INFO] |  +- com.google.cloud:google-cloud-core-grpc:jar:1.21.0:compile
[INFO] |  |  +- com.google.auth:google-auth-library-credentials:jar:0.9.0:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.5.1:compile
[INFO] |  |  +- io.grpc:grpc-protobuf:jar:1.9.0:compile
[INFO] |  |  |  \- io.grpc:grpc-protobuf-lite:jar:1.9.0:compile
[INFO] |  |  +- io.grpc:grpc-context:jar:1.9.0:compile
[INFO] |  |  \- com.google.api:gax-grpc:jar:1.19.0:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.4.0:compile
[INFO] |  +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.4.0:compile
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.9.0:compile
[INFO] |  |  \- io.grpc:grpc-core:jar:1.9.0:compile (version selected from constraint [1.9.0,1.9.0])
[INFO] |  |     +- com.google.errorprone:error_prone_annotations:jar:2.1.2:compile
[INFO] |  |     +- com.google.instrumentation:instrumentation-api:jar:0.4.3:compile
[INFO] |  |     +- io.opencensus:opencensus-api:jar:0.10.0:compile
[INFO] |  |     \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.10.0:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.9.0:compile
[INFO] |  +- io.grpc:grpc-auth:jar:1.9.0:compile
[INFO] |  \- com.google.auto.value:auto-value:jar:1.4:compile
[INFO] \- com.google.cloud:google-cloud-logging-logback:jar:0.39.0-alpha:compile
[INFO]    +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO]    |  +- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO]    |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO]    \- com.google.cloud:google-cloud-logging:jar:1.21.0:compile
[INFO]       \- com.google.api.grpc:proto-google-cloud-logging-v2:jar:0.4.0:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.335 s
[INFO] Finished at: 2018-03-20T23:13:01-07:00
[INFO] Final Memory: 15M/309M
[INFO] ------------------------------------------------------------------------

platform config is as below

$ mvn exec:java
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building Google Cloud Java Compatibility Checker 0.39.0-alpha
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker ---
OS details:
  os.detected.name: osx
  os.detected.arch: x86_64
  os.detected.classifier: osx-x86_64
  os.detected.release: null
  os.detected.release.version: null
JVM details:
  Java version: 1.7.0_80
  Java specification version: 1.7
  JVM bit mode: 64
OpenSSL details:
  open ssl is available: false
  ALPN is supported: false
Checking compatibility...
  [PASS] This OS + architecture is supported.
  [PASS] 64-bit JVM is supported.
  [FAIL] Open SSL is NOT available
         Open SSL Unavailability cause:
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-osx-x86_64, netty-tcnative]
	at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178)
	at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403)
	at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
	at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58)
	at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
	at java.lang.Thread.run(Thread.java:745)
  [FAIL] Open SSL ALPN is NOT supported
  [WARN] gRPC doesn't work on Google App Engine Standard under Java 1.7
Result: FAIL
  Your environment is not supported by Forked Tomcat Native.
  See http://netty.io/wiki/forked-tomcat-native.html for details.
  This means that you won't be able to use grpc-based APIs, but
  http1-based APIs should still work.

@j256
Copy link
Author

j256 commented Mar 26, 2018

Ok. Swapping this back in. Thanks for the help @prayagupd . I'm getting the following from the compatibility checker which seems to match your's.

google-cloud-compat-checker {localhost}  mvn exec:java
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO] 
[INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder with a thread count of 1
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Google Cloud Java Compatibility Checker 0.40.1-alpha-SNAPSHOT
[INFO] ------------------------------------------------------------------------

```[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker ---
OS details:
  os.detected.name: linux
  os.detected.arch: x86_64
  os.detected.classifier: linux-x86_64
  os.detected.release: "rhel"
  os.detected.release.version: 7.3
JVM details:
  Java version: 1.7.0_79
  Java specification version: 1.7
  JVM bit mode: 64
OpenSSL details:
  open ssl is available: false
  ALPN is supported: false
Checking compatibility...
  [PASS] This OS + architecture is supported.
  [PASS] 64-bit JVM is supported.
  [FAIL] Open SSL is NOT available
         Open SSL Unavailability cause:
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora, netty-tcnative]
        at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178)
        at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403)
        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
        at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58)
        at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
        at java.lang.Thread.run(Thread.java:745)
  [FAIL] Open SSL ALPN is NOT supported
  [WARN] gRPC doesn't work on Google App Engine Standard under Java 1.7
Result: FAIL
  Your environment is not supported by Forked Tomcat Native.
  See http://netty.io/wiki/forked-tomcat-native.html for details.
  This means that you won't be able to use grpc-based APIs, but
  http1-based APIs should still work.

However, when I go to run your program I get similar errors to mine and I'm seeing the following in my logs.

17:18:03.502 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information.
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_32, netty_tcnative_linux_x86_32_fedora, netty_tcnative_x86_32, netty_tcnative]

Ultimately I get the followign with the -javaagent:jetty-alpn-agent-2.0.7.jar argument on the commandline.

Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

I'm running on a:

> uname -a
Linux gwatson-l 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 13 10:46:25 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

> cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Looks like none of the shared libraries load for me. I see a bunch of log messages like:

17:18:03.492 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_linux_x86_32', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_x86_32 in java.library.path
17:18:03.493 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_linux_x86_32 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_x86_32 in java.library.path
17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_32', trying next name...
17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_linux_x86_32_fedora', trying other loading mechanism.
17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_linux_x86_32_fedora cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
17:18:03.495 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_32_fedora', trying next name...
17:18:03.495 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_x86_32', trying other loading mechanism.
17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_x86_32 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_32', trying next name...
17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative', trying other loading mechanism.
17:18:03.499 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
17:18:03.499 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative', trying next name...
17:18:03.502 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information.
17:18:03.503 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to initialize netty-tcnative; OpenSslEngine will be unavailable. See http://netty.io/wiki/forked-tomcat-native.html for more information.

That's interesting because I'm on a non-fedora 64-bit machine but the messages seem to be trying to load a 32-bit version. Let me see if I can force that selection. I've tried the static linked stuff and forcing the native classifier <classifier>linux-x86_64</classifier>.

@j256
Copy link
Author

j256 commented Apr 13, 2018

Wow, this just came up again on another developer's box. Related to grpc/grpc-java#3025

@garrettjonesgoogle
Copy link
Member

@j256 , the real error that is relevant is "gRPC doesn't work on Google App Engine Standard under Java 1.7". The easiest way to fix this is to upgrade to Java 1.8 on App Engine, which should work fine.

@garrettjonesgoogle
Copy link
Member

This applies to @prayagupd 's case too.

@j256
Copy link
Author

j256 commented Apr 23, 2018

Moving to Java 8 just isn't possible given my corporate software requirements.

@garrettjonesgoogle
Copy link
Member

That is very unfortunate - the Java 7 runtime of App Engine will be entirely shut down in January next year (2019): https://cloud.google.com/appengine/docs/deprecations/java7

@j256
Copy link
Author

j256 commented Apr 25, 2018

Ok. I think I've finally figured this out. We run with a lot of different JREs on our servers supporting various Java variants and I believe this is a 32bit/64bit issue and not a Java7 versus 8 issue. I'm going to mark this closed but I'll add some details here as I get them. Thanks for all of the help.

@j256 j256 closed this as completed Apr 25, 2018
@j256
Copy link
Author

j256 commented Jun 8, 2018

For the record, I continue to have problems with this on older build boxes running older variants of LibC, mismatched 32-bit and 64-bit JREs, and mismatched JRE versions. It's tricky. Older servers are especially susceptible to this.

@j256
Copy link
Author

j256 commented Jul 16, 2018

Also, older servers with glibc < version 2.7 will not work because of netty requirements. Yes, sigh, in 2018 my company has to support our code on these older servers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies priority: p2 Moderately-important priority. Fix may not be included in next release. type: question Request for information or clarification. Not an issue.
Projects
None yet
Development

No branches or pull requests

4 participants